[gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 3048 bytes --]

Gentoo Dev's,

My name is Sérgio Almeida, I am Portuguese and I am a student for this
year's Google SoC coding the Universal Select Tool project for Gentoo
being Sébastien Fabbro (bicatali) my mentor.

Abstract:

Universal Select Tool is an utility to manage system configuration.
This tool is similar to the unmaintained eselect utility of Gentoo or
Exherbo's eclectic. The idea is to create a tool that  manages both
system settings and user settings with profile creation possibilities.
The utility will use mostly concepts from "modules", "softenv", and
both "eselect" and "eclectic".

My initial proposal does not get in-depth with implementation details
and I need to make some decisions as soon as possible. Implementation
language will be python as it is easy to maintain, easy to code and
faster and more flexible than bash. See attachment for more details.

Besides introducing myself, the purpose of this e-mail is a
call-to-ideas to all Gentoo developers, mainly all eselect-* and
*-config developers.

Here are the main interest ideas:

* keep eselect structure of modules - actions

* symlinking, environment and aliases actions can consist of something
like:

        # module moo comments
        description "Example Module description"
        version "Example Module Version"
        author "moo@farm.moo"
        # action system moo
        description "Moo Action Description"
        symlink "regexp" "regexp"
        env "regexp" "regexp"
        alias "regexp" "regexp"
        # end moo
        
These should get the job done for most of the modules and opens the door
to automatic module creation prior to a successful emerge (if some USE
flag set)

* Actions that consist of code blocks that support any scripting
language (what about binaries?) to do more complex actions (full module
example):

    # module moo comments
        description "Example Module description"
        version "Example Module Version"
        author "moo@farm.moo"
        
        # action user moo
        description "Example Module will moo for any user"
        type runnable
        runner /bin/bash
        # file moo.bash
        #!/bin/bash
        do_moo() {
        	echo "This is the Example Module mooing"
        }
        do_moo()
        # end moo.bash
        # end moo

* actions can be run system-wide and per-user:
        # action user moo
        # action system moo

* automatic module loading and profile management can be managed by some
env.d python scripts that are user-aware and follow some database
        
I've been given this difficult task of unifying all of these tools
together and as you all can understand, I won't be having the time to
read through all eselect-* modules and *-config utilities code.

Please drop me a line here or at freenode if you have anything to add to
these ideas or have any further ideas that can help me on this project.
Thank you all in advance.

Cheers,
Sérgio Almeida
mephx @ freenode


[-- Attachment #2: proposal --]
[-- Type: text/plain, Size: 4429 bytes --]

Universal Select Tool

*Abstract

Universal Select Tool is an utility to manage system configuration.
This tool is similar to the unmaintained eselect utility of Gentoo or
Exherbo's eclectic. The idea is to create a tool that  manages both
system settings and user settings with profile creation possibilities.
The utility will use mostly concepts from "modules", "softenv", and
both "eselect" and "eclectic".

*Objective

The objective of this project is to create a unified configuration
utility for gentoo. Why this new tool and not eselect? Gentoo has a
set of configuration utilities (eselect-* *-config) that work with
global environment settings regarding the system and therefore needed
to be contradicted by hand by users to fulfill their need to change
it's own defaults. This is a common problem within clusters,
shared-servers, etc, basically all multi-user, multi-profile,
multi-use environments. Creating this utility makes environment
profiles, slots, virtuals and multi-implementations easier to manage,
create and manipulate by developers, sysadmins and users. The current
available solutions do not work this way and therefore enlarging the
gap between configuration unification, profiling and usability.

*General Ideas

Some modules that this project wants to unite do much more than just
update environment variables. Further discussion should be done with
all participants in such projects to check if a general tool can be
created and if not, what can be split as a module and what can not.
Modules will need to be thought in a way that no further *-config and
eselect-* utilities need to exist anymore. These utilities will be
replaced by a module of it's own that allows the universal select tool
to manipulate all available configuration that is specified by the
module.
Modules will not only specify to the universal select tool what can be
changed but also how to make those changes.
Modules will be indexed in it's own database file for faster
searching, switching, and profile creation and managing. This database
will need to be updated by portage upon each successful module-able
emerge.

*Key features

Merging all eselect-* and *-config modules into one unified framework.
Quick and easy plugin of new modules when a virtual or several slots
of one package exist to allow modul-ization of several packages.
Fast.
Per-User and Per-System module settings.
Create profiles of module settings, Per-User and Per-System.
Possible new modules: mysql, tex, intel compiler suite, fortran, etc.

*Deliverables

an application (probably uselect) and the corresponding ebuild
documentation
module examples
ebuild examples
patch to portage (possibly)

Programming language is yet to be defined but possibly python, bash or C.

*Timeline

April 20 - May 23

Get to know code/documentation/use of "modules", "softenv", "eselect",
"ecletic" and "portage".
Highlight base features and algorithms to reimplement/discard.
Make decisions regarding the base of the aplication such as databases,
module format/handling, dependencies in portage and user environment
handling.
Set up a gentoo virtual machine that can be replicated to use as clean
sandboxes for testing througout the development with various
environments.

May 23 - July 3

Choose and setup the build environment for the new utility.
Begin implementation of the base features and module system.
Create sample modules.

(Mid Milestone) July 4 - July 30

Retouch wathever is needed.
Implement Portage's module handling part.
Create sample ebuild.

(Final Milestone) August 1 - August 17

Tests
Re-touching
Documentation

*Biography

My name is Sérgio Almeida and I live in Lisbon, Portugal. I am 22
years old and Informatics have always been my interest since I first
got in touch with a friend's Comodore. I study informatics & computers
science engineering @ Instituto Superior Técnico and plan to get my
MsC degree in 2012. My main interests are distributed systems and
computer/network security. I started working as a sysadmin at the age
of 16 at VirtualGames, a game server provider for fun and without any
profit. I am currently working as sysadmin at my University's Physics
Investigation department. Besides this i play bass in a rock band,
«don't disturb my circles» and I enjoy listening, composing, recording
and mixing music.

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Denis Dupeyron]

2009/5/4 Sérgio Almeida <mephx.x@gmail.com>:
> Please drop me a line here or at freenode if you have anything to add

All I have to add is thanks a lot. I wished other SoC students would
also introduce themselves and their project on their list.

Denis.

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Mark Loeser]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

Sérgio Almeida <mephx.x@gmail.com> said:
> Abstract:
> 
> Universal Select Tool is an utility to manage system configuration.
> This tool is similar to the unmaintained eselect utility of Gentoo or
> Exherbo's eclectic. The idea is to create a tool that  manages both
> system settings and user settings with profile creation possibilities.
> The utility will use mostly concepts from "modules", "softenv", and
> both "eselect" and "eclectic".

I guess this is a very high level question...but do we need yet another
eselect?  Why can't we enhance or fix what we already have rather than
creating everything from scratch?

-- 
Mark Loeser
email         -   halcy0n AT gentoo DOT org
email         -   mark AT halcy0n DOT com
web           -   http://www.halcy0n.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Sérgio Almeida]

On Sun, 2009-05-10 at 18:56 -0400, Mark Loeser wrote:
> Sérgio Almeida <mephx.x@gmail.com> said:
> > Abstract:
> > 
> > Universal Select Tool is an utility to manage system configuration.
> > This tool is similar to the unmaintained eselect utility of Gentoo or
> > Exherbo's eclectic. The idea is to create a tool that  manages both
> > system settings and user settings with profile creation possibilities.
> > The utility will use mostly concepts from "modules", "softenv", and
> > both "eselect" and "eclectic".
> 
> I guess this is a very high level question...but do we need yet another
> eselect?  Why can't we enhance or fix what we already have rather than
> creating everything from scratch?
> 

Mark,

>From my point of view, uselect is not YA eselect. eselect wasn't thought
from the beginning to be universal and therefore it would need a full
re-write as you suggest. At this point (and SoC hasn't yet started) I
have implemented uselect with all eselect capabilities in python (served
as well to un-rust myself from python programming) and it is extremely
faster. uselect supports modules in any scripting language (implemented
too) and eselect only supports bash. uselect new architecture supports
the auto-creation of simple symlinking/environment/alias modules (most
of them will be it) using only a few regular expressions to define what
to change and not how to change (uselect will do it for you).

At this point eselect may be considered deprecated as it's
functionalities are limited and there is no much we can do besides
bloating bash code (what you call enhance) and bugfixing.

Let us all consider uselect another utility with some "similar"
functionalities from eselect but with a lot more features from modules
and softenv.

I knew your question would pop up and it is important to everyone to
know the differences between the two utilities.

I will post (in a near future) an example package for everyone to test
and give it's opinion on how should every idea work instead of leaving
all the decisions to me.

Thanks.

Cheers,
Sérgio
-- 
Sérgio Almeida <mephx.x@gmail.com>
mephx @ freenode

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Michael Haubenwallner]

If there is a different place for requirement discussion of Gentoo
specific GSoC projects, please tell, and sorry for bothering here then.

On Mon, 2009-05-04 at 22:01 +0100, Sérgio Almeida wrote:
> Here are the main interest ideas:
> * actions can be run system-wide and per-user:
>         # action user moo
>         # action system moo

Are there any thoughts to support something more fine granular settings
than system and user?

What I'm after:
We are developing multiple source projects with multiple release
branches on the same host here. These projects do have some script to
set up the project specific environment. One os-user is working on
multiple projects or branches, entering a project's environment by
sourcing its environment script.

Naturally, these projects do have different requirements to - for
example - the java-vm version. The project admin needs to select the
java-vm on a per-project basis, and does want to use the system-vm as
fallback, never wants to use the user-vm.

With current eselect (and java-config-2), this would be something like
setting $HOME on the per-project basis - or not using java-config at
all.

Would it make sense to explicitly set the system-vm (whatever version it
is or will be changed to), while leaving it unset would fall back to the
user-vm?

More toughts?

Thanks!

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Sérgio Almeida]

Hello,

On Tue, 2009-05-12 at 10:40 +0200, Michael Haubenwallner wrote:<
> If there is a different place for requirement discussion of Gentoo
> specific GSoC projects, please tell, and sorry for bothering here then.

There is another mailing list, but i am sure it doesn't get to as much
dev's as this one gets.

> 
> On Mon, 2009-05-04 at 22:01 +0100, Sérgio Almeida wrote:
> > Here are the main interest ideas:
> > * actions can be run system-wide and per-user:
> >         # action user moo
> >         # action system moo
> 
> Are there any thoughts to support something more fine granular settings
> than system and user?
> 

Indeed, user is not "for all the users". It's an action that can be run
by the users to change it's own settings without touching the system's
fallback default.

> What I'm after:
> We are developing multiple source projects with multiple release
> branches on the same host here. These projects do have some script to
> set up the project specific environment. One os-user is working on
> multiple projects or branches, entering a project's environment by
> sourcing its environment script.
> 
> Naturally, these projects do have different requirements to - for
> example - the java-vm version. The project admin needs to select the
> java-vm on a per-project basis, and does want to use the system-vm as
> fallback, never wants to use the user-vm.
> 
> With current eselect (and java-config-2), this would be something like
> setting $HOME on the per-project basis - or not using java-config at
> all.
> 
> Would it make sense to explicitly set the system-vm (whatever version it
> is or will be changed to), while leaving it unset would fall back to the
> user-vm?

That's the point of the uselect tool. Per-System settings, Per-User
settings (2 different ssh sessions of the same user can still have
different environment settings too).

I work as a sysadmin and manage mainly multi-user gentoo environments
where people develop calculus c/python/fortran/R/whatever code using
wathever utilities we can imagine. Everytime a new project is beeing
done people need to set the environment variables themselves when they
are kind enough not to ask me. That's mainly the whole purpose of this
uselect tool, easy multi user environments managing.

> 
> More toughts?
> 
> Thanks!

I Thank you!

> 
> /haubi/

I would be glad if developers from the *-config utilities could post
here and say what their utilities can do that eselect is/was incapable
of doing and also the reason why their tools were created for us to find
a way that a new "universal select tool" can gather all of *-config
features with somthing like these proposed modules.

Cheers,
Sérgio
-- 
Sérgio Almeida <mephx.x@gmail.com>
mephx @ freenode

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Michael Haubenwallner]

On Tue, 2009-05-12 at 15:32 +0100, Sérgio Almeida wrote:
> > On Mon, 2009-05-04 at 22:01 +0100, Sérgio Almeida wrote:
> > > Here are the main interest ideas:
> > > * actions can be run system-wide and per-user:
> > >         # action user moo
> > >         # action system moo
> > 
> > Are there any thoughts to support something more fine granular settings
> > than system and user?
> 
> Indeed, user is not "for all the users". It's an action that can be run
> by the users to change it's own settings without touching the system's
> fallback default.

Seems you didn't really understand my problem yet - another try:
The *one* and *same* user does need different settings at the *same*
time depending on the project he's currently working on in different
shells. The actual setting needs to be set up by the project's
environment script (when the users enters the project), not the user's
profile script (when the user logs in).

> That's the point of the uselect tool. Per-System settings, Per-User
> settings (2 different ssh sessions of the same user can still have
> different environment settings too).

This maybe is what I'm after: Question still is how to easily set up the
different environment?

> I work as a sysadmin and manage mainly multi-user gentoo environments
> where people develop calculus c/python/fortran/R/whatever code using
> wathever utilities we can imagine. Everytime a new project is beeing
> done people need to set the environment variables themselves when they
> are kind enough not to ask me. That's mainly the whole purpose of this
> uselect tool, easy multi user environments managing.

Seems like we have a similar job ;)
Maybe it's the wording only, but in addition to the multi-user dimension
I'm still missing the multi-project dimension for one user.

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Sérgio Almeida]

Michael,

> Seems you didn't really understand my problem yet - another try:
> The *one* and *same* user does need different settings at the *same*
> time depending on the project he's currently working on in different
> shells. The actual setting needs to be set up by the project's
> environment script (when the users enters the project), not the user's
> profile script (when the user logs in).
> 

Ok, now I fully understand you and I agree. A new feature? Mmm... This
can be managed by profile creation and switching. Imagine, user can
create a profile with it's current uselect settings, select a default
profile, select the active profile on-the-fly, and on and on... Works,
but not automatic...

Maybe we can optimize this by having a similar behavior of .svn folders
on svn settings. How about having the profile switch automatic whenever
you call "uselect something" getting the current profile settings from
current dir's .uselect folder?

I am starting the plans on profile managing today. This now gets
interesting.

Thanks!

Cheers,
Sérgio
-- 
Sérgio Almeida <mephx.x@gmail.com>
mephx @ freenode

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Michael Haubenwallner]

On Tue, 2009-05-12 at 21:45 +0100, Sérgio Almeida wrote:

> How about having the profile switch automatic whenever
> you call "uselect something" getting the current profile settings from
> current dir's .uselect folder?

Yeah, this indeed could work!

But there is one restriction:
This .uselect folder must be able to be checked in into some VCS, so it
should not contain symlinks, but plain (text?) files only.
We also want to use this on Windows based filesystems where symlinks
don't work at all.

Ohw, I do have another requirement:
We do check-out/compile/develop/run/test the same package on different
hosts and platforms. Each of these hosts might require slightly
different settings. ATM, the package's environment script handles this
by acting differently based on `hostname` and `uname` or "${chost}".
Ohw, it even does different settings based on the username (`id -un` or
`whoami`), because in production these projects usually run under a
specific user with less restrictive settings than for developers.

What if there is some hierarchy in .uselect/ much like the profiles in
gentoo-x86 tree, using some kind of 'parent' files to inherit/override
settings for this one project, where 'parent' can contain something like
$(CHOST), $(UNAME), $(HOSTNAME), $(USERNAME)?

I'm unsure if managing different settings based on hostname, uname/chost
and username (the inheritance tree) is uselect's job. It eventually
should optionally listen to some cmdline-parameter or environment-
variable where to look for the uselect settings instead, which could be
stored and regenerated using such profile hierarchy mechanism/utility.
Or even the whole uselect settings optionally come from stdin (and go to
stdout), to be managed/stored within that profile hierarchy.

Ha! This kind of inheritance tree could be a solution for my long
standing bug here to simplify our way too complex environment script...

Ah, don't forget to put a version number as the very first value into
the uselect settings, to avoid backwards compatibility issues in the
future. And when uselect settings can be read from stdin (stored in
simple text files), this version number could occur multiple times
there, as the stored files simply will be concatenated. And subsequent
values might override previous ones then.

Uh, so many strange ideas!
More of them?

Thanks!

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Sérgio Almeida]

Michael,

> Yeah, this indeed could work!
> 
> But there is one restriction:
> This .uselect folder must be able to be checked in into some VCS, so it
> should not contain symlinks, but plain (text?) files only.
> We also want to use this on Windows based filesystems where symlinks
> don't work at all.
> 

Do we really need more than one file? 

> Ohw, I do have another requirement:
> We do check-out/compile/develop/run/test the same package on different
> hosts and platforms. Each of these hosts might require slightly
> different settings. ATM, the package's environment script handles this
> by acting differently based on `hostname` and `uname` or "${chost}".
> Ohw, it even does different settings based on the username (`id -un` or
> `whoami`), because in production these projects usually run under a
> specific user with less restrictive settings than for developers.
> 

Checklist:
* Hostname
* Uname
* {$chost}

Mmm... Maybe we can simplify this with a parameter like:

# eval something
eval "hostname" "superhost"
what
to
do
# end something

Then if command "hostname" outputs "superhost" we know "what-to-do". 

This way it would get ultra versatile.

> What if there is some hierarchy in .uselect/ much like the profiles in
> gentoo-x86 tree, using some kind of 'parent' files to inherit/override
> settings for this one project, where 'parent' can contain something like
> $(CHOST), $(UNAME), $(HOSTNAME), $(USERNAME)?
> 

Would this really be necessary? We can define hierarchy into a
single .uselect file. Even the use of folders (folder .uselect/) isn't
necessary. I think a single file can handle all this... Lets see an
example:

# profile something
version '0.1'
do this 1
do that 1

# eval hostname
eval "hostname" "supermachina"

do that 2 # this should override the prev. value

# eval whomai <- inheritance inside eval hostname
eval "whoami" "mephx"

do more of this
do more of that

# end whoami
# end hostname

# eval uname
eval "uname" "Linux"

do this 3 <- override the overridden =)

# end uname


The actions to be done like "do this 3" are a simple call to uselect
using module "do" and action "this" with "3" as parameters.

Remember that profile creation/storing/managing should be automatic and
nothing would need to be written by hand. By other words, create a basic
profile automatically using your currently running settings and then
alter the profile with the util to add constrains to it. Remember that
all the machines that this profile is read would need to have the same
uselect modules into it's /usr/share/uselect/modules or similar.

> I'm unsure if managing different settings based on hostname, uname/chost
> and username (the inheritance tree) is uselect's job. It eventually
> should optionally listen to some cmdline-parameter or environment-
> variable where to look for the uselect settings instead, which could be
> stored and regenerated using such profile hierarchy mechanism/utility.
> Or even the whole uselect settings optionally come from stdin (and go to
> stdout), to be managed/stored within that profile hierarchy.
> 
> Ha! This kind of inheritance tree could be a solution for my long
> standing bug here to simplify our way too complex environment script...
> 

This is a basic idea from softenv so it has has it's place into uselect.
The question now is, bloat it all into uselect or create a uprofile
util? I like the idea of having to use only uselect for basic profiling
and using uprofile for further managing.

> Ah, don't forget to put a version number as the very first value into
> the uselect settings, to avoid backwards compatibility issues in the
> future. And when uselect settings can be read from stdin (stored in
> simple text files), this version number could occur multiple times
> there, as the stored files simply will be concatenated. And subsequent
> values might override previous ones then.
> 

Mmmm... As you wrote I realized: Complain if module versions are
different is not a bad idea at all... 

Why would we need more that one profile file per project/folder anyway?


> Uh, so many strange ideas!
> More of them?
> 

Keep 'em coming! Thanks!

Cheers,
Sérgio

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Michael Haubenwallner]

On Wed, 2009-05-13 at 16:40 +0100, Sérgio Almeida wrote:

> > This .uselect 
> > should not contain symlinks, but plain (text?) files only.

> Do we really need more than one file? 

No, at least not to define the _values_ of a profile.

> Checklist:
> * Hostname
> * Uname
> * {$chost}
> 
> Mmm... Maybe we can simplify this with a parameter like:
> 
> # eval something
> eval "hostname" "superhost"
> what
> to
> do
> # end something
> 
> Then if command "hostname" outputs "superhost" we know "what-to-do". 

Eventually, we should pass something like "-Dhostname=superhost" as
cmdline parameter to uselect...

> 
> This way it would get ultra versatile.
> 
> > What if there is some hierarchy in .uselect/ much like the profiles in
> > gentoo-x86 tree, using some kind of 'parent' files to inherit/override
> > settings for this one project, where 'parent' can contain something like
> > $(CHOST), $(UNAME), $(HOSTNAME), $(USERNAME)?
> > 
> 
> Would this really be necessary? We can define hierarchy into a
> single .uselect file. Even the use of folders (folder .uselect/) isn't
> necessary. I think a single file can handle all this... 

Eventually yes.

> Lets see an
> example:
> 
> # profile something

I don't like to have anything interpreted after the usual and well-known
comment-marker, this just feels hackish.

> 
> do this 3 <- override the overridden =)

> The actions to be done like "do this 3" are a simple call to uselect
> using module "do" and action "this" with "3" as parameters.

fine.

I do have something like this content-syntax in mind for .uselect (or
eventually some .uprofile) file:

8<8<8<
version "0.1"

include "path/to/another/uselect/file"

profile "generic solaris" {
  module A actionAX "valueAX1" "valueAX2"
  module B actionBX "valueBX1" "valueBX2
}

profile "generic host" {
  module C actionCX "valueCX1"
}

profile "superhost" {
  inherit profile "generic solaris"
  inherit profile "generic host"
  module C actionCX "newvalueCX1"
  module A actionAX "newvalueAX1" "newvalueAX2"
  module D actionDY "valueBY1" "valueBY2"
}

profile "generic user" {
  module E actionEX "valueEX1"
}

profile "user haubi" {
  inherit profile "generic user"
  module F actionFX "valueFX1"
}

profile "any user on superhost" {
  module G actionGX "valueGX1"
}

profile "fallback host" {
  inherit profile "generic host"
  module A actionAX "valueAX1" "valueAX2"
}

activation "superhost activation" {
  in category "host"
  on fallback level 0
  when $hostname matches string "superhost"
  activate profile "superhost"
}

activation "haubi on superhost" {
  in category "user"
  on fallback level 0
  when $username matches string "haubi"
  when $hostname matches string "superhost"
  activate profile "any user on superhost"
  activate profile "user haubi"
}

activation "haubi" {
  in category "user"
  on fallback level 1
  when $username matches string "haubi"
  activate profile "user haubi"
}

activation "gentoo host" {
  in category "host"
  on fallback level 1
  when $hostname matches regex ".*\.gentoo\.org"
  activate profile "fallback host"
}
>8>8>8

I'm not sure to have an ascending "fallback level" or descending
"priority" value, but both should allow for negative integer values.

The selection of one or more profiles is controlled by "activation"
entries, independent for each "category". The order and selection of
categories is done via a commandline argument, fex "--categories".

Profiles are selected when all of the "when" entries (besides "category"
and "fallback level") in "activation {}" do match.
Selected are *all* of the matching profiles in the *lowest* "fallback
level" *only*, which does have at least one matching profile.

And I'm thinking of something like this commandline:
$ uselect \
  --categories host,user \
  --define hostname=`hostname` \
  --define username=`whoami`

> 
> Remember that profile creation/storing/managing should be automatic and
> nothing would need to be written by hand.

Yes, would be fine. When using something like above configuration file
syntax, some GUIding tool would be useful, at least to see which
profiles are selected for specific cmdline args.

>  By other words, create a basic
> profile automatically using your currently running settings and then
> alter the profile with the util to add constrains to it.

Sounds interesting...

> Remember that
> all the machines that this profile is read would need to have the same
> uselect modules into it's /usr/share/uselect/modules or similar.

Indeed, yes.

> > Ha! This kind of inheritance tree could be a solution for my long
> > standing bug here to simplify our way too complex environment script...
> > 
> 
> This is a basic idea from softenv so it has has it's place into uselect.

Don't know softenv. Found http://www.teragrid.org/userinfo/softenv/
Do you mean this one?

> The question now is, bloat it all into uselect or create a uprofile
> util? I like the idea of having to use only uselect for basic profiling
> and using uprofile for further managing.

That's indeed the question.

> Mmmm... As you wrote I realized: Complain if module versions are
> different is not a bad idea at all... 

Indeed, not only the configuration needs to be versioned, but the
modules too.

> 
> Why would we need more that one profile file per project/folder anyway?

Might not be necessary, at least not for non-profiled uselect.

> > Uh, so many strange ideas!
> > More of them?
> 
> Keep 'em coming! Thanks!

Well, you have asked ;)

/haubi/

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 5977 bytes --]

Michael,

> Eventually, we should pass something like "-Dhostname=superhost" as
> cmdline parameter to uselect...
> 

Didn't want to get into that detail. Afterall `hostname` will always
return the same regarding the host therefore it can be retrieved from
uselect's engine.

> I don't like to have anything interpreted after the usual and well-known
> comment-marker, this just feels hackish.
> 

Agreed, it was just an "easy to read" example. Will adopt {} for those.

> 
> I do have something like this content-syntax in mind for .uselect (or
> eventually some .uprofile) file:
> 
> 8<8<8<
> version "0.1"
> 
> include "path/to/another/uselect/file"
> 
> profile "generic solaris" {
>   module A actionAX "valueAX1" "valueAX2"
>   module B actionBX "valueBX1" "valueBX2
> }
> 
> profile "generic host" {
>   module C actionCX "valueCX1"
> }
> 
> profile "superhost" {
>   inherit profile "generic solaris"
>   inherit profile "generic host"
>   module C actionCX "newvalueCX1"
>   module A actionAX "newvalueAX1" "newvalueAX2"
>   module D actionDY "valueBY1" "valueBY2"
> }
> 
> profile "generic user" {
>   module E actionEX "valueEX1"
> }
> 
> profile "user haubi" {
>   inherit profile "generic user"
>   module F actionFX "valueFX1"
> }
> 
> profile "any user on superhost" {
>   module G actionGX "valueGX1"
> }
> 
> profile "fallback host" {
>   inherit profile "generic host"
>   module A actionAX "valueAX1" "valueAX2"
> }
> 
> activation "superhost activation" {
>   in category "host"
>   on fallback level 0
>   when $hostname matches string "superhost"
>   activate profile "superhost"
> }
> 
> activation "haubi on superhost" {
>   in category "user"
>   on fallback level 0
>   when $username matches string "haubi"
>   when $hostname matches string "superhost"
>   activate profile "any user on superhost"
>   activate profile "user haubi"
> }
> 
> activation "haubi" {
>   in category "user"
>   on fallback level 1
>   when $username matches string "haubi"
>   activate profile "user haubi"
> }
> 
> activation "gentoo host" {
>   in category "host"
>   on fallback level 1
>   when $hostname matches regex ".*\.gentoo\.org"
>   activate profile "fallback host"
> }
> >8>8>8
> 
> I'm not sure to have an ascending "fallback level" or descending
> "priority" value, but both should allow for negative integer values.
> 

I'm not quite shure if this hierarchy fallback method is the most
adequate one. Again, remember that the profiles will be automatically
created and manipulation needs to be easy. I guess we can arrange for a
better way of managing this.

> The selection of one or more profiles is controlled by "activation"
> entries, independent for each "category". The order and selection of
> categories is done via a commandline argument, fex "--categories".
> 

We are mixing some more complex concepts that don't need to be managed
this way. Let's see.

Types of Profiles:

something.uprofile <- local file profile
.uprofile <- folder profile
/usr/share/uprofiles/*.uprofile <- template profiles

Adding categories to a folder profile is the same as having several file
profiles into that directory and load them on demand. Why specify to
uselect --categories=something when you can "uselelect loadprofile
something" (loads .uselect folder profile first, and then overrides
settings as specified in something.uprofile).

> Profiles are selected when all of the "when" entries (besides "category"
> and "fallback level") in "activation {}" do match.
> Selected are *all* of the matching profiles in the *lowest* "fallback
> level" *only*, which does have at least one matching profile.
> 
> And I'm thinking of something like this commandline:
> $ uselect \
>   --categories host,user \
>   --define hostname=`hostname` \
>   --define username=`whoami`
> 
> > 
> > Remember that profile creation/storing/managing should be automatic and
> > nothing would need to be written by hand.
> 
> Yes, would be fine. When using something like above configuration file
> syntax, some GUIding tool would be useful, at least to see which
> profiles are selected for specific cmdline args.
> 

Mmm... later feature for more complex profiling. Let us stop now and
focus on simple profiling with easy managing.

> >  By other words, create a basic
> > profile automatically using your currently running settings and then
> > alter the profile with the util to add constrains to it.
> 
> Sounds interesting...
> 
> > Remember that
> > all the machines that this profile is read would need to have the same
> > uselect modules into it's /usr/share/uselect/modules or similar.
> 
> Indeed, yes.
> 
> > > Ha! This kind of inheritance tree could be a solution for my long
> > > standing bug here to simplify our way too complex environment script...
> > > 
> > 
> > This is a basic idea from softenv so it has has it's place into uselect.
> 
> Don't know softenv. Found http://www.teragrid.org/userinfo/softenv/
> Do you mean this one?
> 

Indeed.

> > The question now is, bloat it all into uselect or create a uprofile
> > util? I like the idea of having to use only uselect for basic profiling
> > and using uprofile for further managing.
> 
> That's indeed the question.
> 
> > Mmmm... As you wrote I realized: Complain if module versions are
> > different is not a bad idea at all... 
> 
> Indeed, not only the configuration needs to be versioned, but the
> modules too.
> 

On to-do!

> > 
> > Why would we need more that one profile file per project/folder anyway?
> 
> Might not be necessary, at least not for non-profiled uselect.
> 
> > > Uh, so many strange ideas!
> > > More of them?
> > 
> > Keep 'em coming! Thanks!
> 
> Well, you have asked ;)
> 
> /haubi/


Thanks again.

Cheers,
Sérgio
-- 
Sérgio Almeida <mephx.x@gmail.com>
mephx @ freenode

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 3778 bytes --]

Am Montag, den 04.05.2009, 22:01 +0100 schrieb Sérgio Almeida:
> Gentoo Dev's,
> 
> My name is Sérgio Almeida, I am Portuguese and I am a student for this
> year's Google SoC coding the Universal Select Tool project for Gentoo
> being Sébastien Fabbro (bicatali) my mentor.
> 
> Abstract:
> 
> Universal Select Tool is an utility to manage system configuration.
> This tool is similar to the unmaintained eselect utility of Gentoo or
> Exherbo's eclectic. The idea is to create a tool that  manages both
> system settings and user settings with profile creation possibilities.
> The utility will use mostly concepts from "modules", "softenv", and
> both "eselect" and "eclectic".
> 
> My initial proposal does not get in-depth with implementation details
> and I need to make some decisions as soon as possible. Implementation
> language will be python as it is easy to maintain, easy to code and
> faster and more flexible than bash. See attachment for more details.
> 
> Besides introducing myself, the purpose of this e-mail is a
> call-to-ideas to all Gentoo developers, mainly all eselect-* and
> *-config developers.
> 
> Here are the main interest ideas:
> 
> * keep eselect structure of modules - actions
> 
> * symlinking, environment and aliases actions can consist of something
> like:
> 
>         # module moo comments
>         description "Example Module description"
>         version "Example Module Version"
>         author "moo@farm.moo"
>         # action system moo
>         description "Moo Action Description"
>         symlink "regexp" "regexp"
>         env "regexp" "regexp"
>         alias "regexp" "regexp"
>         # end moo
>         
> These should get the job done for most of the modules and opens the door
> to automatic module creation prior to a successful emerge (if some USE
> flag set)
> 
> * Actions that consist of code blocks that support any scripting
> language (what about binaries?) to do more complex actions (full module
> example):
> 
>     # module moo comments
>         description "Example Module description"
>         version "Example Module Version"
>         author "moo@farm.moo"
>         
>         # action user moo
>         description "Example Module will moo for any user"
>         type runnable
>         runner /bin/bash
>         # file moo.bash
>         #!/bin/bash
>         do_moo() {
>         	echo "This is the Example Module mooing"
>         }
>         do_moo()
>         # end moo.bash
>         # end moo
> 
> * actions can be run system-wide and per-user:
>         # action user moo
>         # action system moo
> 
> * automatic module loading and profile management can be managed by some
> env.d python scripts that are user-aware and follow some database
>         
> I've been given this difficult task of unifying all of these tools
> together and as you all can understand, I won't be having the time to
> read through all eselect-* modules and *-config utilities code.
> 
> Please drop me a line here or at freenode if you have anything to add to
> these ideas or have any further ideas that can help me on this project.
> Thank you all in advance.

What I'd like to see is the possibility to
... localize messages (will be difficult since modules need translations
as well, but maybe you find a way :)
... encapsulation of methods to set/list/change such that instead of a
CLI- a NCurses- or GUI-Frontend could be developed.

Cheers,
Tiziano



-- 
Tiziano Müller
Gentoo Linux Developer, Council Member
Areas of responsibility:
  Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail   : dev-zero@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5  4DE2 1B05 6A63 AE9C 1E30

[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 3916 bytes --]

On Tue, 2009-05-26 at 22:04 +0200, Tiziano Müller wrote:
> Am Montag, den 04.05.2009, 22:01 +0100 schrieb Sérgio Almeida:
> > Gentoo Dev's,
> > 
> > My name is Sérgio Almeida, I am Portuguese and I am a student for this
> > year's Google SoC coding the Universal Select Tool project for Gentoo
> > being Sébastien Fabbro (bicatali) my mentor.
> > 
> > Abstract:
> > 
> > Universal Select Tool is an utility to manage system configuration.
> > This tool is similar to the unmaintained eselect utility of Gentoo or
> > Exherbo's eclectic. The idea is to create a tool that  manages both
> > system settings and user settings with profile creation possibilities.
> > The utility will use mostly concepts from "modules", "softenv", and
> > both "eselect" and "eclectic".
> > 
> > My initial proposal does not get in-depth with implementation details
> > and I need to make some decisions as soon as possible. Implementation
> > language will be python as it is easy to maintain, easy to code and
> > faster and more flexible than bash. See attachment for more details.
> > 
> > Besides introducing myself, the purpose of this e-mail is a
> > call-to-ideas to all Gentoo developers, mainly all eselect-* and
> > *-config developers.
> > 
> > Here are the main interest ideas:
> > 
> > * keep eselect structure of modules - actions
> > 
> > * symlinking, environment and aliases actions can consist of something
> > like:
> > 
> >         # module moo comments
> >         description "Example Module description"
> >         version "Example Module Version"
> >         author "moo@farm.moo"
> >         # action system moo
> >         description "Moo Action Description"
> >         symlink "regexp" "regexp"
> >         env "regexp" "regexp"
> >         alias "regexp" "regexp"
> >         # end moo
> >         
> > These should get the job done for most of the modules and opens the door
> > to automatic module creation prior to a successful emerge (if some USE
> > flag set)
> > 
> > * Actions that consist of code blocks that support any scripting
> > language (what about binaries?) to do more complex actions (full module
> > example):
> > 
> >     # module moo comments
> >         description "Example Module description"
> >         version "Example Module Version"
> >         author "moo@farm.moo"
> >         
> >         # action user moo
> >         description "Example Module will moo for any user"
> >         type runnable
> >         runner /bin/bash
> >         # file moo.bash
> >         #!/bin/bash
> >         do_moo() {
> >         	echo "This is the Example Module mooing"
> >         }
> >         do_moo()
> >         # end moo.bash
> >         # end moo
> > 
> > * actions can be run system-wide and per-user:
> >         # action user moo
> >         # action system moo
> > 
> > * automatic module loading and profile management can be managed by some
> > env.d python scripts that are user-aware and follow some database
> >         
> > I've been given this difficult task of unifying all of these tools
> > together and as you all can understand, I won't be having the time to
> > read through all eselect-* modules and *-config utilities code.
> > 
> > Please drop me a line here or at freenode if you have anything to add to
> > these ideas or have any further ideas that can help me on this project.
> > Thank you all in advance.
> 
> What I'd like to see is the possibility to
> ... localize messages (will be difficult since modules need translations
> as well, but maybe you find a way :)
> ... encapsulation of methods to set/list/change such that instead of a
> CLI- a NCurses- or GUI-Frontend could be developed.
> 
> Cheers,
> Tiziano
> 

Hello,

These are the reasons of my current prototype refactoring phase. All
this will be possible.

Thanks!

Cheers,
Sérgio

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]