[gentoo-dev] Progress on Universal Select Tool

[gentoo-dev] Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 1551 bytes --]

Hello,

Current State of Universal Select Tool - uselect

Proposal link: 
http://socghop.appspot.com/student_project/show/google/gsoc2009/gentoo/t124022356237

uselect implementation started a few weeks before SoC officially started
to "de-rust" myself on python programming. 

Here follows a checklist of what it already does right now.

* modules syntax defined
* modules support any scripting language
* module conversion from eselect to uselect are very easy (even when not
symlinking)
* per-user/system-wide actions
* simple symlinking actions are defined in 1 line only. uselect does all
the job.
* changing a user python interpreter through adding ~/.uselect/bin to
PATH through /etc/profile (is this the better way?)

Next steps:

* look deeper into eselect already implemented functions that most
modules use (by using inherit) and see what is uselect still lacking
* define profiling system (nice thread on gentoo-dev on this)
* start the profile system implementation
* start using gentoo's git

Further Notes:

* At this point I am ahead on schedule because of my early start. With
this I have plenty of time to implement features that were not described
on the proposal.
* Soon I will push the code into gentoo's git. If you have time, take a
look at it. Hope these reports are taken also as a "call for ideas".

Hope I'm not lacking any information. I'm loving working with Gentoo on
this. Learned a lot until now.

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 1987 bytes --]

On Tue, 2009-06-16 at 15:48 +0100, Sérgio Almeida wrote:
> Hello,
> 
> Current State of Universal Select Tool - uselect
> 
> Proposal link: 
> http://socghop.appspot.com/student_project/show/google/gsoc2009/gentoo/t124022356237
> 
> uselect implementation started a few weeks before SoC officially started
> to "de-rust" myself on python programming. 
> 
> Here follows a checklist of what it already does right now.
> 
> * modules syntax defined
> * modules support any scripting language
> * module conversion from eselect to uselect are very easy (even when not
> symlinking)
> * per-user/system-wide actions
> * simple symlinking actions are defined in 1 line only. uselect does all
> the job.
> * changing a user python interpreter through adding ~/.uselect/bin to
> PATH through /etc/profile (is this the better way?)
> 
> Next steps:
> 
> * look deeper into eselect already implemented functions that most
> modules use (by using inherit) and see what is uselect still lacking
> * define profiling system (nice thread on gentoo-dev on this)
> * start the profile system implementation
> * start using gentoo's git
> 
> Further Notes:
> 
> * At this point I am ahead on schedule because of my early start. With
> this I have plenty of time to implement features that were not described
> on the proposal.
> * Soon I will push the code into gentoo's git. If you have time, take a
> look at it. Hope these reports are taken also as a "call for ideas".
> 
> Hope I'm not lacking any information. I'm loving working with Gentoo on
> this. Learned a lot until now.
> 
> Cheers,
> Sérgio

Hello,

Just made the initial commit on gentoo's git.

http://git.overlays.gentoo.org/gitweb/?p=proj/uselect.git;a=summary

Please take the time to take a look at uselect and leave me some
feedback if possible.

install.sh should get uselect working for you.

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 2817 bytes --]

Hello,

Last week has been a very "paradigmatic" working week. Several problems
urged with the addition of symlinking dependency/inheritance functions. 

sym actions where converted to the form of:

user action bin {
	description "Change Python's Version"
	type sym
	sym python {
		bin python
		target /usr/bin/python 
		prefix /usr/bin/ 
		regexp python([0-9]+\.[0-9]+$)
		sym python-config {
			bin python-config
			destination /usr/bin/python-config
			prefix /usr/bin/ 
			regexp python([0-9]+\.[0-9]+)-config($)
		} python-config
	} python
} bin

Soon urged the need for more complex lexical analysis and started
implementing lex rules and yacc skeleton.

With this step a question bounced into my head. 

Am I reinventing the wheel? 
Why implement lex/yacc to translate a block of code into a python's
block of code?
Why not use plain python in modules?

After discussing with mentor, we decided to adopt python as the base
language for uselect modules.

The final objective is something similar to this:

# Start of Module

# Define the module
module = Module(description = "Python Version Switcher", version =
"0.1", author ="mephx.x@gmail.com")

# Define a Symlinking Action
bin = Action (description = "Change Python's Version", type = "sym")

#Define the SymLinks
python = Link(bin = "python", target = "/usr/bin/python", prefix =
"/usr/bin/", regexp = "python([0-9]+\.[0-9]+$")

python-config = Link(bin = "python-config", target =
"/usr/bin/python-config", prefix = "/usr/bin/", regexp =
"python-config([0-9]+\.[0-9]+$")

python.add_link(python-config) # For inheritance
bin.add_link(python) # Adding The Link to the action
module.add_action(bin) #Adding the action to the module

# End of Module

This may seem complex at a glance. But why learn uselect scripting
language when you do modules without even knowing python?

Keep in mind that runnable actions will still be supported and therefore
supporting any scripting language for actions. Python will be the
easiest one as will depend on basic class inheritance and implementation
of functions.

All this lead to the start of a new phase. Define a strong and simple
API definition for all of uselect's capabilities creating the interfaces
easily usable.

uselect's profiles will also be in plain python, using the API of
uselect's Python Module umodule.

Last week had no commits to git due to all this.

I intend to clean up all the "basic (ang buggy) lexical analysis" that
was already implemented and re-implement the new module API during this
week. 

Stay tuned on git for more updates during this week.

http://git.overlays.gentoo.org/gitweb/?p=proj/uselect.git;a=summary 

Cheers,
Sérgio

-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Sebastian Pipping]

Sérgio Almeida wrote:
> user action bin {
> 	description "Change Python's Version"
> 	type sym
> 	sym python {
> 		bin python
> 		target /usr/bin/python 
> 		prefix /usr/bin/ 
> 		regexp python([0-9]+\.[0-9]+$)
> 		sym python-config {
> 			bin python-config
> 			destination /usr/bin/python-config
> 			prefix /usr/bin/ 
> 			regexp python([0-9]+\.[0-9]+)-config($)
> 		} python-config
> 	} python
> } bin
> 
> Soon urged the need for more complex lexical analysis and started
> implementing lex rules and yacc skeleton.
> 
> With this step a question bounced into my head. 
> 
> Am I reinventing the wheel? 
> Why implement lex/yacc to translate a block of code into a python's
> block of code?
> Why not use plain python in modules?
> 
> After discussing with mentor, we decided to adopt python as the base
> language for uselect modules.

It seems to me that the original langauge is "static"/"descriptive"
while Python is not.  Why not move to XML or JSON (former seems more
common with Gentoo) instead of Python?  Think about how much easier it
is to pull information from metadata.xml than from .ebuild files - it's
the same difference in your case.

You know much better where you want to go with this than I do, but
please triple-check this move, as you cannot go back.

Thanks for listening,



Sebastian

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 1928 bytes --]

Sebastian,

> It seems to me that the original langauge is "static"/"descriptive"
> while Python is not.  Why not move to XML or JSON (former seems more
> common with Gentoo) instead of Python?  Think about how much easier it
> is to pull information from metadata.xml than from .ebuild files - it's
> the same difference in your case.

I considered XML/JSON in this decision and did not even discussed it
with my mentor. This is indeed not and abandoned idea as I explain
below.

> 
> You know much better where you want to go with this than I do, but
> please triple-check this move, as you cannot go back.
> 

I'm in no position to chose the path to take but for now python still
seems the chosen "vehicle". Remember that using XML/JSON for modules
would never need a re-write from uselect but would only be an extension
to the module system. Let us see.

In uselect everything are objects. 

Module is a class
Action is a class

Module(s) have Action(s)

Actions are interfaces to many kinds of actions, Sym, Path, Runnable

Sym and Path actions have Link(s)

Runnable actions have code and know how to execute it.

Therefore the backend scenario (object space) is the same as the new
suggested module syntax.

Basically this decision is not adding a feature. It is abandoning the
"uselect syntax" thus removing a feature.

This will help during development as parsing is the task that has been
more time consuming during uselect's development.

With this uselect will have an open interface for extension to any
markup language that comes in handy in module readability/programming.

> 
> Thanks for listening,
> 
> 
> 
> Sebastian
> 

I thank you. Will now add the markup language support to the to-do list,
not to implement right away but to have in mind while expanding the API.

Cheers,
Sérgio

-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

Hello,

Missed a weekly report, busy week. Will try to post 2 reports this week
as I am also working twice as much time.

Progress since last report:

* Converted all modules into plain python.
* uselect got even faster
* symlinking dependencies (ex: ruby and ruby-gems)
* several links per action (ex: bash-completion)

New module example:

# Python Module

from umodule import *

module = Module(name = "python", description = "Python Version
Switcher", version = "0.1", author ="mephx.x@gmail.com") # We define the
module
bin = Action (name = 'bin', description = "Change Python's Version",
type = "sym") # Define a Symlinking Action

python = Link(alias = "python", target = "/usr/bin/python", prefix =
"/usr/bin/", regexp = "python([0-9]+\.[0-9]+$)") 
python_config = Link(alias = "python-config", target =
"/usr/bin/python-config", prefix = "/usr/bin/", regexp =
"python([0-9]+\.[0-9]+)-config$")


bin.add_link(python_config) # For inheritance
bin.add_link(python) # Adding The Link to the action
module.add_action(bin) #Adding the action to the module

# End

Only 1 module per file and modules are retrieved via the variable
"module" on each module. I couldn't find out a better way of doing this
but I'm sure there is one. Anyone?

This seems a bit messy but will be much easier to create support for
markup languages (I'm starting to love the idea of JSON).

This is a very good progress but brought a problem. Managing symlinking
dependencies (tree-like) became a huge headache. (Due to infinite
dependency capabilities). Suggestions are welcome in this part.

At this point I am creating a new way of managing several targets and
it's dependencies automatically in an indexed way either for easy
displaying and also for easy choosing. (No indexing is beeing done right
now, just plain object count)

I plan to start the profiling capabilities later this week.

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Progress on Universal Select Tool

[Michael Haubenwallner]

On Mon, 2009-07-13 at 16:36 +0100, Sérgio Almeida wrote:
> Hello,
> 
> Missed a weekly report, busy week. Will try to post 2 reports this week
> as I am also working twice as much time.
> 
> Progress since last report:

> * symlinking dependencies (ex: ruby and ruby-gems)

Sorry, I've lost track on what's really going on here:
Does 'symlink' mean symlinks on the filesystem like 'ln -s' does?
Or, more important, does uselect create those symlinks itself?
If yes, please don't forget that OS without symlinks...

Thank you!
/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

Re: [gentoo-dev] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

Hello,

On Tue, 2009-07-14 at 11:20 +0200, Michael Haubenwallner wrote:
> 
> Sorry, I've lost track on what's really going on here:
> Does 'symlink' mean symlinks on the filesystem like 'ln -s' does?
> Or, more important, does uselect create those symlinks itself?
> If yes, please don't forget that OS without symlinks...
> 

What is the best to way to "workaround" these cases?

Linking /usr/bin/python with /usr/bin/python2.6 can be done using a
similar script to this:

#!/bin/bash
# This is /usr/bin/python
/usr/bin/python2.6 $@

Is there a better way when no symlinking is supported?

> Thank you!
> /haubi/

Cheers,
Sérgio

-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

On Tue, 2009-07-14 at 17:07 +0200, Michael Haubenwallner wrote:
> 
> Usually the files are just copied instead of symlinked there.
> 
> But as far as I know, there is no Unix-ish OS without symlink support -
> even Interix does support them.
> Although I cannot definitively say we won't have to select between
> multiple executables and/or shared libraries for native Windows once -
> they would have to be copied instead of symlinked then.
> 
> So it should be fine as long as 'symlink' can potentially be implemented
> as 'copy' for specific platforms.
> 

Can be done...

def create_symlink(self, source, destination):
	self.delete_file(destination)
	os.symlink(source, destination)
	
Therefore can be easily replaced for copy. What should be the default?
Should uselect be able to supply both options like "uselect --copy bla
bla" overrides os.symlink to copy function?

> Thanks!
> /haubi/

Thanks for the hint!

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Progress on Universal Select Tool

[Michael Haubenwallner]

On Wed, 2009-07-15 at 16:43 +0100, Sérgio Almeida wrote:
> On Tue, 2009-07-14 at 17:07 +0200, Michael Haubenwallner wrote:
 
> > So it should be fine as long as 'symlink' can potentially be implemented
> > as 'copy' for specific platforms.
	
> ... can be easily replaced for copy. What should be the default?
> Should uselect be able to supply both options like "uselect --copy bla
> bla" overrides os.symlink to copy function?

Ideally it is transparent to the uselect-user (either end-user or
developer) if the implementation actually does the symlink or emulates
it via copy.
It might be enough to select the implementation based on the
"host-system" (CHOST), which may be different to the
"build-system" (CBUILD).
Eventually, uselect won't need to detect the CHOST itself, but get it
from some external resource (cmdline, env-var, config-file, ...).

> Thanks for the hint!

nP.

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

[gentoo-dev] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 6600 bytes --]

Hello,

This has been a very productive week regarding uselect.

I'm still not doing any commits as uselect can still break your python
environment while testing and i don't have the time to learn how to
handle branches in git.

Status:

* Fully Converted all action types to new module syntax
* Infinite symlinking dependency fully working
* Runnable Actions now support dynamic usage showing

An example fully commented module follows

### Start of Module

#!/usr/bin/env python
# Copyright 1999-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# python.py mephx.x@gmail.com 
# (will explain .py instead of .uselect below)
# still lacking vim/emacs python line

from umodule import * 

# We Create The Module
module = Module(name = "python", description = "Python Version
Switcher", version = "0.1", author ="mephx.x@gmail.com")

# We Create a Linking Action
action_bin = Action (name = 'bin', description = "Change Python's
Version", type = "sym")

# We Create a Link
link_python = Link(alias = "python", target = "/usr/bin/python", prefix
= "/usr/bin/", regexp = "python([0-9]+\.[0-9]+$)")

# We Create Another Link
link_python_config = Link(alias = "python-config", target =
"/usr/bin/python-config", prefix = "/usr/bin/", regexp =
"python([0-9]+\.[0-9]+)-config$")

# Make the link "python_config" depend of "link_python"
link_python.add_link(link_python_config)

# Add the link to the action
action_bin.add_link(python)

# Add the action to the module
module.add_action(action_bin)


# We create a simple runnable action that "tests" python environment
test = Action (name = 'test', description = 'Test Python Environment',
type = 'runnable')

# We add the parameters/arguments

test.add_parameter('<times>')
test.add_parameter('[<what>]')

# We add the usage code

test.add_usage("""#!/usr/bin/python
print "Python Test will echo <times> * <what>"
print "Usage can be multi-line Yay!"
print ""
for i in range(5):
	print "Usage can have dynamic options! " + str(i)
""")

# We add the action code
test.add_code("""#!/usr/bin/python
import sys

def is_number(s):
    try:
        int(s)
        return True
    except ValueError:
        return False
		
argv = sys.argv[1:]
times = sys.argv[1:][0]
if is_number(times):
	times = int(times)	
	if len(argv) >= 2:
		what = sys.argv[1:][1]
	else: 
		what = 'Hello World...'
	print str(what) * times
else:
	print 'Invalid Option(s)'
	exit(1)
exit(0)
""")

# We add the action to the module
module.add_action(test)

### End of Module


So... what's the output of this?

### Start Examples

* uselect 

Usage: uselect <options> <module> <action> 

 Options:
    -v  		(*) Verbose Mode
    -nc 		(*) No Colors
    -version    	(*) Version Information

Modules:
    python      	(*) Python Version Switcher
    gcc 		(*) Python GCC Version Switcher

* uselect python

Usage: uselect <options> python <action> 

Module python:
Author: mephx.x@gmail.com Version: 0.1

Actions:
    bin 		Change Python's Version
    test		Test Python Environment


* uselect python bin

Usage: uselect <options> python bin <target> ... <target> 

    Change Python's Version

      0 - /usr/bin/python2.6 - (>)
          1 - /usr/bin/python2.6-config - (>)
          2 - /usr/bin/python2.5-config - (!)
      3 - /usr/bin/python2.5 - (!)
          4 - /usr/bin/python2.6-config - (>)
          5 - /usr/bin/python2.5-config - (!)

* uselect python bin 2

Setting /usr/bin/python2.5-config success!
Setting /usr/bin/python2.6 success

* uselect python bin (again)

Usage: uselect <options> python bin <target> ... <target> 

    Change Python's Version

      0 - /usr/bin/python2.6 - (>)
          1 - /usr/bin/python2.6-config - (!)
          2 - /usr/bin/python2.5-config - (>)
      3 - /usr/bin/python2.5 - (!)
          4 - /usr/bin/python2.6-config - (>)
          5 - /usr/bin/python2.5-config - (!)

* uselect python test

Usage: uselect <options> python test <times> [<what>] 

    Test Python Environment

    Python Test will echo <times> * <what>
    Usage can be multi-line Yay!
        		
    Usage can have dynamic options! 0
    Usage can have dynamic options! 1
    Usage can have dynamic options! 2
    Usage can have dynamic options! 3
    Usage can have dynamic options! 4

* uselect python test 10 "yeah! "

yeahyeahyeahyeahyeahyeahyeahyeahyeahyeah

I guess this is all for the past week. Symlinking dependency really
cranked my brain and almost got me stuck.

Next steps:

* add the --system option for system-wide changes
* Implement env actions
* Structure folders ".uselect/" (probabily will have a bin folder and a
env.d folder)
* Start structuring uprofile application and api
* Profiles will be xml/json (and will use this part of code to uselect
too)


* Call for ideas

        * Python gurus: Atm i'm retrieving the module from .uselect
        files this way:
        
        def get_modules(self):
        	self.modules = []
        	for module in filesystem.list_dir(modules_dir):
        		if re.match('__init__.py$', module):
        			continue
        		match = re.match('(.*).py$', module)
        		if match:
        			import modules
        			modname = match.group(1)
        			modpath = 'modules.'+ modname
        			__import__(modpath)
        			module = eval(modpath + '.module')
        			self.modules.append(module)		
        
        This is a bit ugly. Should there be a better way? I can't seem
        to achieve one. This is the reason why new modules are still
        called module.py and not module.uselect as should be.
        

        * If you are a developer of any eselect modules or
        something-config please reply with suggestions. All ideas are
        welcome.

    * Everyone else: 
        User profiles will stand in ~/.uselect
        Folder Profiles will stand in $FOLDER/.uselect
        What about group profiles?
        
        Modules like bash-completion work in a "env.d" style. Should an
        action type be created for actions like these (I believe they
        are very common).
        
        Well, all ideas are welcome again.

P.S: I have successfully passed the first midterm evaluation. Thanks to
everyone that has contributed to this new project, especially my mentor
Sébastien Fabbro (bicatali).

Cheers,
Sérgio
        
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Nirbheek Chauhan]

2009/7/23 Sérgio Almeida <mephx.x@gmail.com>:
> I'm still not doing any commits as uselect can still break your python
> environment while testing and i don't have the time to learn how to
> handle branches in git.
>

It's probably wise to commit code in small-ish (and self-containing)
discrete units each of which add something without breaking anything.
Otherwise, it becomes very difficult to track down which change broke
something via git bisect. I would recommend that you try to do this,
if only just to learn how to make good commits.

You could take a look at how the kernel folks handle this -- features
go in as several small commits/patches.

-- 
~Nirbheek Chauhan

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

Hello,

On Thu, 2009-07-23 at 08:39 +0530, Nirbheek Chauhan wrote:
> It's probably wise to commit code in small-ish (and self-containing)
> discrete units each of which add something without breaking anything.
> Otherwise, it becomes very difficult to track down which change broke
> something via git bisect. I would recommend that you try to do this,
> if only just to learn how to make good commits.
> 

Will try to do this from this state on. Starting this week.

> You could take a look at how the kernel folks handle this -- features
> go in as several small commits/patches.

Thanks, will surely look at it.



Now, a little call for help to everybody:

A child process cannot (or shouldn't be able to) change parent's process
environment.

uprofile will need to change env var's on-the-fly. For instance tag $PS1
with the current profile in use

Reliable options:

* Spawn a child shell with the chosen environment
	os.execv('/bin/bash', []) # can be whatever shell

* Wrap uprofile into a /bin/whateversh script that changes environment
acording to what child uprofile says. Ugly stuff.

Unreliable options:

* gdb/similar attach to parent and do some "call"s
* wrap watheversh into something that supports env changes from child
processes

I like the first idea the most even though it's a sub-shell. This method
kind of breaks automatic startup of uprofile for HOME.

Anyone has a clue on this? Off to sleep...

Cheers,
Sérgio 
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Nirbheek Chauhan]

2009/7/23 Sérgio Almeida <mephx.x@gmail.com>:
> A child process cannot (or shouldn't be able to) change parent's process
> environment.
>
> uprofile will need to change env var's on-the-fly. For instance tag $PS1
> with the current profile in use
>

I don't understand what use this feature has. Won't the "current
profile" be persistent across shells? If so, won't PS1 not be
persistent if you change in on-the-fly? If you don't intend to keep it
persistent, what's the use? (Actually, I don't see the use of having
it at all)


-- 
~Nirbheek Chauhan

[gentoo-dev] Re: Progress on Universal Select Tool

[Duncan]

Nirbheek Chauhan <nirbheek@gentoo.org> posted
8b4c83ad0907222009sba2c36fu59d2caf68ebcfd95@mail.gmail.com, excerpted
below, on  Thu, 23 Jul 2009 08:39:00 +0530:

> 2009/7/23 Sérgio Almeida:
>> I'm still not doing any commits as uselect can still break your python
>> environment while testing and i don't have the time to learn how to
>> handle branches in git.

> It's probably wise to commit code in small-ish (and self-containing)
> discrete units each of which add something without breaking anything.
> Otherwise, it becomes very difficult to track down which change broke
> something via git bisect. I would recommend that you try to do this, if
> only just to learn how to make good commits.
> 
> You could take a look at how the kernel folks handle this -- features go
> in as several small commits/patches.

As a non-dev direct git kernel tester and bug filer that now regularly 
bisects new bugs I find down to the individual commit, absolutely 100% 
agreed.  It's SO much easier for even a non-dev to properly test and 
bisect a properly commit unitized git source, than it was to do it 
manually, before, and the quality of my bugs and the resolutions thereof 
well demonstrate that fact.

While git branches are FWIW trivial, if you don't want to deal with them 
ATM, don't.  Just get the unitary commits right, and use tagging to 
demarcate points where you believe it's stable enough to test.  The rest 
will come, in time.  (Again, I say this as a non-dev, but even tho I'm 
not a dev and don't fully understand git myself, in git, branching 
really /is/ trivial, even for non-dev testers. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

On Thu, 2009-07-23 at 11:02 +0530, Nirbheek Chauhan wrote:
> 2009/7/23 Sérgio Almeida <mephx.x@gmail.com>:
> > A child process cannot (or shouldn't be able to) change parent's process
> > environment.
> >
> > uprofile will need to change env var's on-the-fly. For instance tag $PS1
> > with the current profile in use
> >
> 
> I don't understand what use this feature has. Won't the "current
> profile" be persistent across shells? If so, won't PS1 not be
> persistent if you change in on-the-fly? If you don't intend to keep it
> persistent, what's the use? (Actually, I don't see the use of having
> it at all)
> 
> 

Nirbheek,

You will have a persistent profile across shells and you can change it.
(This one requires sourcing). A capability of having a profile per
folder (like a folder .uselect per profiled folder)  was discussed
previously in these lists. (Actually prior to the start of SoC) I still
consider this a great feature. You changedir, you call uprofile, and
voila, new profile. You login again, default profile.

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 2008 bytes --]

On Thu, 2009-07-23 at 13:17 +0530, Arun Raghavan wrote:
> 2009/7/23 Nirbheek Chauhan <nirbheek@gentoo.org>:
> > 2009/7/23 Sérgio Almeida <mephx.x@gmail.com>:
> >> A child process cannot (or shouldn't be able to) change parent's process
> >> environment.
> >>
> >> uprofile will need to change env var's on-the-fly. For instance tag $PS1
> >> with the current profile in use
> >>
> >
> > I don't understand what use this feature has. Won't the "current
> > profile" be persistent across shells? If so, won't PS1 not be
> > persistent if you change in on-the-fly? If you don't intend to keep it
> > persistent, what's the use? (Actually, I don't see the use of having
> > it at all)
> 
> I think the point he is making is that when you update the environment
> variable, you want it to start reflecting in the current shell
> immediately. Correct me if I'm wrong.
> 

That's correct and incorrect. Let's distinguish YOUR profile from A
profile. As I explained to Nirbheek folders can have profiles and these
we want to be able to take effect immediately. 

> If this is the case, spawning a new shell might not be an option,
> since you will lose the shell's history (you'll even need to do some
> work to make sure you spawn the same shell (bash/csh/zsh) as the user
> is currently using. And there's going to be an element of surprise if
> multiple calls to uselect end up meaning that the user needs to use
> Ctrl-D/logout/exit several times to end the current session.
> 

I agree, just have in mind that it won't be uselect that needs to change
these on the fly. Uselect will notify to source your profile as eselect
does.

For uprofile we can always wrap it under a sh script and source it's
stdout after it ran. <- solution?

> IMO, it is cleaner to just print a message telling the user to source
> /etc/profile as eselect does currently.
> 
> Keep up the good work.

Thanks. Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Robert Buchholz]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

On Thursday 23 July 2009, Sérgio Almeida wrote:
> You changedir, you call uprofile, and
> voila, new profile. You login again, default profile.

Most shells have the ability to execute a command when a new prompt is 
generated. Users do not need to call uprofile themselves, they could 
set up their ~/.*shrc to do this.

For zsh (and tcsh), you can define a function:
       chpwd  Executed whenever the current working directory is
              changed.
       precmd Executed before each prompt.  Note that precommand 
              functions are not re-executed simply because the
              command line is redrawn, as happens, for example, when
              a notification about an exiting job is displayed.

For bash, you can set a variable:
       PROMPT_COMMAND If set, the value is executed as a
                      command prior to issuing each primary prompt.


You could utilize this to call uprofile, have it output environment 
variables and set them in the shell environment. A portable (bug ugly, 
code wise) way would be to do this as part of the PS1 variable.



Robert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 1667 bytes --]

On Thu, 2009-07-23 at 17:28 +0200, Robert Buchholz wrote:
> On Thursday 23 July 2009, Sérgio Almeida wrote:
> > You changedir, you call uprofile, and
> > voila, new profile. You login again, default profile.
> 
> Most shells have the ability to execute a command when a new prompt is 
> generated. Users do not need to call uprofile themselves, they could 
> set up their ~/.*shrc to do this.
> 
> For zsh (and tcsh), you can define a function:
>        chpwd  Executed whenever the current working directory is
>               changed.
>        precmd Executed before each prompt.  Note that precommand 
>               functions are not re-executed simply because the
>               command line is redrawn, as happens, for example, when
>               a notification about an exiting job is displayed.
> 
> For bash, you can set a variable:
>        PROMPT_COMMAND If set, the value is executed as a
>                       command prior to issuing each primary prompt.
> 
> 
> You could utilize this to call uprofile, have it output environment 
> variables and set them in the shell environment. A portable (bug ugly, 
> code wise) way would be to do this as part of the PS1 variable.
> 

This seems interesting... The problem would be to get a unified way of
doing this with each and every shell. Can still be done but it's ugly.

I'm shure we wouldn't want it to run on every PROMPT but surely on every
chdir. We can wrap this into a PROMTP precmd... 

if cmd = 'chdir':
  uprofile

This gets me into coding...

What do you guys think?

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Michael Haubenwallner]

Sérgio Almeida wrote:
> On Thu, 2009-07-23 at 17:28 +0200, Robert Buchholz wrote:
>> On Thursday 23 July 2009, Sérgio Almeida wrote:
>>> You changedir, you call uprofile, and
>>> voila, new profile. You login again, default profile.

..., change back to your home dir, call uprofile, and you have your
default (=login) environment.

> if cmd = 'chdir':
>   uprofile

> What do you guys think?

While the per-directory profile sounds interesting and useful (a really
good idea!), as it might solve the requirement for per-project
environment here, the automatism for the 'cd' command feels like more
confusing than useful: "WTF does 'cd' more than change directory?"

Instead, provide a command to update the environment for the current
directory, which does search for an .uprofile/ in all the parent
directories when there is no local one.
Additionally, (let the user) define a *new* command that does both
changing directory and updating the environment.

Another point: the per-directory profile solution feels like there is no
need to distinguish between user- and directory-profile any more - as
the user-profile would not be anything different than ~/.uprofile/, no?

Thank you!

/haubi/

Re: [gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 2611 bytes --]

On Fri, 2009-07-24 at 10:22 +0200, Michael Haubenwallner wrote: 
> Sérgio Almeida wrote:
> > On Thu, 2009-07-23 at 17:28 +0200, Robert Buchholz wrote:
> >> On Thursday 23 July 2009, Sérgio Almeida wrote:
> >>> You changedir, you call uprofile, and
> >>> voila, new profile. You login again, default profile.
> 
> ..., change back to your home dir, call uprofile, and you have your
> default (=login) environment.
> 

Indeed... that's what's supposed to happen. Who wants to call uprofile?
Who doesn't?

> > if cmd = 'chdir':
> >   uprofile
> 
> > What do you guys think?
> 
> While the per-directory profile sounds interesting and useful (a really
> good idea!), as it might solve the requirement for per-project
> environment here, the automatism for the 'cd' command feels like more
> confusing than useful: "WTF does 'cd' more than change directory?"
> 

Atm, cd just changes dir as it is supposed to. Robert alerted us to the
fact that we can trigger a PRE_CMD on most shells when a CHANGEDIR
occurs. 

> Instead, provide a command to update the environment for the current
> directory, which does search for an .uprofile/ in all the parent
> directories when there is no local one.
> Additionally, (let the user) define a *new* command that does both
> changing directory and updating the environment.
> 

This is the question... Call uprofile manually or detect the profile
automatically? Both capabilities? Mmm...

> Another point: the per-directory profile solution feels like there is no
> need to distinguish between user- and directory-profile any more - as
> the user-profile would not be anything different than ~/.uprofile/, no?
> 

Yes and no. ~/.uselect/ contains a bin/ environment (prepended to your
PATH by /etc/profile or something) a env.d/ and most probabily
something else that gets executed uppon login.

This does not invalidate you having a ~/.uprofile/. uprofile will
configure your ~/.uselect/ and your environment variables. Your user
profile will not be interpreted by python, uprofile turns profile files
(from python) into bin/ and env.d/ environment on your ~/.uselect.

This may seem confusing, but that's the best way I can explain. Later
this weekend will send a call for ideas/call for modules to the dev
list to get everyone known with the uselect environment. I'm just
finishing cleaning up the code to start commiting and using git
branches.

> 
> Thank you!
> 

I thank you! All! Have a nice weekend!

> 
> /haubi/
> 

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Michael Haubenwallner]

Sérgio Almeida wrote:
> On Fri, 2009-07-24 at 10:22 +0200, Michael Haubenwallner wrote: 

>>> if cmd = 'chdir':
>>>   uprofile

>> ..., the automatism for the 'cd' command feels like more
>> confusing than useful...

> Atm, cd just changes dir as it is supposed to. Robert alerted us to the
> fact that we can trigger a PRE_CMD on most shells when a CHANGEDIR
> occurs. 
 
>> Instead, provide a command to update the environment for the current
>> directory, which does search for an .uprofile/ in all the parent
>> directories when there is no local one.
>> Additionally, (let the user) define a *new* command that does both
>> changing directory and updating the environment.
>>
> 
> This is the question... Call uprofile manually or detect the profile
> automatically? Both capabilities? Mmm...

I'd say leave it to the user to either use the CHANGEDIR event,
or define some alias like 'ucd', or call 'uprofile' manually only.
Eh - provide an uselect-module to select the variant...

>> Another point: the per-directory profile solution feels like there is no
>> need to distinguish between user- and directory-profile any more - as
>> the user-profile would not be anything different than ~/.uprofile/, no?
> 
> Yes and no. ~/.uselect/ contains a bin/ environment (prepended to your
> PATH by /etc/profile or something) a env.d/ and most probabily
> something else that gets executed uppon login.
> 
> This does not invalidate you having a ~/.uprofile/. uprofile will
> configure your ~/.uselect/ and your environment variables. Your user
> profile will not be interpreted by python, uprofile turns profile files
> (from python) into bin/ and env.d/ environment on your ~/.uselect.

Ohw, there is .uselect/{bin,env.d}/ ...

What about a per-directory .uselect/{bin,env.d}/ too?
I'd like to set up the complete environment/profile for a development project,
to completely take precedence over the users' environment/profile who are
working on that project, to have something like this order:
PATH=/my/project/.uselect/bin:/home/user/.uselect/bin:/etc/.uselect/bin:/usr/bin:/bin

Maybe this already is how it works?

> 
> This may seem confusing, but that's the best way I can explain. Later
> this weekend will send a call for ideas/call for modules to the dev
> list to get everyone known with the uselect environment. I'm just
> finishing cleaning up the code to start commiting and using git
> branches.

Whenever I come to test uselect/uprofile, I'll do this in Gentoo Prefix
on several platforms, not on my stable Gentoo Desktop...

Thank you!

/haubi/

Re: [gentoo-dev] Re: [gentoo-soc] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

On Mon, 2009-07-27 at 10:33 +0200, Michael Haubenwallner wrote: 
> 
> I'd say leave it to the user to either use the CHANGEDIR event,
> or define some alias like 'ucd', or call 'uprofile' manually only.
> Eh - provide an uselect-module to select the variant...
> 

Never though of that... Nice idea!

> 
> Maybe this already is how it works?
> 

It's how it "will" work =) atm only /bin works and per-user (not yet per
folder)

> Whenever I come to test uselect/uprofile, I'll do this in Gentoo Prefix
> on several platforms, not on my stable Gentoo Desktop...
> 

Just did a commit this afternoon.

Feel free to try it out. I haven't implemented any modules besides my
test module "python" and it's most probably buggy because of
python-config dependency. sh install.sh should do the work.

http://git.overlays.gentoo.org/gitweb/?p=proj/uselect.git;a=summary

I'm starting profiles implementation today and will have more active
commits.

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 3567 bytes --]

Hello all,

Once again, another productive week. This week's work focused mostly on
profiles. Uprofile's implementation exceeded my expectations and was
much easier than what I thought it would be, the reason: uselect's
simple api.

What's working on uprofile:

* uprofile when called with no arguments, finds current folder profile.
If not present, it lists available profiles.

** Available profiles are profiles other than './uprofile/folder.json'

* Profiles are written in json 

** Why json and not xml? Well, whoever said that xml is human-readable
should reconsider the clause. In need of a markup language, easy bridge
between python and markup, easily human-readable, I have chosen json.

Example:

{"profile": {
	"description": "Sample Profile",
	"author": "mephx.x@gmail.com",
	"version": "0.1",
	"modules": {
		"python": {
			"actions": {
				"bin": [
					"/usr/bin/python2.6",
					"/usr/bin/python2.6-config"
				]
			}
		}
	}
}}

In this profile, uprofile would call module python, action bin with
args: python2.6 and python2.6-config. 

uprofile when called:

mephx - profiledfolder $ uprofile
Setting Folder Profile...
Setting /usr/bin/python2.6 success!
Setting /usr/bin/python2.6-config success!
Folder Profile Set!

* Profiles read modules (from uselect's module dir), actions from
modules, and call's Action.do_action() method with the specified args
list.
** In order to profiles to work properly, arguments cannot be Integers
as in uselect/eselect. All modules's actions which feature profiling
capabilities, need to accept either indexed values, either absolute
string values for specific selection. Why? Example: New python bin
appears in /usr/bin, indexes change, profile gets broken.

* Decided to keep a bin/ and a env.d/ in each .uprofile directory. These
are updated as they normally would via uselect.

* Automatic profile changing in bash can be done via a specially crafted
PROMPT_COMMAND. I'm using this one now:

PROMPT_COMMAND="test -e $HOME/.uselect/env.d/ &&
PROFILE='$HOME/.uselect' ; test -e .uprofile/env.d/ &&
PROFILE='./.uprofile' ; source \$PROFILE/env.d/*"

** This actually changes the profile quite fast and reflects the changes
on $PS1 with the folder name, neat =)

mephx - ~/ $ cd profiledfolder
[folder] mephx - profiledfolder $

** To generate .uprofile/ directory, uprofile needs to be called by
hand. Sourcing env.d/ automatically also updates the user's PATH to that
bin DIR (this is still not implemented) therefore not needing to call
uprofile every time you wish to activate the profile. 


Next steps:

* Finish implementing env actions. (It's now much funnier to test env
actions using profiles)
* Implement uprofile module for uselect as suggested.
* Implement some more modules.

What do you think?

* json modules?
* profile constraints (this is basically adding if's to profiles if we
want profiles to behave differently on certain conditions (hostname,
arch, etc...)

During the next week, I will deploy a properly packed version for
testing. I will also launch a call for modules to every *-eselect dev
and *-config dev as I do not have the time to implement most of the
modules just for testing purposes. Most modules are very easy to do
(symlinking ones) and conversion of eselect to uselect can be done
instantly, yet in an ugly way of still using all eselect's libs.

Keep tuned for more. And stay in school =)

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Progress on Universal Select Tool

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 2919 bytes --]

Hello,

Regardless that firm pencil's down date has come for this year's GSoC as
discussed with my mentor I will continue coding. These final weeks will
be split between coding, documentation, sample module creation and
polishing. Also a live ebuild creation is in progress.

Comparing uselect with eselect:

* uselect is much faster (even with lots of modules)
* uselect can already use (in a very ugly way) all eselect's modules
with a copy/paste step. It's still faster in this way.
* uselect menu is much more user friendly even though it's based on
eselect/ecletic.
* uselect has a general user-profile schema (using a ~/.uselect/
folder).
* uselect does not have a base function libraries as eselect does.
uselect objective was to create a simple framework that needs no logic
coding within modules.
* uselect can be easily expanded with more action classes with not much
coding as we are using an object based language with great inheritance
features.

Uselect is far from ready, but it's current state (besides usable) is
"presentable". What do I mean by presentable?

Uselect and Uprofile are a Proof of Concept of what both can be. I was
expecting better interaction and belief in the Universal Select Tool
project during all SoC. As most ideas/suggestions were purely based on
necessity and not on community-wide contribution as suggestion for new
action types templates a very solid and flexible framework was not
achieved. A solid example (symlinking modules) is fully supported either
user-wide and system-wide. A fine structure for .uselect folders
and .uprofile folders was not strictly defined as it is a very
decisional matter for general gentoo usability. Automatic module
creation during emerge is not possible without major changes to ebuilds
due to the lack of a list of "to-be-installed" files (there are ugly
workarounds, but none are worth implementing). Module's API is very
simple and can be easily changed from plain python to json or a similar
markup language. Uprofile (as current concept was not in my initial
proposal) is strictly a proof of concept of the interaction beween
uprofile and uselect's API. Once documentation and examples are done
uselect and uprofile will be ready to be presented to the gentoo
community and to be discussed within gentoo's decision organs.

It's been great to be working with gentoo. I hope that this tool gets
integrated in gentoo and I am willing to continue developing it after
SoC's end.

Thanks to everyone that helped, especially my mentor bicatali, people at
gentoo-dev@ and gentoo-soc@ and a few unknown folks at #python that
helped me solve complex python class mutation problems and recursiveness
paradoxes (this was the real deal).

Will also continue posting weekly reports on gentoo-soc@ until soc ends.

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]