[gentoo-dev] reworking python-wrapper and wrapper scripts.

[gentoo-dev] reworking python-wrapper and wrapper scripts.

[Brian Harring]

If folks haven't looked at python_generate_wrapper_scripts in 
python.eclass, I'd suggest doing so.  For examples of it's usage, grep 
for 'python_generate_wrapper_scripts' in /usr/bin/; any place you see 
it, look for <that-script-name>-${PYTHON_TARGETS} (for example, 
/usr/bin/sphinx-build{,-2.7,-3.2}.

Each usage there is a separate custom script for that specific binary; 
if there is a bug in the script, well, we're screwed- requires 
re-merging the package.

This setup, at least on my hardware, is .04s added to every 
invocation; this is ignoring the inode cost for each, and the issue if 
a bug ever appears in the script generation code (in which case we're 
screwed- would require re-merging the package).

In parallel, we've got python-wrapper (ls /usr/bin/python -l); this is 
provided by eselect-python and basically discern what the active 
python version is, and use that in the absense of any directives.  
This is implemented in C, and is reasonably sane; the overhead for 
that is basically non-existant.

Roughly, I'm proposing we do away with python eclass's 
generate_python_wrapper_scripts generation of a script, instead having 
that just symlink to a binary provided by eselect-python that handles 
this.  This centralizes the implementation (fix in one spot), and 
would allow a c version to be used- basically eliminating the 
overhead.


There's a trick to this; currently, those generated scripts hardcode 
the allowed/known python versions for that package.  We obviously have 
to preserve that; I propose we shove it into the symlink path.

Basically, we add a /usr/libexec/python directory; within it, we have 
a wrapper binary (explained below), and a set of symlinks pointing at 
the root of that directory.  To cover our current python versions, the 
following would suffice:

for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8} 
\2.7-pypy-1.9; do
  ln -s ./ /usr/libexec/python/$x 
done

While that seems insane, there is a reason; via that, we can encode 
the allowed versions into the symlink.  Using pkgcore's pquery for 
example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3) 
instead of a wrapper script at /usr/bin/pquery, we'd have thus:

targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
targets=$(IFS=/;echo -n "${targets[*]}")
# This results in
# targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
ln -s "/usr/libexec/python/${targets}/wrapper" \
  /usr/bin/pquery

/usr/libexec/python/wrapper upon invocation, takes a look at argv[0]; 
sees how it was invoked basically.  This will be the /usr/bin/whatever 
pathway.  It reads the symlink, in the process getting the allowed 
versions and preferred order of the versions.

Few notes; vast majority of filesystems will store the symlink target 
into the core inode if at all possible- in doing so, this avoids 
wasting an inode and is only limited by the length of the target.  
That length is capped by PATH_MAX- which can range from 256 to 4k (or 
higher).

For the pquery example above, that comes out to ~73 bytes for the 
symlink pathway; well under PATH_MAX.

For the scenarios where PATH_MAX caps the symlink pathway, or for 
whatever reason we don't want to use that trick, a tree of files 
contained within /usr/libexec/python/ holding the allowed versions for 
the matching pathway would suffice.

Either proposal here would be far faster than what we've got now; also 
will use less space (ancillary benefit).

One subtle aspect here is that if we did this, it makes it possible to 
avoid losing the invocation information- currently if you did 
`/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things 
are implemented now (specifically the two layers of wrappers)- you'll 
get python2.7 running that sphinx-build invocation.

This is wrong (it's directly discarding what the invocation 
requested), although you're only going to see it for scripts that 
do python introspection.

Via doing the restructuring I'm mentioning above, that issue can be 
fixed, while making things faster/saner.

On a related note; we currently install multiple versions of the same 
script- the only difference being the shebang.  If one ignores the 
shebang, in some cases this is necessary- where the script is 2to3 
translated, and the code for py2k vs py3k differs.  For most, the only 
difference is in the shebang however.

While it's minor in space savings, it's possible to eliminate that 
redundancy via a shebang target that looks at the pathway it was 
invoked via.  Fairly easy actually, and basically zero overhead if 
done.

Either way, thoughts?

What I'm proposing isn't perfect, but I'm of the view it's a step up 
from what's in place now- and via centralizing this crap, makes it 
easier to change/maintain this going forward as necessary.
~harring

[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.

[Brian Harring]

On Thu, Oct 18, 2012 at 12:00:43PM +0200, Dirkjan Ochtman wrote:
> On Thu, Oct 18, 2012 at 11:15 AM, Brian Harring <ferringb@gmail.com> wrote:
> > Either way, thoughts?
> 
> It looks like you haven't looked at the python-r1 effort. That means
> you probably also aren't subscribed to the gentoo-python mailing list.
> I suggest you do both first. :)

Yeah, I'm not on the ML- as for the eclass, I read the source; 
nothing in there in re: to wrappers currently.

Only thread I see that is relevant is 
http://article.gmane.org/gmane.linux.gentoo.python/11 ; roughly 
executable wrapper, just what I'm spelling out here is the 
exact mechanism's of how to do it.

Re: python-r1; frankly... if folks are fine w/ the wrapper, I don't 
want it delayed waiting on python-r1 (or only used there).

As I laid out, what I'm proposing can be dropped in *now* for existing 
python eclass- basically moves that logic out of it and into 
eselect-python.

It'll be faster/saner to maintain, less code in python eclass, and 
reusable going forward for the new attempts; plus I'm willing to do it 
(meaning it'll get done in the next week or two).

That said; +1 or -1, or some where in between?

~harring

Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 5873 bytes --]

On Thu, 2012-10-18 at 02:15 -0700, Brian Harring wrote:
> If folks haven't looked at python_generate_wrapper_scripts in 
> python.eclass, I'd suggest doing so.  For examples of it's usage, grep 
> for 'python_generate_wrapper_scripts' in /usr/bin/; any place you see 
> it, look for <that-script-name>-${PYTHON_TARGETS} (for example, 
> /usr/bin/sphinx-build{,-2.7,-3.2}.
> 
> Each usage there is a separate custom script for that specific binary; 
> if there is a bug in the script, well, we're screwed- requires 
> re-merging the package.
> 
> This setup, at least on my hardware, is .04s added to every 
> invocation; this is ignoring the inode cost for each, and the issue if 
> a bug ever appears in the script generation code (in which case we're 
> screwed- would require re-merging the package).
> 
> In parallel, we've got python-wrapper (ls /usr/bin/python -l); this is 
> provided by eselect-python and basically discern what the active 
> python version is, and use that in the absense of any directives.  
> This is implemented in C, and is reasonably sane; the overhead for 
> that is basically non-existant.
> 
> Roughly, I'm proposing we do away with python eclass's 
> generate_python_wrapper_scripts generation of a script, instead having 
> that just symlink to a binary provided by eselect-python that handles 
> this.  This centralizes the implementation (fix in one spot), and 
> would allow a c version to be used- basically eliminating the 
> overhead.
> 
> 
> There's a trick to this; currently, those generated scripts hardcode 
> the allowed/known python versions for that package.  We obviously have 
> to preserve that; I propose we shove it into the symlink path.
> 
> Basically, we add a /usr/libexec/python directory; within it, we have 
> a wrapper binary (explained below), and a set of symlinks pointing at 
> the root of that directory.  To cover our current python versions, the 
> following would suffice:
> 
> for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8} 
> \2.7-pypy-1.9; do
>   ln -s ./ /usr/libexec/python/$x 
> done
> 
> While that seems insane, there is a reason; via that, we can encode 
> the allowed versions into the symlink.  Using pkgcore's pquery for 
> example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3) 
> instead of a wrapper script at /usr/bin/pquery, we'd have thus:
> 
> targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
> targets=$(IFS=/;echo -n "${targets[*]}")
> # This results in
> # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
> ln -s "/usr/libexec/python/${targets}/wrapper" \
>   /usr/bin/pquery
> 
> /usr/libexec/python/wrapper upon invocation, takes a look at argv[0]; 
> sees how it was invoked basically.  This will be the /usr/bin/whatever 
> pathway.  It reads the symlink, in the process getting the allowed 
> versions and preferred order of the versions.
> 
> Few notes; vast majority of filesystems will store the symlink target 
> into the core inode if at all possible- in doing so, this avoids 
> wasting an inode and is only limited by the length of the target.  
> That length is capped by PATH_MAX- which can range from 256 to 4k (or 
> higher).
> 
> For the pquery example above, that comes out to ~73 bytes for the 
> symlink pathway; well under PATH_MAX.
> 
> For the scenarios where PATH_MAX caps the symlink pathway, or for 
> whatever reason we don't want to use that trick, a tree of files 
> contained within /usr/libexec/python/ holding the allowed versions for 
> the matching pathway would suffice.
> 
> Either proposal here would be far faster than what we've got now; also 
> will use less space (ancillary benefit).
> 
> One subtle aspect here is that if we did this, it makes it possible to 
> avoid losing the invocation information- currently if you did 
> `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things 
> are implemented now (specifically the two layers of wrappers)- you'll 
> get python2.7 running that sphinx-build invocation.
> 
> This is wrong (it's directly discarding what the invocation 
> requested), although you're only going to see it for scripts that 
> do python introspection.
> 
> Via doing the restructuring I'm mentioning above, that issue can be 
> fixed, while making things faster/saner.
> 
> On a related note; we currently install multiple versions of the same 
> script- the only difference being the shebang.  If one ignores the 
> shebang, in some cases this is necessary- where the script is 2to3 
> translated, and the code for py2k vs py3k differs.  For most, the only 
> difference is in the shebang however.
> 

What if the invoking script is not needed to be 2to3 translated (super
minimal python code) but the remaining python libs need to be?

> While it's minor in space savings, it's possible to eliminate that 
> redundancy via a shebang target that looks at the pathway it was 
> invoked via.  Fairly easy actually, and basically zero overhead if 
> done.
> 
> Either way, thoughts?
> 
> What I'm proposing isn't perfect, but I'm of the view it's a step up 
> from what's in place now- and via centralizing this crap, makes it 
> easier to change/maintain this going forward as necessary.
> ~harring
> 

+1 from me.

Eclean has been checking the name it was invoked with long before I did
the major re-write.  From that it cleans either distfiles or packages if
invoked by either the eclean-dist or eclean-pkg symlinks.  If invoked by
eclean itself then it looks for the target in the arguments.  So Brian's
proposal is not something totally new, never been done before...

While this proposes something a little different.  It is still very much
along the same line and in my opinion a much better solution.
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.

[Brian Harring]

On Thu, Oct 18, 2012 at 08:02:42AM -0700, Brian Dolbec wrote:
> On Thu, 2012-10-18 at 02:15 -0700, Brian Harring wrote:
> > On a related note; we currently install multiple versions of the same 
> > script- the only difference being the shebang.  If one ignores the 
> > shebang, in some cases this is necessary- where the script is 2to3 
> > translated, and the code for py2k vs py3k differs.  For most, the only 
> > difference is in the shebang however.
> > 
> 
> What if the invoking script is not needed to be 2to3 translated (super
> minimal python code) but the remaining python libs need to be?

Then we dedup 'em, using a shebang that knows to look at argv[0]'s 
last few chars to find the appropriate python binary to invoke via.

As for the installation- either hardlinked'd scripts, or symlinked; 
doesn't really matter, although hardlink gets my vote (so 
sphinx-build-2.7 would be a hardlink to the same inode as 
sphinx-build-3.2).

> > While it's minor in space savings, it's possible to eliminate that 
> > redundancy via a shebang target that looks at the pathway it was 
> > invoked via.  Fairly easy actually, and basically zero overhead if 
> > done.
> > 
> > Either way, thoughts?
> > 
> > What I'm proposing isn't perfect, but I'm of the view it's a step up 
> > from what's in place now- and via centralizing this crap, makes it 
> > easier to change/maintain this going forward as necessary.
> > ~harring
> > 
> 
> +1 from me.
> 
> Eclean has been checking the name it was invoked with long before I did
> the major re-write.  From that it cleans either distfiles or packages if
> invoked by either the eclean-dist or eclean-pkg symlinks.  If invoked by
> eclean itself then it looks for the target in the arguments.  So Brian's
> proposal is not something totally new, never been done before...
>
> While this proposes something a little different.  It is still very much
> along the same line and in my opinion a much better solution.

Regarding "having the script check on it's own"... yeah, this has been 
on pkgcore's todo for a while, but basically got blocked by the 
current wrapper bits.  I expect once this wrapper is in place, I may 
screw with the setup a bit to find a way so that scripts that can 
handle this on their own, are left to handle this on their own.

Any other +1's/-1's?  Else I may do this on the weekend for 
destress-hacking time...

~harring

[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.

[Mike Gilbert]

On Thu, Oct 18, 2012 at 5:15 AM, Brian Harring <ferringb@gmail.com> wrote:
> If folks haven't looked at python_generate_wrapper_scripts in
> python.eclass, I'd suggest doing so.  For examples of it's usage, grep
> for 'python_generate_wrapper_scripts' in /usr/bin/; any place you see
> it, look for <that-script-name>-${PYTHON_TARGETS} (for example,
> /usr/bin/sphinx-build{,-2.7,-3.2}.
>
> Each usage there is a separate custom script for that specific binary;
> if there is a bug in the script, well, we're screwed- requires
> re-merging the package.
>
> This setup, at least on my hardware, is .04s added to every
> invocation; this is ignoring the inode cost for each, and the issue if
> a bug ever appears in the script generation code (in which case we're
> screwed- would require re-merging the package).
>
> In parallel, we've got python-wrapper (ls /usr/bin/python -l); this is
> provided by eselect-python and basically discern what the active
> python version is, and use that in the absense of any directives.
> This is implemented in C, and is reasonably sane; the overhead for
> that is basically non-existant.
>
> Roughly, I'm proposing we do away with python eclass's
> generate_python_wrapper_scripts generation of a script, instead having
> that just symlink to a binary provided by eselect-python that handles
> this.  This centralizes the implementation (fix in one spot), and
> would allow a c version to be used- basically eliminating the
> overhead.
>
>
> There's a trick to this; currently, those generated scripts hardcode
> the allowed/known python versions for that package.  We obviously have
> to preserve that; I propose we shove it into the symlink path.
>
> Basically, we add a /usr/libexec/python directory; within it, we have
> a wrapper binary (explained below), and a set of symlinks pointing at
> the root of that directory.  To cover our current python versions, the
> following would suffice:
>
> for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8}
> \2.7-pypy-1.9; do
>   ln -s ./ /usr/libexec/python/$x
> done
>
> While that seems insane, there is a reason; via that, we can encode
> the allowed versions into the symlink.  Using pkgcore's pquery for
> example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3)
> instead of a wrapper script at /usr/bin/pquery, we'd have thus:
>
> targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
> targets=$(IFS=/;echo -n "${targets[*]}")
> # This results in
> # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
> ln -s "/usr/libexec/python/${targets}/wrapper" \
>   /usr/bin/pquery
>
> /usr/libexec/python/wrapper upon invocation, takes a look at argv[0];
> sees how it was invoked basically.  This will be the /usr/bin/whatever
> pathway.  It reads the symlink, in the process getting the allowed
> versions and preferred order of the versions.
>
> Few notes; vast majority of filesystems will store the symlink target
> into the core inode if at all possible- in doing so, this avoids
> wasting an inode and is only limited by the length of the target.
> That length is capped by PATH_MAX- which can range from 256 to 4k (or
> higher).
>
> For the pquery example above, that comes out to ~73 bytes for the
> symlink pathway; well under PATH_MAX.
>
> For the scenarios where PATH_MAX caps the symlink pathway, or for
> whatever reason we don't want to use that trick, a tree of files
> contained within /usr/libexec/python/ holding the allowed versions for
> the matching pathway would suffice.
>
> Either proposal here would be far faster than what we've got now; also
> will use less space (ancillary benefit).
>
> One subtle aspect here is that if we did this, it makes it possible to
> avoid losing the invocation information- currently if you did
> `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things
> are implemented now (specifically the two layers of wrappers)- you'll
> get python2.7 running that sphinx-build invocation.
>
> This is wrong (it's directly discarding what the invocation
> requested), although you're only going to see it for scripts that
> do python introspection.
>
> Via doing the restructuring I'm mentioning above, that issue can be
> fixed, while making things faster/saner.
>
> On a related note; we currently install multiple versions of the same
> script- the only difference being the shebang.  If one ignores the
> shebang, in some cases this is necessary- where the script is 2to3
> translated, and the code for py2k vs py3k differs.  For most, the only
> difference is in the shebang however.
>
> While it's minor in space savings, it's possible to eliminate that
> redundancy via a shebang target that looks at the pathway it was
> invoked via.  Fairly easy actually, and basically zero overhead if
> done.
>
> Either way, thoughts?
>
> What I'm proposing isn't perfect, but I'm of the view it's a step up
> from what's in place now- and via centralizing this crap, makes it
> easier to change/maintain this going forward as necessary.
> ~harring

If we are somehow going to eliminate the installation of a separate
script for each python version, then the symlink idea sounds like a
good solution for expressing the supported python versions. However,
I'm really not sure how you would eliminate the separate scripts, in
light of the 2to3 issue.

Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example:
invoking python on a binary (or a symlink to a binary) isn't going to
work at all. So I don't see how you've solved that issue.

Back to the discussion djc pointed out: mgorny has implemented a
binary solution already, called "python-exec", in conjunction with
python-r1.eclass.

https://bitbucket.org/mgorny/python-exec

python-exec assumes there will be a separate script for each version,
and just calls exec several times. In shell code, it would look like
this:

scriptname=$0 # argv[0]
exec scriptname-${EPYTHON}
exec scriptname-$(< /etc/env.d/python/config)
for x in python2.7 python2.6 python3.2 python3.1 ... ; do
  exec scriptname-${x}
done

The list of implementations for the loop at the end is hard-coded when
python-exec is built. This is the weak point in the solution. It could
be improved with your symlink idea.

What are your thoughts on mgorny's python-exec solution?

We could adapt it for python.eclass if you really want to support
that. Probably by modifying python.eclass to install scripts like
sphinx-python2.7 instead of sphinx-2.7.

[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.

[Brian Harring]

On Thu, Oct 18, 2012 at 11:28:59AM -0400, Mike Gilbert wrote:
> On Thu, Oct 18, 2012 at 5:15 AM, Brian Harring <ferringb@gmail.com> wrote:
> If we are somehow going to eliminate the installation of a separate
> script for each python version, then the symlink idea sounds like a
> good solution for expressing the supported python versions. However,
> I'm really not sure how you would eliminate the separate scripts, in
> light of the 2to3 issue.

The elimination is just a dedup run; track the PYTHON_ABI script's 
being installed- check the md5 (minus the shebang).  If it's the same, 
they can be hardlinked together with the appropriate shebang wrapper.

It's doable, is the short version.  Matter of fact, the dedupe run 
there could be extended to do deduplication in general across python 
versions, but that's an optimization for further down the line.


> Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example:
> invoking python on a binary (or a symlink to a binary) isn't going to
> work at all. So I don't see how you've solved that issue.

Kindly provide reasoning/justification for such a claim in the future- 
saves a round of email ping/ponging, and saves me getting pissy 
towards people if they persist in it. :)

Lecture aside, "nuh uh, it does work!" is appropriate.  I already 
hacked up a prototype locally that works fine and does this via an 
env passing hack- w/ what was proposed, this can be done w/out those 
hacks, and would be faster.


> Back to the discussion djc pointed out: mgorny has implemented a
> binary solution already, called "python-exec", in conjunction with
> python-r1.eclass.
> 
> https://bitbucket.org/mgorny/python-exec

Another one I didn't see in the ML emails... joy.  This is partial.  
Finishing this to what I'm proposing above isn't too hard to do 
however- that said, I strongly believe this doesn't belong in a 
separate repo/package, this should be shoved into eselect-python.

eselect-python already gives us python-wrapper after all; no point 
in having two layers of wrappers, and content in two repos.


> python-exec assumes there will be a separate script for each version,
> and just calls exec several times. In shell code, it would look like
> this:
> 
> scriptname=$0 # argv[0]
> exec scriptname-${EPYTHON}
> exec scriptname-$(< /etc/env.d/python/config)
> for x in python2.7 python2.6 python3.2 python3.1 ... ; do
>   exec scriptname-${x}
> done
> 
> The list of implementations for the loop at the end is hard-coded when
> python-exec is built. This is the weak point in the solution. It could
> be improved with your symlink idea.
> 
> What are your thoughts on mgorny's python-exec solution?

Code's a bit raw, but as I mentioned in my other email, it's on a 
similar path to what I'm proposing- same core idea, just specifics in 
mine + the symlink trick.


> We could adapt it for python.eclass if you really want to support
> that. Probably by modifying python.eclass to install scripts like
> sphinx-python2.7 instead of sphinx-2.7.

Honestly, w/ what I'm intending, we don't need to change the python 
eclass much- we can preserve the same naming in use if desired.  If a 
rename is desired, sure, do it when this is phased in.

That said if there isn't a reason for the rename, then don't do it, 
imo.

~harring

[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.

[Mike Gilbert]

On Thu, Oct 18, 2012 at 3:29 PM, Brian Harring <ferringb@gmail.com> wrote:
> On Thu, Oct 18, 2012 at 11:28:59AM -0400, Mike Gilbert wrote:
>> Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example:
>> invoking python on a binary (or a symlink to a binary) isn't going to
>> work at all. So I don't see how you've solved that issue.
>
> Kindly provide reasoning/justification for such a claim in the future-
> saves a round of email ping/ponging, and saves me getting pissy
> towards people if they persist in it. :)
>
> Lecture aside, "nuh uh, it does work!" is appropriate.  I already
> hacked up a prototype locally that works fine and does this via an
> env passing hack- w/ what was proposed, this can be done w/out those
> hacks, and would be faster.
>

When you invoke a command like:

/usr/bin/python3.2 /usr/bin/somebinary

The python3.2 program assumes somebinary is a python script and tries
to parse it. Since somebinary is actually an ELF object, python bombs
out immediately.

Am I missing something fundamental here?

[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.

[Mike Gilbert]

On Thu, Oct 18, 2012 at 4:50 PM, Mike Gilbert <floppym@gentoo.org> wrote:
> On Thu, Oct 18, 2012 at 3:29 PM, Brian Harring <ferringb@gmail.com> wrote:
>> On Thu, Oct 18, 2012 at 11:28:59AM -0400, Mike Gilbert wrote:
>>> Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example:
>>> invoking python on a binary (or a symlink to a binary) isn't going to
>>> work at all. So I don't see how you've solved that issue.
>>
>> Kindly provide reasoning/justification for such a claim in the future-
>> saves a round of email ping/ponging, and saves me getting pissy
>> towards people if they persist in it. :)
>>
>> Lecture aside, "nuh uh, it does work!" is appropriate.  I already
>> hacked up a prototype locally that works fine and does this via an
>> env passing hack- w/ what was proposed, this can be done w/out those
>> hacks, and would be faster.
>>
>
> When you invoke a command like:
>
> /usr/bin/python3.2 /usr/bin/somebinary
>
> The python3.2 program assumes somebinary is a python script and tries
> to parse it. Since somebinary is actually an ELF object, python bombs
> out immediately.
>
> Am I missing something fundamental here?

Oh, perhaps you are suggesting that /usr/bin/python3.2 would no longer
be the actual python binary, but instead be a link to your wrapper
program? It's really not clear to me.

[gentoo-dev] Re: reworking python-wrapper and wrapper scripts.

[Brian Harring]

On Thu, Oct 18, 2012 at 04:50:04PM -0400, Mike Gilbert wrote:
> On Thu, Oct 18, 2012 at 3:29 PM, Brian Harring <ferringb@gmail.com> wrote:
> > On Thu, Oct 18, 2012 at 11:28:59AM -0400, Mike Gilbert wrote:
> >> Regarding your /usr/bin/python3.2 /usr/bin/sphinx-build example:
> >> invoking python on a binary (or a symlink to a binary) isn't going to
> >> work at all. So I don't see how you've solved that issue.
> >
> > Kindly provide reasoning/justification for such a claim in the future-
> > saves a round of email ping/ponging, and saves me getting pissy
> > towards people if they persist in it. :)
> >
> > Lecture aside, "nuh uh, it does work!" is appropriate.  I already
> > hacked up a prototype locally that works fine and does this via an
> > env passing hack- w/ what was proposed, this can be done w/out those
> > hacks, and would be faster.
> >
> 
> When you invoke a command like:
> 
> /usr/bin/python3.2 /usr/bin/somebinary
> 
> The python3.2 program assumes somebinary is a python script and tries
> to parse it. Since somebinary is actually an ELF object, python bombs
> out immediately.
> 
> Am I missing something fundamental here?

Nope, you've got me embarassingly, absolutely fricking dead to rights 
on that one; I overlooked at the prototype I did was a python script 
for 'somebinary', which worked via the env passing.

Would've spotted it upon implementing it in c to say the least, but 
still, *cough*. ;)

As you mentioned in a separate email; if we stick a wrapper in front 
of the python binaries, that would fly; I don't much like that 
however.

Offhand... and this makes it a bit more complex, but still tenuable 
imo, but we could get around this via shoving the symlink pathway into 
the shebang itself.

scenario 1:
Script gets installed for 2.7, 3.1, 3.2; ignoring the shebang, it's 
exactly the same md5 (meaning we can dedup it).  The shebang instead 
of being /usr/bin/python-whatever, instead is the symlink pathway I'd 
referenced.  Meaning it would be 
/usr/libexec/python/2.7/3.1/3.2/wrapper .  If that script was 
sphinx-build, this means that sphinx-build{,-{2.7,3.1,3.2}} are all 
valid python scripts- they can be invoked either via /path/to/script 
or ${python} /path/to/script.  There's a few details to sort there, 
but nothing egregious best I can tell- implicit in this proposal is 
that the wrapper would be smart enough to tell in which context it's 
being invoked (sphinx-build vs sphinx-build-2.7).

Scenario 2:
2.7, 3.1, 3.2 are in play; they do /not/ dedupe to the same content, 
meaning we actually need indirection in 'sphinx-build' (but not 
sphinx-build-{2.7,3.1,3.2} although we would dedupe where possible 
mind you).

In that case, to support python3.2 sphinx-build, we'd need a python 
script that does the searching/re-exec for the wrapper.  This can be 
standardized- just have it search for sphinx-build-${PYTHON_ABIS} to 
identify the installed versions, then do the usual active 
version/epython trick to decide which to use.  This honestly isn't too 
far off from what we have now.

We can however optimize that; same trick, for the wrapper the shebang 
isn't /usr/bin/python; it's just a c version of the python script, 
done in c for speed.

So... caveats.  If we screw w/ the shebang like this, file isn't going 
to like it- it'll fail to identify certain scripts as python.  That 
said, it's worth keeping in mind that if we use the symlink route, a 
file /usr/bin/sphinx-build would result in ELF, vs file 
/usr/bin/sphinx-build-2.7 being python.

This can be fixed via an additional rule added to the magic db.  
Pretty straight forward, and eselect-python (once it gets these 
wrappers) could level an appropriate dep to ensure that's sorted.

So... my preference is shebang; it's basically an extension of the 
symlink trick I'd originally mentioned, just cleaner imo.

What are folks thoughts on that?
~harring

Re: [gentoo-dev] Re: reworking python-wrapper and wrapper scripts.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

On Thu, 18 Oct 2012 15:37:13 -0700
Brian Harring <ferringb@gmail.com> wrote:

> Offhand... and this makes it a bit more complex, but still tenuable 
> imo, but we could get around this via shoving the symlink pathway into 
> the shebang itself.
> 
> scenario 1:
> Script gets installed for 2.7, 3.1, 3.2; ignoring the shebang, it's 
> exactly the same md5 (meaning we can dedup it).  The shebang instead 
> of being /usr/bin/python-whatever, instead is the symlink pathway I'd 
> referenced.  Meaning it would be 
> /usr/libexec/python/2.7/3.1/3.2/wrapper .  If that script was 
> sphinx-build, this means that sphinx-build{,-{2.7,3.1,3.2}} are all 
> valid python scripts- they can be invoked either via /path/to/script 
> or ${python} /path/to/script.  There's a few details to sort there, 
> but nothing egregious best I can tell- implicit in this proposal is 
> that the wrapper would be smart enough to tell in which context it's 
> being invoked (sphinx-build vs sphinx-build-2.7).

How can you distinguish (from a Python script) whether it was run
directly or passed to pythonX.Y?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4608 bytes --]

On Thu, 18 Oct 2012 02:15:43 -0700
Brian Harring <ferringb@gmail.com> wrote:

> There's a trick to this; currently, those generated scripts hardcode 
> the allowed/known python versions for that package.  We obviously have 
> to preserve that; I propose we shove it into the symlink path.
> 
> Basically, we add a /usr/libexec/python directory; within it, we have 
> a wrapper binary (explained below), and a set of symlinks pointing at 
> the root of that directory.  To cover our current python versions, the 
> following would suffice:
> 
> for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8} 
> \2.7-pypy-1.9; do
>   ln -s ./ /usr/libexec/python/$x 
> done
> 
> While that seems insane, there is a reason; via that, we can encode 
> the allowed versions into the symlink.  Using pkgcore's pquery for 
> example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3) 
> instead of a wrapper script at /usr/bin/pquery, we'd have thus:
> 
> targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
> targets=$(IFS=/;echo -n "${targets[*]}")
> # This results in
> # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
> ln -s "/usr/libexec/python/${targets}/wrapper" \
>   /usr/bin/pquery
> 
> /usr/libexec/python/wrapper upon invocation, takes a look at argv[0]; 
> sees how it was invoked basically.  This will be the /usr/bin/whatever 
> pathway.  It reads the symlink, in the process getting the allowed 
> versions and preferred order of the versions.
> 
> Few notes; vast majority of filesystems will store the symlink target 
> into the core inode if at all possible- in doing so, this avoids 
> wasting an inode and is only limited by the length of the target.  
> That length is capped by PATH_MAX- which can range from 256 to 4k (or 
> higher).
> 
> For the pquery example above, that comes out to ~73 bytes for the 
> symlink pathway; well under PATH_MAX.
> 
> For the scenarios where PATH_MAX caps the symlink pathway, or for 
> whatever reason we don't want to use that trick, a tree of files 
> contained within /usr/libexec/python/ holding the allowed versions for 
> the matching pathway would suffice.

While I agree that it's a clever trick, I doubt it's worth the effort.
Did you got any numbers proving it being superior over, say, trying to
exec() scripts like I do in python-exec?

While I can imagine, that in an worst case that bunch of exec()s is
going to be definitely slower than storing the list anyway, I doubt
such a bad case is often.

Considering that the most common Python version used now is Python 2,
how often doesn't the script support that Python version? That's a very
rare case, and often just executing "foo-${EPYTHON}" works. In your
case, that common case involves readlink() + parsing + exec().

Even in case of Python 3 being selected, I doubt the overhead
of multiple exec()s on the scripts not supporting Python 3 is really
relevant. Please measure it if you believe so.

To be honest, I don't see any real advantage in this solution. It is
complex; understanding it requires explanation or some thinking.
The code will be fragile, and I'm not even sure if I'm not missing
something important here.

Thus, unless you've got a good arguments how this solution is superior
to the straightforward one done in python-exec, or numbers proving that
it is more efficient in a way noticeable to our users, -1. Smart hack,
yes, but not really beneficial.

> Either proposal here would be far faster than what we've got now; also 
> will use less space (ancillary benefit).
> 
> One subtle aspect here is that if we did this, it makes it possible to 
> avoid losing the invocation information- currently if you did 
> `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things 
> are implemented now (specifically the two layers of wrappers)- you'll 
> get python2.7 running that sphinx-build invocation.
> 
> This is wrong (it's directly discarding what the invocation 
> requested), although you're only going to see it for scripts that 
> do python introspection.
> 
> Via doing the restructuring I'm mentioning above, that issue can be 
> fixed, while making things faster/saner.

I don't see how this is relevant to the wrapper. As Mike pointed out,
python3.2 is the actual Python executable, and the wrapper you're
suggesting is a C executable -- something that does not really work
together like that.

So please elaborate on how you are actually going to solve this. Hope
it's not through patching Python...

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.

[Brian Harring]

On Thu, Oct 18, 2012 at 11:54:21PM +0200, Micha?? G??rny wrote:
> On Thu, 18 Oct 2012 02:15:43 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> 
> > There's a trick to this; currently, those generated scripts hardcode 
> > the allowed/known python versions for that package.  We obviously have 
> > to preserve that; I propose we shove it into the symlink path.
> > 
> > Basically, we add a /usr/libexec/python directory; within it, we have 
> > a wrapper binary (explained below), and a set of symlinks pointing at 
> > the root of that directory.  To cover our current python versions, the 
> > following would suffice:
> > 
> > for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8} 
> > \2.7-pypy-1.9; do
> >   ln -s ./ /usr/libexec/python/$x 
> > done
> > 
> > While that seems insane, there is a reason; via that, we can encode 
> > the allowed versions into the symlink.  Using pkgcore's pquery for 
> > example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3) 
> > instead of a wrapper script at /usr/bin/pquery, we'd have thus:
> > 
> > targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
> > targets=$(IFS=/;echo -n "${targets[*]}")
> > # This results in
> > # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
> > ln -s "/usr/libexec/python/${targets}/wrapper" \
> >   /usr/bin/pquery
> > 
> > /usr/libexec/python/wrapper upon invocation, takes a look at argv[0]; 
> > sees how it was invoked basically.  This will be the /usr/bin/whatever 
> > pathway.  It reads the symlink, in the process getting the allowed 
> > versions and preferred order of the versions.
> > 
> > Few notes; vast majority of filesystems will store the symlink target 
> > into the core inode if at all possible- in doing so, this avoids 
> > wasting an inode and is only limited by the length of the target.  
> > That length is capped by PATH_MAX- which can range from 256 to 4k (or 
> > higher).
> > 
> > For the pquery example above, that comes out to ~73 bytes for the 
> > symlink pathway; well under PATH_MAX.
> > 
> > For the scenarios where PATH_MAX caps the symlink pathway, or for 
> > whatever reason we don't want to use that trick, a tree of files 
> > contained within /usr/libexec/python/ holding the allowed versions for 
> > the matching pathway would suffice.
> 
> While I agree that it's a clever trick, I doubt it's worth the effort.
> Did you got any numbers proving it being superior over, say, trying to
> exec() scripts like I do in python-exec?
> 
> While I can imagine, that in an worst case that bunch of exec()s is
> going to be definitely slower than storing the list anyway, I doubt
> such a bad case is often.

The difference in performance there is going to be negligable; I'm not 
particularly concerned about that, and it shouldn't be a debate point 
between my notion and yours (it's only a debate point when one is 
talking about a c binary vs a python script).

One thing you're ignoring here is that the route(s) I mentioned all 
allow for control/specifying what the order of preference is for 
lookup/fallback (each link/shebang can encode that order how ever the 
hell it wants).


> Considering that the most common Python version used now is Python 2,
> how often doesn't the script support that Python version? That's a very
> rare case, and often just executing "foo-${EPYTHON}" works. In your
> case, that common case involves readlink() + parsing + exec().
>
> Even in case of Python 3 being selected, I doubt the overhead
> of multiple exec()s on the scripts not supporting Python 3 is really
> relevant. Please measure it if you believe so.
> 
> To be honest, I don't see any real advantage in this solution. It is
> complex; understanding it requires explanation or some thinking.
> The code will be fragile, and I'm not even sure if I'm not missing
> something important here.

Not particularly sure how you claim this is fragile, but whatever, 
your view.

I suggest you read my response to floppym.  There's differing ways to 
tackle this while getting the required speed, and covering what I want 
(python3.2 sphinx-build working).


> Thus, unless you've got a good arguments how this solution is superior
> to the straightforward one done in python-exec, or numbers proving that
> it is more efficient in a way noticeable to our users, -1. Smart hack,
> yes, but not really beneficial.

Your -1 is duly noted.


> > Either proposal here would be far faster than what we've got now; also 
> > will use less space (ancillary benefit).
> > 
> > One subtle aspect here is that if we did this, it makes it possible to 
> > avoid losing the invocation information- currently if you did 
> > `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things 
> > are implemented now (specifically the two layers of wrappers)- you'll 
> > get python2.7 running that sphinx-build invocation.
> > 
> > This is wrong (it's directly discarding what the invocation 
> > requested), although you're only going to see it for scripts that 
> > do python introspection.
> > 
> > Via doing the restructuring I'm mentioning above, that issue can be 
> > fixed, while making things faster/saner.
> 
> I don't see how this is relevant to the wrapper. As Mike pointed out,
> python3.2 is the actual Python executable, and the wrapper you're
> suggesting is a C executable -- something that does not really work
> together like that.
> 
> So please elaborate on how you are actually going to solve this. Hope
> it's not through patching Python...

Same thing, read my responses to floppym; not going to reiterate those 
points over here.

I will note your python-exec bit still doesn't fly for python3.2 
sphinx-build- which frankly, is the wrong direction to go in my views.

~harring

Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 6063 bytes --]

On Thu, 18 Oct 2012 16:12:55 -0700
Brian Harring <ferringb@gmail.com> wrote:

> On Thu, Oct 18, 2012 at 11:54:21PM +0200, Micha?? G??rny wrote:
> > On Thu, 18 Oct 2012 02:15:43 -0700
> > Brian Harring <ferringb@gmail.com> wrote:
> > 
> > > There's a trick to this; currently, those generated scripts hardcode 
> > > the allowed/known python versions for that package.  We obviously have 
> > > to preserve that; I propose we shove it into the symlink path.
> > > 
> > > Basically, we add a /usr/libexec/python directory; within it, we have 
> > > a wrapper binary (explained below), and a set of symlinks pointing at 
> > > the root of that directory.  To cover our current python versions, the 
> > > following would suffice:
> > > 
> > > for x in {2.{4,5,6,7},3.{0,1,2,3,4}}-cpy 2.5-jython 2.7-pypy-1.{7,8} 
> > > \2.7-pypy-1.9; do
> > >   ln -s ./ /usr/libexec/python/$x 
> > > done
> > > 
> > > While that seems insane, there is a reason; via that, we can encode 
> > > the allowed versions into the symlink.  Using pkgcore's pquery for 
> > > example (which should support cpy: 2.5, 2.6, 2.7, 3.1, 3.2, 3.3) 
> > > instead of a wrapper script at /usr/bin/pquery, we'd have thus:
> > > 
> > > targets=( 2.{5,6,7}-cpy 3.{1,2,3}-cpy )
> > > targets=$(IFS=/;echo -n "${targets[*]}")
> > > # This results in
> > > # targets=2.5-cpy/2.6-cpy/2.7-cpy/3.1-cpy/3.2-cpy/3.3-cpy
> > > ln -s "/usr/libexec/python/${targets}/wrapper" \
> > >   /usr/bin/pquery
> > > 
> > > /usr/libexec/python/wrapper upon invocation, takes a look at argv[0]; 
> > > sees how it was invoked basically.  This will be the /usr/bin/whatever 
> > > pathway.  It reads the symlink, in the process getting the allowed 
> > > versions and preferred order of the versions.
> > > 
> > > Few notes; vast majority of filesystems will store the symlink target 
> > > into the core inode if at all possible- in doing so, this avoids 
> > > wasting an inode and is only limited by the length of the target.  
> > > That length is capped by PATH_MAX- which can range from 256 to 4k (or 
> > > higher).
> > > 
> > > For the pquery example above, that comes out to ~73 bytes for the 
> > > symlink pathway; well under PATH_MAX.
> > > 
> > > For the scenarios where PATH_MAX caps the symlink pathway, or for 
> > > whatever reason we don't want to use that trick, a tree of files 
> > > contained within /usr/libexec/python/ holding the allowed versions for 
> > > the matching pathway would suffice.
> > 
> > While I agree that it's a clever trick, I doubt it's worth the effort.
> > Did you got any numbers proving it being superior over, say, trying to
> > exec() scripts like I do in python-exec?
> > 
> > While I can imagine, that in an worst case that bunch of exec()s is
> > going to be definitely slower than storing the list anyway, I doubt
> > such a bad case is often.
> 
> The difference in performance there is going to be negligable; I'm not 
> particularly concerned about that, and it shouldn't be a debate point 
> between my notion and yours (it's only a debate point when one is 
> talking about a c binary vs a python script).
> 
> One thing you're ignoring here is that the route(s) I mentioned all 
> allow for control/specifying what the order of preference is for 
> lookup/fallback (each link/shebang can encode that order how ever the 
> hell it wants).

Where's the use-case for that? Do we really prefer having the order of
preference hard-wired when installing a particular package instead of
having a control over it in the eclass/a single package?

> > Considering that the most common Python version used now is Python 2,
> > how often doesn't the script support that Python version? That's a very
> > rare case, and often just executing "foo-${EPYTHON}" works. In your
> > case, that common case involves readlink() + parsing + exec().
> >
> > Even in case of Python 3 being selected, I doubt the overhead
> > of multiple exec()s on the scripts not supporting Python 3 is really
> > relevant. Please measure it if you believe so.
> > 
> > To be honest, I don't see any real advantage in this solution. It is
> > complex; understanding it requires explanation or some thinking.
> > The code will be fragile, and I'm not even sure if I'm not missing
> > something important here.
> 
> Not particularly sure how you claim this is fragile, but whatever, 
> your view.

The fragile part is using a symlink to store *data*.

> > > Either proposal here would be far faster than what we've got now; also 
> > > will use less space (ancillary benefit).
> > > 
> > > One subtle aspect here is that if we did this, it makes it possible to 
> > > avoid losing the invocation information- currently if you did 
> > > `/usr/bin/python3.2 $(which sphinx-build) blah`, because of how things 
> > > are implemented now (specifically the two layers of wrappers)- you'll 
> > > get python2.7 running that sphinx-build invocation.
> > > 
> > > This is wrong (it's directly discarding what the invocation 
> > > requested), although you're only going to see it for scripts that 
> > > do python introspection.
> > > 
> > > Via doing the restructuring I'm mentioning above, that issue can be 
> > > fixed, while making things faster/saner.
> > 
> > I don't see how this is relevant to the wrapper. As Mike pointed out,
> > python3.2 is the actual Python executable, and the wrapper you're
> > suggesting is a C executable -- something that does not really work
> > together like that.
> > 
> > So please elaborate on how you are actually going to solve this. Hope
> > it's not through patching Python...
> 
> Same thing, read my responses to floppym; not going to reiterate those 
> points over here.
> 
> I will note your python-exec bit still doesn't fly for python3.2 
> sphinx-build- which frankly, is the wrong direction to go in my views.

Yes, indeed it is a problem. I will be happy to fix it when a good
solution comes out.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] reworking python-wrapper and wrapper scripts.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

Ah, and a minor note.

On Thu, 18 Oct 2012 02:15:43 -0700
Brian Harring <ferringb@gmail.com> wrote:

> /usr/libexec/python/wrapper upon invocation, takes a look at argv[0]; 
> sees how it was invoked basically.  This will be the /usr/bin/whatever 
> pathway.  It reads the symlink, in the process getting the allowed 
> versions and preferred order of the versions.
> 
> Few notes; vast majority of filesystems will store the symlink target 
> into the core inode if at all possible- in doing so, this avoids 
> wasting an inode and is only limited by the length of the target.  
> That length is capped by PATH_MAX- which can range from 256 to 4k (or 
> higher).

It is capped by SYMLINK_MAX, I believe.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]