[gentoo-python] [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[gentoo-python] [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Michał Górny]

The idea is very simple: /etc/env.d/python/python[23] with a one-line
value similar to the main interpreter /config file.

That should be simpler & more reliable than reading a symlink. And at
some point we can replace the symlink with an $EPYTHON-aware wrapper
as well.
---
 python.eselect | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/python.eselect b/python.eselect
index f72e5cc..8d4858a 100644
--- a/python.eselect
+++ b/python.eselect
@@ -117,6 +117,10 @@ set_scripts_and_symlinks() {
 
 		pushd "${INTERPRETER_PATH}" 1> /dev/null
 
+		# store pythonX preferred interpreter in a file as well
+		echo "${target}" > "${ENV_D_PATH}/python/${target%.*}"
+
+		# create pythonX -> pythonX.Y symlink
 		ln -nfs "${target}" "${target%.*}"
 		if [[ "${SET_MAIN_ACTIVE_PYTHON_INTERPRETER}" == "1" ]]; then
 			set_python "${target}"
-- 
1.7.12.4

[gentoo-python] [PATCH eselect-python 2/2] Re-set the same interpreters on 'update --if-unset'.

[Michał Górny]

This will handle migrating/fixing user settings. In this particular
case, it's needed to write /etc/env.d/python/python* files on
eselect-python upgrade.
---
 python.eselect | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/python.eselect b/python.eselect
index 8d4858a..f69d5df 100644
--- a/python.eselect
+++ b/python.eselect
@@ -382,10 +382,16 @@ do_update() {
 
 	if [[ "${if_unset}" == "1" && -f "${INTERPRETER_PATH}python" && -f "${ENV_D_PATH}/python/config" ]]; then
 		if [[ "${python2}" == "1" && -f "${INTERPRETER_PATH}python2" ]]; then
-			return
+			target="$(readlink "${INTERPRETER_PATH}python2")"
 		elif [[ "${python3}" == "1" && -f "${INTERPRETER_PATH}python3" ]]; then
-			return
+			target="$(readlink "${INTERPRETER_PATH}python3")"
 		elif [[ "${python2}" == "0" && "${python3}" == "0" ]]; then
+			target="$(<"${ENV_D_PATH}/python/config")"
+		fi
+
+		if [[ ${target} ]]; then
+			echo "Resetting ${target}"
+			do_set ${python_version_option} ${target}
 			return
 		fi
 	fi
-- 
1.7.12.4

[gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Mike Gilbert]

On Thu, Nov 1, 2012 at 11:44 AM, Mike Gilbert <floppymaster@gmail.com> wrote:
> On Thu, Nov 1, 2012 at 7:54 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> The idea is very simple: /etc/env.d/python/python[23] with a one-line
>> value similar to the main interpreter /config file.
>>
>> That should be simpler & more reliable than reading a symlink. And at
>> some point we can replace the symlink with an $EPYTHON-aware wrapper
>> as well.
>
> I don't understand the point of this. Do we have some need to enable
> EPYTHON usages for scripts that have a python2 or python3 shebang?
>
> I also don't understand how a text file is more reliable than a
> symlink; they are basically the same thing, but the symlink has a
> different file mode.

Another reply that got lost due to wrong From: address.

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1686 bytes --]

On Thu, 1 Nov 2012 14:48:55 -0400
Mike Gilbert <floppym@gentoo.org> wrote:

> On Thu, Nov 1, 2012 at 11:44 AM, Mike Gilbert <floppymaster@gmail.com> wrote:
> > On Thu, Nov 1, 2012 at 7:54 AM, Michał Górny <mgorny@gentoo.org> wrote:
> >> The idea is very simple: /etc/env.d/python/python[23] with a one-line
> >> value similar to the main interpreter /config file.
> >>
> >> That should be simpler & more reliable than reading a symlink. And at
> >> some point we can replace the symlink with an $EPYTHON-aware wrapper
> >> as well.
> >
> > I don't understand the point of this. Do we have some need to enable
> > EPYTHON usages for scripts that have a python2 or python3 shebang?

I'm mostly thinking of a future potential. Or maybe just I dislike that
partially we're using wrappers, and partially symlinks.

> > I also don't understand how a text file is more reliable than a
> > symlink; they are basically the same thing, but the symlink has a
> > different file mode.

They have a different goal, I believe. The goal of a symlink is to tie
one path onto another; the goal of a file is to store data. I'd say
that symlink is valid as long as it points to the correct path, one way
or the other.

But well, it's more like 'we are keeping python in one place, but
python2/3 in another'. python-exec would also become simpler if one
function would be needed to get both kinds of implementations rather
than two different approaches.

Last but probably least, POSIX doesn't list 'readlink' utility. Not
sure how various Prefixes (interix?) handle that but I really dislike
the idea of storing data in symlinks.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]

On Thu, 1 Nov 2012 14:48:55 -0400
Mike Gilbert <floppym@gentoo.org> wrote:

> On Thu, Nov 1, 2012 at 11:44 AM, Mike Gilbert <floppymaster@gmail.com> wrote:
> > On Thu, Nov 1, 2012 at 7:54 AM, Michał Górny <mgorny@gentoo.org> wrote:
> >> The idea is very simple: /etc/env.d/python/python[23] with a one-line
> >> value similar to the main interpreter /config file.
> >>
> >> That should be simpler & more reliable than reading a symlink. And at
> >> some point we can replace the symlink with an $EPYTHON-aware wrapper
> >> as well.
> >
> > I don't understand the point of this. Do we have some need to enable
> > EPYTHON usages for scripts that have a python2 or python3 shebang?
> >
> > I also don't understand how a text file is more reliable than a
> > symlink; they are basically the same thing, but the symlink has a
> > different file mode.

Ah, and I'd forget. I have the following dream:

  /etc/env.d/python/python2
  /etc/env.d/python/python3
  /etc/env.d/python/config -> python2

So, python{2,3} keep the per-version interpreters, and config is just
a symlink to one of them. Two bird with one stone -- readlink to get
which group is enabled, cat to get the exact interpreter. I'm proud
of myself!

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Brian Harring]

On Thu, Nov 01, 2012 at 10:27:19PM +0100, Micha?? G??rny wrote:
> On Thu, 1 Nov 2012 14:48:55 -0400
> Mike Gilbert <floppym@gentoo.org> wrote:
> 
> > On Thu, Nov 1, 2012 at 11:44 AM, Mike Gilbert <floppymaster@gmail.com> wrote:
> > > On Thu, Nov 1, 2012 at 7:54 AM, Micha?? G??rny <mgorny@gentoo.org> wrote:
> > >> The idea is very simple: /etc/env.d/python/python[23] with a one-line
> > >> value similar to the main interpreter /config file.
> > >>
> > >> That should be simpler & more reliable than reading a symlink. And at
> > >> some point we can replace the symlink with an $EPYTHON-aware wrapper
> > >> as well.
> > >
> > > I don't understand the point of this. Do we have some need to enable
> > > EPYTHON usages for scripts that have a python2 or python3 shebang?
> > >
> > > I also don't understand how a text file is more reliable than a
> > > symlink; they are basically the same thing, but the symlink has a
> > > different file mode.
> 
> Ah, and I'd forget. I have the following dream:
> 
>   /etc/env.d/python/python2
>   /etc/env.d/python/python3
>   /etc/env.d/python/config -> python2
> 
> So, python{2,3} keep the per-version interpreters, and config is just
> a symlink to one of them. Two bird with one stone -- readlink to get
> which group is enabled, cat to get the exact interpreter. I'm proud
> of myself!

I too have a dream; git://pkgcore.org/eslect-python .

That's a shebang based version of what I proposed a while back.  It 
works now and has tests.  It's written to be basically a drop in for 
existing python.eclass generation of shebangs, w/ the 
hardlinking/de-duplication/farking-fast/fix python3.2 
/usr/bin/sphinx-build scenario.

The remaining work for that is thus:

1) If the EPYTHON targets aren't given via shebang arg, then it needs
 to fallback to grabbing the targets from the file (easy enough).

2) Add a few helpers/wrappers to make it easier to do the 
deduplication/shebang rewriting.

Frankly, you should be looking at this imo, rather than trying 
standalone files.  Yes, files is simpler- but you'll wind up sooner or 
later rebuilding it into what I already built out here.

Either way, will let the commit message speak for it:

"""
Add a python-shebang utility, slave python-wrapper to it.

Going forward, the intent is to push the PYTHON_ABIs of a given
script down into it's shebang, pointing the shebang at python-shebang.

In this way, all known/supported abi's are available at the time of
execution; further, if the target is told to respect-EPYTHON (meaning
no searching for something to execute, either active python version or
EPYTHON var), we can push this down into that list and have that code
handle it.

Using sphinx-build as an example, we go from the current setup of:
echo > sphinx-build-2.7 <<EOF
#!/usr/bin/python2.7
<code>
EOF

echo > sphinx-build-3.2 <<EOF
#!/usr/bin/python3.2
<code>
EOF

echo > sphinx-build << EOF
#!/usr/bin/env python
<python wrapper code to decide which of the above to execute>
EOF

to the simpler form of:
for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
#!/usr/bin/python-shebang python2.7,python3.2
<code>
EOF
done

The gains of this are thus:

1) This simplifies the pypy EPYTHON->script_name parsing code; since
 we have the list of PYTHON_ABIs available at the script level, we
 can use that info to map it directly (thus we don't have to update
 that code till pypy changes their interp name).

2) Measurably faster implementation; minimally, via moving this to
 c we're not paying the VM overhead that the current wrapper solution
 requires; further, via accessing the underlying python config file,
 we can get the active version without spawning bash (the python.eclass
 wrapper solution semi-validly doesn't do this since the given wrapper
 isn't part of eselect-python; python-shebang is, thus we can do the
 faster path).  The code will still fallback to eselect python show as
 a protective measure, but that pathway should never be executed
 realistically.

3) Via this approach, we can do de-duplication of the versioned
 scripts; since each (including the versioned forms) carry the
 python ABIs as an argument, their content is identical- so we can just
 hardlink them together (which the code looks for and exploits to avoid
 a double exec, instead exec'ing directly to the target).

4) The underlying python-wrapper in turn, just becomes a dumb re-exec
 to `python-shebang ACTIVE_PYTHON`- a special mode of python-shebang,
 which is essentially respect-EPYTHON, while defaulting EPYTHON to
 the active python version.  Single implementation basically.

5) java-config-2's jython cycle can be explicitly broken by the script itself
 now; since the script now specifies the ABIs it targets, java-config-2's shebang
 just would drop jython2.5 from the list of targets.  Via this, on a system that
 has jython2.5 as the active interp, when it goes to invoke java-config-2 (a necessary
 step to actually start the jython interp itself) it would use an implementation other
 than jython2.5- no fork bomb/exec cycle.

6) python3.2 sphinx-build now does the correct thing- execute sphinx-build-3.2, even
 if the active is python2.7.
"""

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4070 bytes --]

On Thu, 1 Nov 2012 15:42:42 -0700
Brian Harring <ferringb@gmail.com> wrote:

> On Thu, Nov 01, 2012 at 10:27:19PM +0100, Micha?? G??rny wrote:
> > On Thu, 1 Nov 2012 14:48:55 -0400
> > Mike Gilbert <floppym@gentoo.org> wrote:
> > 
> > > On Thu, Nov 1, 2012 at 11:44 AM, Mike Gilbert <floppymaster@gmail.com> wrote:
> > > > On Thu, Nov 1, 2012 at 7:54 AM, Micha?? G??rny <mgorny@gentoo.org> wrote:
> > > >> The idea is very simple: /etc/env.d/python/python[23] with a one-line
> > > >> value similar to the main interpreter /config file.
> > > >>
> > > >> That should be simpler & more reliable than reading a symlink. And at
> > > >> some point we can replace the symlink with an $EPYTHON-aware wrapper
> > > >> as well.
> > > >
> > > > I don't understand the point of this. Do we have some need to enable
> > > > EPYTHON usages for scripts that have a python2 or python3 shebang?
> > > >
> > > > I also don't understand how a text file is more reliable than a
> > > > symlink; they are basically the same thing, but the symlink has a
> > > > different file mode.
> > 
> > Ah, and I'd forget. I have the following dream:
> > 
> >   /etc/env.d/python/python2
> >   /etc/env.d/python/python3
> >   /etc/env.d/python/config -> python2
> > 
> > So, python{2,3} keep the per-version interpreters, and config is just
> > a symlink to one of them. Two bird with one stone -- readlink to get
> > which group is enabled, cat to get the exact interpreter. I'm proud
> > of myself!
> 
> I too have a dream; git://pkgcore.org/eslect-python .
> 
> That's a shebang based version of what I proposed a while back.  It 
> works now and has tests.  It's written to be basically a drop in for 
> existing python.eclass generation of shebangs, w/ the 
> hardlinking/de-duplication/farking-fast/fix python3.2 
> /usr/bin/sphinx-build scenario.

Could you please provide an explanation of what it does? I mean, a few
points of what happens when you run it in various scenarios. That's 600
lines of C code, I think Python folks would appreciate not having to
read that line by line.

> The remaining work for that is thus:
> 
> 1) If the EPYTHON targets aren't given via shebang arg, then it needs
>  to fallback to grabbing the targets from the file (easy enough).
> 
> 2) Add a few helpers/wrappers to make it easier to do the 
> deduplication/shebang rewriting.

How would you perform deduplication while you're at it? Could you
measure or at least consider the benefits of it?

> Frankly, you should be looking at this imo, rather than trying 
> standalone files.  Yes, files is simpler- but you'll wind up sooner or 
> later rebuilding it into what I already built out here.

I think you are missing the point I'm raising here. Mostly because your
python-wrapper doesn't handle per-version preferences like python-exec
does, unless I'm missing something.

> """
> Add a python-shebang utility, slave python-wrapper to it.
> 
> Going forward, the intent is to push the PYTHON_ABIs of a given
> script down into it's shebang, pointing the shebang at python-shebang.
> 
> In this way, all known/supported abi's are available at the time of
> execution; further, if the target is told to respect-EPYTHON (meaning
> no searching for something to execute, either active python version or
> EPYTHON var), we can push this down into that list and have that code
> handle it.
> 
> [...]
>
> to the simpler form of:
> for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
> #!/usr/bin/python-shebang python2.7,python3.2
> <code>
> EOF
> done

Hmm, are you assuming that /usr/bin/sphinx carries the actual code?
Or some special wrapper code?

Not to mention you still bind the whole thing to installed package.
If you need to fix those files, you need to reinstall all packages
installing them.

> The gains of this are thus:

Could you please compare it to the modern solution (python-exec) rather
than the deprecated one? You are no longer on the uncharted waters.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Brian Harring]

On Fri, Nov 02, 2012 at 10:03:05AM +0100, Micha?? G??rny wrote:
> On Thu, 1 Nov 2012 15:42:42 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> 
> > On Thu, Nov 01, 2012 at 10:27:19PM +0100, Micha?? G??rny wrote:
> > > On Thu, 1 Nov 2012 14:48:55 -0400
> > > Mike Gilbert <floppym@gentoo.org> wrote:
> > > 
> > > > On Thu, Nov 1, 2012 at 11:44 AM, Mike Gilbert <floppymaster@gmail.com> wrote:
> > > > > On Thu, Nov 1, 2012 at 7:54 AM, Micha?? G??rny <mgorny@gentoo.org> wrote:
> > > > >> The idea is very simple: /etc/env.d/python/python[23] with a one-line
> > > > >> value similar to the main interpreter /config file.
> > > > >>
> > > > >> That should be simpler & more reliable than reading a symlink. And at
> > > > >> some point we can replace the symlink with an $EPYTHON-aware wrapper
> > > > >> as well.
> > > > >
> > > > > I don't understand the point of this. Do we have some need to enable
> > > > > EPYTHON usages for scripts that have a python2 or python3 shebang?
> > > > >
> > > > > I also don't understand how a text file is more reliable than a
> > > > > symlink; they are basically the same thing, but the symlink has a
> > > > > different file mode.
> > > 
> > > Ah, and I'd forget. I have the following dream:
> > > 
> > >   /etc/env.d/python/python2
> > >   /etc/env.d/python/python3
> > >   /etc/env.d/python/config -> python2
> > > 
> > > So, python{2,3} keep the per-version interpreters, and config is just
> > > a symlink to one of them. Two bird with one stone -- readlink to get
> > > which group is enabled, cat to get the exact interpreter. I'm proud
> > > of myself!
> > 
> > I too have a dream; git://pkgcore.org/eslect-python .
> > 
> > That's a shebang based version of what I proposed a while back.  It 
> > works now and has tests.  It's written to be basically a drop in for 
> > existing python.eclass generation of shebangs, w/ the 
> > hardlinking/de-duplication/farking-fast/fix python3.2 
> > /usr/bin/sphinx-build scenario.
> 
> Could you please provide an explanation of what it does? I mean, a few
> points of what happens when you run it in various scenarios. That's 600
> lines of C code, I think Python folks would appreciate not having to
> read that line by line.

Grumble.  Don't think you read my commit messages at all, just 
responded looking for points to claim "it sucks".

What I have written here is a python-shebang based solution.  That 
shebang target includes the EPYTHON's it supports.

For the usual sphinx-build example, sphinx-build-2.7 goes from
#!/usr/bin/python2.7

to

#!/usr/bin/python-shebang python2.7,python3.2

This means that sphinx-build-2.7 and sphinx-build-3.2 are now the same 
content.  They can be hardlinked together (space savings).

This additionally means that sphinx-build can be hardlinked to the 
same underlying inode.  Why?  Because python-shebang is smart enough 
to recognize the context it is invoked in.

This additionally means that the follow commands:

eselect python set 3.2
python2.7 /usr/bin/sphinx-build

properly work- invoking sphinx-bulid in a 2.7 context (for anyone 
trying to write distutils/setup bits that aren't gentoo/EPYTHON 
specific, this is fricking useful to say the least).

python-shebang does thus:

1) parse argv[1]; split it first on ' ' (everything following is split 
by ' ', and used as args for the python interp invoked).  The leftmost
is split on ','; this is the list of interpretter targets this script 
was installed for.

2) identify the context it was invoked via.  If invoked via an interp 
specific pathway (aka, /usr/bin/sphinx-build-2.7; a forced python2.7 
context) then it honors that, and re-execs to python2.7 $0 w/ the 
additional interp args.  That re-exec fail, it does the usual failure 
return.

3) Making it here means it was invoked in a non forced interp context; 
meaning /usr/bin/sphinx-build.  If the installation required EPYTHON 
to be honored always (this actually is used if you analyzed the tree), 
w/in the interp targets there is a EPYTHON value- literally, 
"EPYTHON".  If EPYTHON env var is set, and EPYTHON is set, force that 
interpretter or fail.

3) If EPYTHON is set but the script wasn't marked as "honor EPYTHON 
always", check the list of interp targets, if we can the EPYTHON 
target, re-exec to it.

4) Making it here means that env EPYTHON wasn't forced, nor can it be 
honored.  Grab the active python version, re-exec to that if it's a 
support interpretter for this script.

5) If it ain't, loop over the list of supported targets, l->r, trying 
to re-exec in that ordering.


> > The remaining work for that is thus:
> > 
> > 1) If the EPYTHON targets aren't given via shebang arg, then it needs
> >  to fallback to grabbing the targets from the file (easy enough).
> > 
> > 2) Add a few helpers/wrappers to make it easier to do the 
> > deduplication/shebang rewriting.
> 
> How would you perform deduplication while you're at it? Could you
> measure or at least consider the benefits of it?

I described this in the original dev ml thread you responded to 
bitching about symlinks; rewrite the shebang, md5 comparison, what's 
the same is hardlinked; if all are the same, hardlink the wrapper 
(meaning if sphinx-build-{2.7,3.2} are the same md5, hardlink 
sphinx-build to the same underlying inode).

It's pretty simple.


> > Frankly, you should be looking at this imo, rather than trying 
> > standalone files.  Yes, files is simpler- but you'll wind up sooner or 
> > later rebuilding it into what I already built out here.
> 
> I think you are missing the point I'm raising here. Mostly because your
> python-wrapper doesn't handle per-version preferences like python-exec
> does, unless I'm missing something.

You're misunderstanding what was written; hell, I even referenced 
that it supports EPYTHON preference in #5 (java-config cycle 
breaking).


> > """
> > Add a python-shebang utility, slave python-wrapper to it.
> > 
> > Going forward, the intent is to push the PYTHON_ABIs of a given
> > script down into it's shebang, pointing the shebang at python-shebang.
> > 
> > In this way, all known/supported abi's are available at the time of
> > execution; further, if the target is told to respect-EPYTHON (meaning
> > no searching for something to execute, either active python version or
> > EPYTHON var), we can push this down into that list and have that code
> > handle it.
> > 
> > [...]
> >
> > to the simpler form of:
> > for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
> > #!/usr/bin/python-shebang python2.7,python3.2
> > <code>
> > EOF
> > done
> 
> Hmm, are you assuming that /usr/bin/sphinx carries the actual code?
> Or some special wrapper code?
> 
> Not to mention you still bind the whole thing to installed package.
> If you need to fix those files, you need to reinstall all packages
> installing them.

This critique makes no sense... and I'm pretty sure that any point you 
may manage to come up with, will apply directly to python-exec just as 
equally.

Either clarify your claim, or drop the noise there.


> > The gains of this are thus:
> 
> Could you please compare it to the modern solution (python-exec) rather
> than the deprecated one? You are no longer on the uncharted waters.

Respectfully, your 'modern solution' (vs deprecated) is purely in your 
head; distutils/python eclasses of your making aren't the tree.  

They're your playground right now- one that you're pushing, but I've 
yet to see real traction on.  That's just reality; you may think it's 
a done deal (or will be), but the on the ground reality right now- 
including the pushback you've been getting from people in certain 
cases- makes me unwilling to pin my hopes on your work.


Either way, you asked, so here's the list of python-exec con's in 
comparison to python-shebang:

1) Your solution actually doesn't work as a fricking wrapper for a 
core usage case; `python /usr/bin/sphinx-build`.  That perfectly 
innocuous/valid invocation would result in the interpretter trying 
to execute an elf binary.  That pretty much renders your solution as a 
no go right from the get go.  Your only way to fix that is to have 
sphinx-build be a python implementation that does wrapping, and use a 
shebang of python-exec (so you can maintain speed for direct 
/usr/bin/sphinx-build invocations); meaning once you spotted that 
flaw, you'd evolve your attempt into something nearing what I wrote.

2) Everytime there is a new python interpretter, python-exec has to 
be re-built updating it's hardcoded list of interpretters.  Goes without 
saying, this sucks a fair bit; your solution there likely would be 
walk env.d/python/* in some fashion extracting the list of targets (or 
walk /usr/bin/python* and run into problems w/ the names used there 
now matching EPYTHON usage).  End result, you'd evolve it into 
something nearing what I did w/ shebangs.

3) It's a hardcoded lookup list that is forced for all scripts; if 
that list is python3.2,python3.1,python2.7, and the active python 
version doesn't intersect that (say jython-2.5), your approach forces 
3.2.  My solution actually allows the script, in the absense of an 
enforcement via active python or EPYTHON, to use it's own preferred 
ordering.  Using pkgcore as an example, we prefer py2k execution over 
py3k- py2k is faster since the extensions are enabled.  This 
capability doesn't exist in your solution, nor the existing wrapper; 
it's a useful byproduct of how my solution is structured.  Etc etc, 
you know the response; where you'll go with this sooner or later is 
where python-shebang is already.

5) Via going with a standalone package from eselect-python, you're 
introducing the potential for the two to be out of sync/incompatible.  
Addressable via folding it inline into eselect-python (as I did), but 
it's a con against your solution in it's current claimed form.  Same 
thing, you'll do what I did sooner or later since it's the better 
approach.

6) With N=# of python implementations, you're forcing N copies of that 
script.  My solution will de-dupe it down to where the content is the 
same; most scripts, that means all script-* and script collapses down 
to one single inode.  If py3k content differs from py2k, and it's 
installed for py2k/py3k, this means in that case 3 inodes.  Your 
solution has no way to avoid the inode/space overhead, not w/out 
switching the shebang.

7) My solution doesn't hardcode /usr/bin/* pathways.  User sticks a 
python2.7 into /usr/local/ (for whatever reason they choose to do so), 
it will honor that.

8) Your solution doesn't work in the context of being used for 
/usr/bin/python wrapper; it's close, but as you yourself said, it 
requires symlinks (python-python2.7).

9) No tests.  Every. Fucking. Pathway. is tested with what I wrote.  
Python-exec lacks that, else issue #1 would've been spotted up front.


What you've got only works for direct invocation of a target via 
/usr/bin/sphinx-build; python /usr/bin/sphinx-build won't fly; nor can 
it be used as the the python wrapper w/out changing how we install the 
interpretter.


Python shebang however works all existing scenarios, and fixes the 
core scenario that got me started here; `python2.7 
/usr/bin/sphinx-build`.

The only flaws with it at this point are thus:

1) There is a 127 char limit on shebang length.  This is addressable 
via having python-shebang reach grab the list from the target itself 
if that limit is encountered.

2) fleshing out the tools for doing de-duplication; it's easy enough, 
I've just been busy with other things, and this code was done as a 
side project for entertainment.  I'll finish it soon enough (else if 
someone wants to sort the remaining issues, I can detail exactly how).


The nice part about my solution here is that it's not something that 
is hinged mgorny's python/distutils eclasses becoming the norm; it's 
usable now for existing python eclass (simple enough to plug it in), 
and any followup solution would be insane not to use it.

~harring

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 12005 bytes --]

On Fri, 2 Nov 2012 20:35:38 -0700
Brian Harring <ferringb@gmail.com> wrote:

> On Fri, Nov 02, 2012 at 10:03:05AM +0100, Micha?? G??rny wrote:
> > On Thu, 1 Nov 2012 15:42:42 -0700
> > Brian Harring <ferringb@gmail.com> wrote:
> > 
> > > That's a shebang based version of what I proposed a while back.  It 
> > > works now and has tests.  It's written to be basically a drop in for 
> > > existing python.eclass generation of shebangs, w/ the 
> > > hardlinking/de-duplication/farking-fast/fix python3.2 
> > > /usr/bin/sphinx-build scenario.
> > 
> > Could you please provide an explanation of what it does? I mean, a few
> > points of what happens when you run it in various scenarios. That's 600
> > lines of C code, I think Python folks would appreciate not having to
> > read that line by line.
> 
> Grumble.  Don't think you read my commit messages at all, just 
> responded looking for points to claim "it sucks".

Unless I'm missing something, the commit message says how to use it
and what are its advantages. But thanks for the list, that's what I was
asking for.

> What I have written here is a python-shebang based solution.  That 
> shebang target includes the EPYTHON's it supports.
> 
> For the usual sphinx-build example, sphinx-build-2.7 goes from
> #!/usr/bin/python2.7
> 
> to
> 
> #!/usr/bin/python-shebang python2.7,python3.2
> 
> This means that sphinx-build-2.7 and sphinx-build-3.2 are now the same 
> content.  They can be hardlinked together (space savings).
> 
> This additionally means that sphinx-build can be hardlinked to the 
> same underlying inode.  Why?  Because python-shebang is smart enough 
> to recognize the context it is invoked in.
> 
> This additionally means that the follow commands:
> 
> eselect python set 3.2
> python2.7 /usr/bin/sphinx-build
> 
> properly work- invoking sphinx-bulid in a 2.7 context (for anyone 
> trying to write distutils/setup bits that aren't gentoo/EPYTHON 
> specific, this is fricking useful to say the least).

I think you've missed my question: what code does the 'sphinx-build'
contain, except for the shebang? Is that a Python version
of the wrapper? Some kind of os.exec()?

> > > Frankly, you should be looking at this imo, rather than trying 
> > > standalone files.  Yes, files is simpler- but you'll wind up sooner or 
> > > later rebuilding it into what I already built out here.
> > 
> > I think you are missing the point I'm raising here. Mostly because your
> > python-wrapper doesn't handle per-version preferences like python-exec
> > does, unless I'm missing something.
> 
> You're misunderstanding what was written; hell, I even referenced 
> that it supports EPYTHON preference in #5 (java-config cycle 
> breaking).

EPYTHON preference is rarely used by users. Users are using
eselect-python. And eselect-python uses two levels of preferences:

1. 'Main' python interpreter preference,

2. Per-Python2/Python3 interpreter preference.

With the former being stored in a file, the latter in symlinks.
The goal of my patches were to store both in files so that the symlink
reading code could be removed to make the code simpler.

> > > """
> > > Add a python-shebang utility, slave python-wrapper to it.
> > > 
> > > Going forward, the intent is to push the PYTHON_ABIs of a given
> > > script down into it's shebang, pointing the shebang at python-shebang.
> > > 
> > > In this way, all known/supported abi's are available at the time of
> > > execution; further, if the target is told to respect-EPYTHON (meaning
> > > no searching for something to execute, either active python version or
> > > EPYTHON var), we can push this down into that list and have that code
> > > handle it.
> > > 
> > > [...]
> > >
> > > to the simpler form of:
> > > for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
> > > #!/usr/bin/python-shebang python2.7,python3.2
> > > <code>
> > > EOF
> > > done
> > 
> > Hmm, are you assuming that /usr/bin/sphinx carries the actual code?
> > Or some special wrapper code?
> > 
> > Not to mention you still bind the whole thing to installed package.
> > If you need to fix those files, you need to reinstall all packages
> > installing them.
> 
> This critique makes no sense... and I'm pretty sure that any point you 
> may manage to come up with, will apply directly to python-exec just as 
> equally.
> 
> Either clarify your claim, or drop the noise there.

python-exec can't be executed like that currently, correct. By I can
make it so just by replacing the python-exec executable as the symlinks
will make it effectively to every single package.

In your case, any change won't apply globally.

> > > The gains of this are thus:
> > 
> > Could you please compare it to the modern solution (python-exec) rather
> > than the deprecated one? You are no longer on the uncharted waters.
> 
> Respectfully, your 'modern solution' (vs deprecated) is purely in your 
> head; distutils/python eclasses of your making aren't the tree.  
> 
> They're your playground right now- one that you're pushing, but I've 
> yet to see real traction on.  That's just reality; you may think it's 
> a done deal (or will be), but the on the ground reality right now- 
> including the pushback you've been getting from people in certain 
> cases- makes me unwilling to pin my hopes on your work.

It is just a difference in methods. You want to push your solution to
the whole tree at once and see what happens next. I rather prefer
testing it on new ebuilds first, then getting it stable and thinking
about replacing the old one when the new one is ready.

> Either way, you asked, so here's the list of python-exec con's in 
> comparison to python-shebang:
> 
> 1) Your solution actually doesn't work as a fricking wrapper for a 
> core usage case; `python /usr/bin/sphinx-build`.  That perfectly 
> innocuous/valid invocation would result in the interpretter trying 
> to execute an elf binary.  That pretty much renders your solution as a 
> no go right from the get go.  Your only way to fix that is to have 
> sphinx-build be a python implementation that does wrapping, and use a 
> shebang of python-exec (so you can maintain speed for direct 
> /usr/bin/sphinx-build invocations); meaning once you spotted that 
> flaw, you'd evolve your attempt into something nearing what I wrote.

No, not really. I still am afraid that you're missing some point of how
Python interpreter works but considering the fight you put protecting
'python /usr/bin/c-file', I won't continue that topic until you finally
answer my questions wrt that file.

By the way, I doubt it is a 'core usage case'. Maybe just because it
never worked as intended, maybe because people simply don't do that
usually because people usually don't care whether stuff in /usr/bin
is Python, C or shell.

> 2) Everytime there is a new python interpretter, python-exec has to 
> be re-built updating it's hardcoded list of interpretters.  Goes without 
> saying, this sucks a fair bit; your solution there likely would be 
> walk env.d/python/* in some fashion extracting the list of targets (or 
> walk /usr/bin/python* and run into problems w/ the names used there 
> now matching EPYTHON usage).  End result, you'd evolve it into 
> something nearing what I did w/ shebangs.

How often does this happen? Is this really problematic?

And how an user can disable a particular (say, problematic) Python
implementation globally in your solution? Without reinstalling all
the packages which provide scripts for it?

> 3) It's a hardcoded lookup list that is forced for all scripts; if 
> that list is python3.2,python3.1,python2.7, and the active python 
> version doesn't intersect that (say jython-2.5), your approach forces 
> 3.2.  My solution actually allows the script, in the absense of an 
> enforcement via active python or EPYTHON, to use it's own preferred 
> ordering.  Using pkgcore as an example, we prefer py2k execution over 
> py3k- py2k is faster since the extensions are enabled.  This 
> capability doesn't exist in your solution, nor the existing wrapper; 
> it's a useful byproduct of how my solution is structured.  Etc etc, 
> you know the response; where you'll go with this sooner or later is 
> where python-shebang is already.

And, hmm, are you the first person requesting such a feature over
the whole lifetime of python-wrapper?

I believe that hardcoded list is *simpler*. If user enabled jython-2.5
(suicidal type?), then he's aware of the consequences. And in my
solution, preferred Python 2 will be used. Then preferred Python 3.
Then Python 2.7, 2.6, 2.5... then 3.2. It is predictable.

In your solution, it is completely unpredictable. Developer can
(supposedly) do whatever he wants. Most of the devs probably won't use
that, except for a few cases where users will simply hit (if ever)
an unexpected behavior.

> 5) Via going with a standalone package from eselect-python, you're 
> introducing the potential for the two to be out of sync/incompatible.  
> Addressable via folding it inline into eselect-python (as I did), but 
> it's a con against your solution in it's current claimed form.  Same 
> thing, you'll do what I did sooner or later since it's the better 
> approach.

It's a better development practice to split things. You should've known
that by now. In any case, python-exec can work *without*
eselect-python. You can try it if you don't believe.

> 7) My solution doesn't hardcode /usr/bin/* pathways.  User sticks a 
> python2.7 into /usr/local/ (for whatever reason they choose to do so), 
> it will honor that.

Errr? Where does your suggestion come from?

As far as I'm aware, user can put into per-implementation copy whatever
he wants. Including a shell script, a C executable or whatever. It will
work. With your, it won't.

As far as I'm aware, you solution can't handle per-implementation
Python options (command-line arguments) either. In mine, it's as simple
as dropping them onto the shebang.

> 8) Your solution doesn't work in the context of being used for 
> /usr/bin/python wrapper; it's close, but as you yourself said, it 
> requires symlinks (python-python2.7).

Or adding a conditional which your solution simply does. So, where are
you going with this point? To the fact that I'm asking others for
opinion while you're ignoring their existence?

> 9) No tests.  Every. Fucking. Pathway. is tested with what I wrote.  
> Python-exec lacks that, else issue #1 would've been spotted up front.

Great. Awesome. You are so awesome I can't even spell it.

Now feel free to grep the bugzie. That 'issue #1' is known. So far,
I haven't get anyone to say he believes that's really important
or useful. 

> The only flaws with it at this point are thus:
> 
> 1) There is a 127 char limit on shebang length.  This is addressable 
> via having python-shebang reach grab the list from the target itself 
> if that limit is encountered.
> 
> 2) fleshing out the tools for doing de-duplication; it's easy enough, 
> I've just been busy with other things, and this code was done as a 
> side project for entertainment.  I'll finish it soon enough (else if 
> someone wants to sort the remaining issues, I can detail exactly how).
> 
> 
> The nice part about my solution here is that it's not something that 
> is hinged mgorny's python/distutils eclasses becoming the norm; it's 
> usable now for existing python eclass (simple enough to plug it in), 
> and any followup solution would be insane not to use it.

Did you ever have a bit of modesty? I think this is what gets me so mad
at you. Every thing you do is so perfect and bug-free everyone else
should be insane not to start using it immediately. Except when it
doesn't work at all but then others are just 'getting you pissy'.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Brian Harring]

On Sat, Nov 03, 2012 at 08:33:46AM +0100, Micha?? G??rny wrote:
> On Fri, 2 Nov 2012 20:35:38 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> 
> > On Fri, Nov 02, 2012 at 10:03:05AM +0100, Micha?? G??rny wrote:
> > > On Thu, 1 Nov 2012 15:42:42 -0700
> > > Brian Harring <ferringb@gmail.com> wrote:
> > What I have written here is a python-shebang based solution.  That 
> > shebang target includes the EPYTHON's it supports.
> > 
> > For the usual sphinx-build example, sphinx-build-2.7 goes from
> > #!/usr/bin/python2.7
> > 
> > to
> > 
> > #!/usr/bin/python-shebang python2.7,python3.2
> > 
> > This means that sphinx-build-2.7 and sphinx-build-3.2 are now the same 
> > content.  They can be hardlinked together (space savings).
> > 
> > This additionally means that sphinx-build can be hardlinked to the 
> > same underlying inode.  Why?  Because python-shebang is smart enough 
> > to recognize the context it is invoked in.
> > 
> > This additionally means that the follow commands:
> > 
> > eselect python set 3.2
> > python2.7 /usr/bin/sphinx-build
> > 
> > properly work- invoking sphinx-bulid in a 2.7 context (for anyone 
> > trying to write distutils/setup bits that aren't gentoo/EPYTHON 
> > specific, this is fricking useful to say the least).
> 
> I think you've missed my question: what code does the 'sphinx-build'
> contain, except for the shebang? Is that a Python version
> of the wrapper? Some kind of os.exec()?

Covered this in the email.

As I said, if dedup'ing <script>-* results in the same 
content... then the wrapper as I said, is a hardlink to it.  It's 
literally the script itself, no point in needing indirection since a 
$INTERP /usr/bin/sphinx-build would do the correct thing.

If the content varies across the python interp specific scripts, then 
a simple wrapper transfers control back to shebang, passing on the 
forced interpretter.  That wrapper is generated via thus:


abi_list=python2.7,python3.2,etc,you,get,the,jist

cat > wherever-to-write-the-wrapper << EOF
#!$(type -P python-shebang) ${abi_list}${interpreter_args:+ ${interpreter_args}}
import os, sys
os.environ["EPYTHON"]=os.path.basename(sys.executable)
targets="${abi_list}"
if hasattr(os, "execvp"):
        os.execvp("python-shebang", [sys.argv[0], targets] + sys.argv)
import subprocess
sys.exit(subprocess.Popen(["python-shebang", targets] + sys.argv).wait())
EOF


> > > > Frankly, you should be looking at this imo, rather than trying 
> > > > standalone files.  Yes, files is simpler- but you'll wind up sooner or 
> > > > later rebuilding it into what I already built out here.
> > > 
> > > I think you are missing the point I'm raising here. Mostly because your
> > > python-wrapper doesn't handle per-version preferences like python-exec
> > > does, unless I'm missing something.
> > 
> > You're misunderstanding what was written; hell, I even referenced 
> > that it supports EPYTHON preference in #5 (java-config cycle 
> > breaking).
> 
> EPYTHON preference is rarely used by users. Users are using
> eselect-python. And eselect-python uses two levels of preferences:
> 
> 1. 'Main' python interpreter preference,
> 
> 2. Per-Python2/Python3 interpreter preference.
>
> With the former being stored in a file, the latter in symlinks.
> The goal of my patches were to store both in files so that the symlink
> reading code could be removed to make the code simpler.

I've yet to see a real world justification for #2 in terms of 
it actually being desirable.

I've not supported #2 in python-shebang; that said if the python team 
(as a whole, not just mgorny yelling I didn't add it) wants it, it's 
quick to add the support and requisite tests.  See my comments further 
down however, I strongly think such control is incorrect.


> > > > """
> > > > Add a python-shebang utility, slave python-wrapper to it.
> > > > 
> > > > Going forward, the intent is to push the PYTHON_ABIs of a given
> > > > script down into it's shebang, pointing the shebang at python-shebang.
> > > > 
> > > > In this way, all known/supported abi's are available at the time of
> > > > execution; further, if the target is told to respect-EPYTHON (meaning
> > > > no searching for something to execute, either active python version or
> > > > EPYTHON var), we can push this down into that list and have that code
> > > > handle it.
> > > > 
> > > > [...]
> > > >
> > > > to the simpler form of:
> > > > for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
> > > > #!/usr/bin/python-shebang python2.7,python3.2
> > > > <code>
> > > > EOF
> > > > done
> > > 
> > > Hmm, are you assuming that /usr/bin/sphinx carries the actual code?
> > > Or some special wrapper code?
> > > 
> > > Not to mention you still bind the whole thing to installed package.
> > > If you need to fix those files, you need to reinstall all packages
> > > installing them.
> > 
> > This critique makes no sense... and I'm pretty sure that any point you 
> > may manage to come up with, will apply directly to python-exec just as 
> > equally.
> > 
> > Either clarify your claim, or drop the noise there.
> 
> python-exec can't be executed like that currently, correct. By I can
> make it so just by replacing the python-exec executable as the symlinks
> will make it effectively to every single package.
> 
> In your case, any change won't apply globally.

This isn't much of a claimed con; yes, there is an API contract here.  

That's specifically that python-shebang is invoked w/ minimally two 
arguments; the first being the EPYTHON's for the intended target; the 
second being the intended target.

That's it.  Nothing more.  As for "having to repair it"- were it 
required, it's easy enough to spot 'em (loop over vdb contents 
identifying files with /usr/bin/python-shebang as the shebang); that 
said, it's never going to be needed.


> > > > The gains of this are thus:
> > > 
> > > Could you please compare it to the modern solution (python-exec) rather
> > > than the deprecated one? You are no longer on the uncharted waters.
> > 
> > Respectfully, your 'modern solution' (vs deprecated) is purely in your 
> > head; distutils/python eclasses of your making aren't the tree.  
> > 
> > They're your playground right now- one that you're pushing, but I've 
> > yet to see real traction on.  That's just reality; you may think it's 
> > a done deal (or will be), but the on the ground reality right now- 
> > including the pushback you've been getting from people in certain 
> > cases- makes me unwilling to pin my hopes on your work.
> 
> It is just a difference in methods. You want to push your solution to
> the whole tree at once and see what happens next. I rather prefer
> testing it on new ebuilds first, then getting it stable and thinking
> about replacing the old one when the new one is ready.

Mgorny, quit it with the insinuations that I'm not paying attention to 
QA.

My solution doesn't break existing deployment, has tests, and 
actually works; yours breaks real world usage already and requires 
arbitrary renaming of scripts... which can induce it's own breakage 
(any script that know to find a python2.7 version of a script at 
<script-name>-2.7 would break).

Minimally, any solution pushed is going to require QA; this includes 
validating the implementation (already got that covered), and full 
scale testing (which I've already been doing locally as initial 
steps).

Find it funny you claim I'm just trying to jam it into the tree; 
deployment of this wasn't mentioned since I've not finished it in full 
yet.

Beyond local testing and tinderbox runs, the rough deployment plan 
in my head is thus; it's fairly cautious additionally:

1) add a use flag (defaulted off) that turns on the new implementation
2) Nudge adventurous folk to flip it on and get more testing
3) Minimally get prefix involved since if there are issues, it'll come 
from them; get them converted over.
4) Bug rate hits zero, drop the flag and make it the new norm for 
python.eclass and your 8 ebuilds.


> > Either way, you asked, so here's the list of python-exec con's in 
> > comparison to python-shebang:
> > 
> > 1) Your solution actually doesn't work as a fricking wrapper for a 
> > core usage case; `python /usr/bin/sphinx-build`.  That perfectly 
> > innocuous/valid invocation would result in the interpretter trying 
> > to execute an elf binary.  That pretty much renders your solution as a 
> > no go right from the get go.  Your only way to fix that is to have 
> > sphinx-build be a python implementation that does wrapping, and use a 
> > shebang of python-exec (so you can maintain speed for direct 
> > /usr/bin/sphinx-build invocations); meaning once you spotted that 
> > flaw, you'd evolve your attempt into something nearing what I wrote.
> 
> No, not really. I still am afraid that you're missing some point of how
> Python interpreter works but considering the fight you put protecting
> 'python /usr/bin/c-file', I won't continue that topic until you finally
> answer my questions wrt that file.

Either your english sucks, or you're not understanding your proffered 
solution doesn't work... or you're not going to acknowledge it 
doesn't actually work.  Honestly not sure which considering the 
insanity of your claims here.

You propose, using trial for this example, to have trial-{2.7,3.1,3.2} 
be interpretter specific versions of the script, and  and to ln -s 
/path/to/python-exec /usr/bin/trial

That means /usr/bin/trial is an elf binary.

This means, regardless of what crack-adled fricking claim you make 
about my cpython knowledge... python /usr/bin/trial, which resolves 
to, `python /usr/bin/python-exec`... ain't going to work.  Ever.  
Never ever.

Ironically, floppym pointed out this sort of flaw in my initial brain 
dump of a symlink approach (which lead to the python-shebang 
approach, emails you responded to but aparently didn't fix in your 
own proposal).  You were active in that thread, trying pretty heavily 
to call the proposal hack/horrible/crappy/a meany, you should've 
caught this flaw and dealt w/ it in your own implementation.


> By the way, I doubt it is a 'core usage case'. Maybe just because it
> never worked as intended, maybe because people simply don't do that
> usually because people usually don't care whether stuff in /usr/bin
> is Python, C or shell.

re-execing and invoking scripts w/in a source via using sys.executable 
is standard.  If in doubt, go sample the python universe.  Hell, think 
through standard re-exec; what you've got implemented here doesn't 
preserve argv[0] as the raw script name (which is a regression from 
existing solutions); were it to do so, you'd be triggering this fault 
via re-exec.

Even if you wanted to make such a claim, you're ignoring that python 
/usr/bin/trial works *now*, and you'd be breaking that.  Meaning 
minimally a regression of what frankly, is insane to break (perl 
/usr/bin/spamassassin shouldn't fail anymore than python 
/usr/bin/emerge should fail).


> > 2) Everytime there is a new python interpretter, python-exec has to 
> > be re-built updating it's hardcoded list of interpretters.  Goes without 
> > saying, this sucks a fair bit; your solution there likely would be 
> > walk env.d/python/* in some fashion extracting the list of targets (or 
> > walk /usr/bin/python* and run into problems w/ the names used there 
> > now matching EPYTHON usage).  End result, you'd evolve it into 
> > something nearing what I did w/ shebangs.
> 
> How often does this happen? Is this really problematic?

It's a sign that your solution is hacky, frankly.

We shouldn't have to rebuild binaries for configuration data 
changing- you may not like it, but that's usually a sign of bad code.


> And how an user can disable a particular (say, problematic) Python
> implementation globally in your solution? Without reinstalling all
> the packages which provide scripts for it?

Depends on your definition of 'disable'.  With your solution, the only 
way to disable the interpretter in question is thus

1) Unmerge the interpretter
2) mangle the python abi list for python-exec via rebuilding it.

However.  #2 carries with it one helluva of an unstated gotcha; that 
every single time the available interpretters change, it has to be 
recompiled, or the full list of targets has to be passed in every 
time.  Both of those are arbitrary... lets call it 'hobbiest' 
limitations that aren't production worthy.

Either way, in my solution, if we wanted users to be able to have 
pypy-1.8 installed, have things compiled for it, but blacklist it 
after the fact (while leaving the interpretter installed) I'd just add 
a file into env.d listing the blacklisted interpretters, and update 
python-shebang accordingly.

Keep in mind since the patchset targets python-eselect, deploying that 
is an easy shot.

I'm not particularly convinced you're presenting a real world 
scenario however.  Regardless, it's easily addressed via my solution.


> > 3) It's a hardcoded lookup list that is forced for all scripts; if 
> > that list is python3.2,python3.1,python2.7, and the active python 
> > version doesn't intersect that (say jython-2.5), your approach forces 
> > 3.2.  My solution actually allows the script, in the absense of an 
> > enforcement via active python or EPYTHON, to use it's own preferred 
> > ordering.  Using pkgcore as an example, we prefer py2k execution over 
> > py3k- py2k is faster since the extensions are enabled.  This 
> > capability doesn't exist in your solution, nor the existing wrapper; 
> > it's a useful byproduct of how my solution is structured.  Etc etc, 
> > you know the response; where you'll go with this sooner or later is 
> > where python-shebang is already.
> 
> And, hmm, are you the first person requesting such a feature over
> the whole lifetime of python-wrapper?

I'm the first person who got pissed enough to address it.  Most of the 
time it works fine- but when you invoke something like trial or 
sphinx-build, things that analyze *other* python namespaces, it 
breaks.

It's inarguably broken; arfrever's proposed solution to it when I 
pointed it out was to hardcode gentooisms into the solution- I don't 
agree with that, fix it properly as I've done.


> I believe that hardcoded list is *simpler*. If user enabled jython-2.5
> (suicidal type?), then he's aware of the consequences. And in my
> solution, preferred Python 2 will be used. Then preferred Python 3.
> Then Python 2.7, 2.6, 2.5... then 3.2. It is predictable.
> 
> In your solution, it is completely unpredictable. Developer can
> (supposedly) do whatever he wants. Most of the devs probably won't use
> that, except for a few cases where users will simply hit (if ever)
> an unexpected behavior.

Fact is, the option is opened up via the shebang approach to solving 
this.

You're advocating that the fallback choice should be user controlled.  
Frankly... that's the wrong approach.

If a script is invoked- offlineimap, iotop, cvs2*, btrfs-bcp, 
revdep-rebuild, etc; the user actually rarely cares which 
implementation is used.  Up front, they decided which implementations 
they wanted to allow- but w/in that, sans the active interpretter, 
the user does not *care*.

That's just fact.  A user won't give a shit if it's offlineimap 2.7 or 
offlineimap 3.2.   The implementation however *may* have a preference 
there- pkgcore/snakeoil aren't the only pkgs that have py2k extensions 
but not py3k extension.  If you looked at that level of python, you'd 
recognize the API changes weren't exactly minor.  Doable, but a pita.

The kicker here is if we wanted users to be able to override the 
defaults... it's doable with my solution; just extend python-shebang.

That said, your solution provides no way to enumerate the pkgs own 
preferences- not without shoving that information into a new file 
(yet another file required in comparison to my solution).


> > 5) Via going with a standalone package from eselect-python, you're 
> > introducing the potential for the two to be out of sync/incompatible.  
> > Addressable via folding it inline into eselect-python (as I did), but 
> > it's a con against your solution in it's current claimed form.  Same 
> > thing, you'll do what I did sooner or later since it's the better 
> > approach.
> 
> It's a better development practice to split things.
> You should've known that by now.
> In any case, python-exec can work *without*
> eselect-python. You can try it if you don't believe.

I didn't claim it couldn't; I'm pointing out such a separation has no 
gain, but has costs.

Python-wrapper already exists in python-eselect; python-wrapper is 
dependent on the internal details of how eselect python stores it's 
configuration.  This is fine- python-wrapper can be kept lockstep up 
to date with eselect pythons internals.

Your solution however cannot be, not without leveling deps to force it 
to be lock step (and the transition there will require compatibility 
code left in eselect python).  If you don't see this, then I suggest 
you try modifying eselect python to move the config file, or change 
the semantics of how/what it stores.

Regardless, there's zero reason to separate them.  Separation is done 
when it makes sense, not for random shits and giggles.

python-shebang is effectively a generalized/sane python-wrapper; 
python-wrapper itself is just a re-exec to python-shebang after all.  
The proper place, and the place it will wind up at (regardless of your 
bitching) is eselect python.


> > 7) My solution doesn't hardcode /usr/bin/* pathways.  User sticks a 
> > python2.7 into /usr/local/ (for whatever reason they choose to do so), 
> > it will honor that.
> 
> Errr? Where does your suggestion come from?
>
> As far as I'm aware, user can put into per-implementation copy whatever
> he wants. Including a shell script, a C executable or whatever. It will
> work.

Your solution is reliant on the hardcoded shebang w/in each script; 
meaning /usr/bin/python* or whatever it may be.

Literally, sphinx-build-2.7, it has the hardcoded pathway 
#!/usr/bin/python2.7

Now, if you wanted that to not be hardcoded, it would be
#!/usr/bin/env python2.7

But in doing so, you no longer can pass arguments to the interpretter.

> With your, it won't.

If you're going to make such claims, at least bother to read the 
source please.  My solution searches path; meaning if it's trying to 
find python2.7, it'll search the path for it (same basic logic 
python-wrapper uses).

Kicker is, it's a minor point in favor of python-shebang.  Can either 
ignore the point, or make it into a bigger issue... your choice.


> As far as I'm aware, you solution can't handle per-implementation
> Python options (command-line arguments) either. In mine, it's as simple
> as dropping them onto the shebang.

If you'd read the email you're responding to, you'd see that per 
implementation python options are supported- and work fine now.

Look at the arg parsing bit I posted; or go read the !#@*ing source 
already rather than immediately clicking reply.


> > 8) Your solution doesn't work in the context of being used for 
> > /usr/bin/python wrapper; it's close, but as you yourself said, it 
> > requires symlinks (python-python2.7).
> 
> Or adding a conditional which your solution simply does. So, where are
> you going with this point? To the fact that I'm asking others for
> opinion while you're ignoring their existence?

What conditional am I adding?  

Your solution isn't usable *now* as a drop in for python-wrapper; it 
would have to be python-python2.7.

In the same vane, you're implementation actually requires scripts to 
be named trial-python2.7 trial-python3.2 etc.

That's not "asking peoples views"; that's the reality of your 
implementation.


> > 9) No tests.  Every. Fucking. Pathway. is tested with what I wrote.  
> > Python-exec lacks that, else issue #1 would've been spotted up front.
> 
> Great. Awesome. You are so awesome I can't even spell it.
> 
> Now feel free to grep the bugzie. That 'issue #1' is known. So far,
> I haven't get anyone to say he believes that's really important
> or useful. 

Your implementation mandates the following:

*) scripts be changed from sphinx-build-2.7 to sphinx-build-python2.7.  

*) mandates symlinks of python-EPYTHON were it to be attempted to be 
used for python-wrapper.

*) removes the ability to do $PYTHON_INTERP /usr/bin/script

These are the sort of things that would've been identified with tests.  

Meaning you're "lets start using python-exec" emails would've been a 
helluva lot more accurate in those cases.


Pretty much, thus:
*) python-exec regresses from existing standard wrapper- you no longer 
can do python /usr/bin/sphinx-build and it requires a recompile 
everytime a new python interpretter is installed.
*) python-shebang has no regressions from the standard, fixes the 
python2.7 /usr/bin/sphinx-bulid scenario, and decreases the number of 
files on disk.

pretty much, that's the end of conversation with you mgorny.

Python herd, assuming you've not ignored the thread due to the noise, 
your views would be useful.
~harring

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Mike Gilbert]

On Sat, Nov 3, 2012 at 7:47 AM, Brian Harring <ferringb@gmail.com> wrote:

Some quick notes on these last points. Don't get too hung up on this;
my general opinion of the solution follows at the end.

> Your implementation mandates the following:
>
> *) scripts be changed from sphinx-build-2.7 to sphinx-build-python2.7.
>

The was a design decision to simplify the implementation in
conjunction with the new eclass and avoids having to map a plain
number (2.7) to an actual interpreter name (python2.7). I still like
this idea, but it really isn't safe to do this in the old eclass.

> *) mandates symlinks of python-EPYTHON were it to be attempted to be
> used for python-wrapper.
>

python-exec was not originally intended to replace python-wrapper so
there is no code in place to handle this more gracefully.

> *) removes the ability to do $PYTHON_INTERP /usr/bin/script
>

Nobody dreamed up this scenario when python-exec was proposed. It
seems pretty obvious now, but I didn't think about it until our email
exchange about your original proposal.

> These are the sort of things that would've been identified with tests.
>

I don't see how tests themselves would have identified these issues,
since we are talking about testing use cases that had not yet been
identified. Or that had deliberately been changed, in the case of the
script rename.

I guess you may be implying that thinking of test cases would have
triggered some deeper thought and revealed flaws that were not
immediately apparent. That's probably true.


> Python herd, assuming you've not ignored the thread due to the noise,
> your views would be useful.

Just so everyone is on the same page here: python-exec was invented as
an answer to the problem of how to implement script wrappers in the
new eclass. The new eclass(es) were invented because nobody wanted to
make significant modifications to the old one, given its complexity
and code style.

I don't think we were aiming for a perfect solution with python-r1 and
python-exec, rather one that was simple so people could understand,
and worked in the cases we considered at the time.

Your main goal with python-shebang seems to be a fix to the "python2.7
/usr/bin/sphinx-build" thing, which is fine. But it lead you to a
different place.


That said, I like this python-shebang solution, and your replies to
mgorny's questions have provided very useful insight.

However, I have a few issues:

1. I still don't really understand the guts of python-shebang.

It's a significant chunk of C that is doing things I am not yet
familiar with. I'm sure I can figure it out if I take some time, but
right now I have a "fear of the unknown" thing happening in my head.

I don't want us to end up with only you understanding how to modify that code.


2. Somebody needs to write the python.eclass code if we really want to go there.

I think we would need to modify
python_merge_intermediate_installation_images, and replace
python_generate_wrapper_scripts. If anyone wants to take a crack at
it, great.


3. We need to reconcile the script naming convention so we can use
this with the new eclass as well.

I just don't see us implementing PYTHON_TARGETS in python.eclass. I
have considered it, but I just don't have the motivation to do it. I
still think we are going to proceed with the python-r1 effort.

What do you think about extending python-shebang to understand the
script-$EPYTHON naming convention in addition to the
script-$PYTHON_ABI convention? I only ask because I dislike the
$PYTHON_ABI values and would rather not see that re-implemented in
python-r1. I think script-$EPYTHON gives a clearer indication as to
its intended purpose anyway.

Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version interpreter preference in a file as well.

[Brian Harring]

On Sat, Nov 03, 2012 at 11:55:37AM -0400, Mike Gilbert wrote:
> On Sat, Nov 3, 2012 at 7:47 AM, Brian Harring <ferringb@gmail.com> wrote:
> 
> Some quick notes on these last points. Don't get too hung up on this;
> my general opinion of the solution follows at the end.
> 
> > Your implementation mandates the following:
> >
> > *) scripts be changed from sphinx-build-2.7 to sphinx-build-python2.7.
> >
> 
> The was a design decision to simplify the implementation in
> conjunction with the new eclass and avoids having to map a plain
> number (2.7) to an actual interpreter name (python2.7). I still like
> this idea, but it really isn't safe to do this in the old eclass.

Renaming sphinx-build-2.7-pypy-1.8 to sphinx-build-pypy-c1.8 (or just 
1.8) personally I think is sane; the original naming was off, and I 
doubt any code actually hardcodes it.

I don't see the point in shoving -pythonx.y when -x.y 
suffices however- the parsing scenario there is easy to handle for 
any wrapper, and is understood for folks these days.

Additionally, because the advocated route of forcing a specific interp 
for a target has thus been invoking the interpretter specific script 
(sphinx-build-2.7 for example), changing the name for python means 
breakage may occur.

How much breakage, I've no clue; just pointing out that it is 
possible.


> > *) mandates symlinks of python-EPYTHON were it to be attempted to be
> > used for python-wrapper.
> >
> 
> python-exec was not originally intended to replace python-wrapper so
> there is no code in place to handle this more gracefully.

It was proposed, which is why I responded to it.

Same angle, the cons of python-exec compared to python-shebang were 
asked, so this got brought up.  python-wrapper is literally just a a 
simple re-exec around python-shebang now.


> > Python herd, assuming you've not ignored the thread due to the noise,
> > your views would be useful.
> 
> Just so everyone is on the same page here: python-exec was invented as
> an answer to the problem of how to implement script wrappers in the
> new eclass. The new eclass(es) were invented because nobody wanted to
> make significant modifications to the old one, given its complexity
> and code style.
> 
> I don't think we were aiming for a perfect solution with python-r1 and
> python-exec, rather one that was simple so people could understand,
> and worked in the cases we considered at the time.
> 
> Your main goal with python-shebang seems to be a fix to the "python2.7
> /usr/bin/sphinx-build" thing, which is fine. But it lead you to a
> different place.

Close; my main goal was replacing that fucking python wrapper, and 
fixing `python2.7 /usr/bin/sphinx-build` while speeding everything up.

I've got a couple of core requirements, but they pretty much sum up to 
"the existing wrapper needs to die"- everything else (de-duplication 
reducing the number of files on disk) is icing on the cake.


> That said, I like this python-shebang solution, and your replies to
> mgorny's questions have provided very useful insight.
> 
> However, I have a few issues:
> 
> 1. I still don't really understand the guts of python-shebang.
> 
> It's a significant chunk of C that is doing things I am not yet
> familiar with. I'm sure I can figure it out if I take some time, but
> right now I have a "fear of the unknown" thing happening in my head.
> 
> I don't want us to end up with only you understanding how to modify that code.

That can be rectified via more comments.  Honestly it's pretty simple- 
just trace the main(); everything else is just helpers to it.  The 
flow there should be pretty clear as to what it's doing.

The actual invocation scenarios, and details of how it covers each of 
those, can be addressed via a README (or similar) easily enough.  
There's a few subtlies there as to how it manages to do the correct 
thing, which I should document.


> 2. Somebody needs to write the python.eclass code if we really want to go there.
> 
> I think we would need to modify
> python_merge_intermediate_installation_images, and replace
> python_generate_wrapper_scripts. If anyone wants to take a crack at
> it, great.

Already have local patches to that effect; as indicated in my original 
email, I'm busy with some other things atm, but I'll loop back to it 
in the next week or two.

If someone else wants to finish it, they're free to.  Else I will.


> 3. We need to reconcile the script naming convention so we can use
> this with the new eclass as well.
> 
> I just don't see us implementing PYTHON_TARGETS in python.eclass. I
> have considered it, but I just don't have the motivation to do it. I
> still think we are going to proceed with the python-r1 effort.
> 
> What do you think about extending python-shebang to understand the
> script-$EPYTHON naming convention in addition to the
> script-$PYTHON_ABI convention? I only ask because I dislike the
> $PYTHON_ABI values and would rather not see that re-implemented in
> python-r1. I think script-$EPYTHON gives a clearer indication as to
> its intended purpose anyway.

Off the top of the head, it's easy enough to support both w/ 
python-shebang code; it's just an extra case or two for the core 
parsing.

I *do* intend to keep the current parsing in place though- as I said, 
I want this idiocy fixed in python.eclass now (and as demonstrated, 
I'll do the work myself if necessary).  Removing that blob of code out 
of python.eclass won't fix the eclass, but it'll at least bring down 
the line count a bit so there's less to have to work with.

Plus if a replacement actual ground, this component doesn't need 
changing (if it ain't broke, don't fix it).

~harring