From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <gentoo-python+bounces-202-garchives=archives.gentoo.org@lists.gentoo.org>
Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80])
by finch.gentoo.org (Postfix) with ESMTP id 4BA2D1381F3
for <garchives@archives.gentoo.org>; Sat, 3 Nov 2012 03:35:51 +0000 (UTC)
Received: from pigeon.gentoo.org (localhost [127.0.0.1])
by pigeon.gentoo.org (Postfix) with SMTP id 4DC80E0417;
Sat, 3 Nov 2012 03:35:42 +0000 (UTC)
Received: from mail-ie0-f181.google.com (mail-ie0-f181.google.com [209.85.223.181])
by pigeon.gentoo.org (Postfix) with ESMTP id 56E4EE0417
for <gentoo-python@lists.gentoo.org>; Sat, 3 Nov 2012 03:35:40 +0000 (UTC)
Received: by mail-ie0-f181.google.com with SMTP id 16so6009880iea.40
for <gentoo-python@lists.gentoo.org>; Fri, 02 Nov 2012 20:35:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=date:from:to:cc:subject:message-id:references:mime-version
:content-type:content-disposition:in-reply-to:user-agent;
bh=22ZCdwyE7HubaJ9RhffFVVd++XEK0FkprXcqSztV5uI=;
b=j+DZBVSUt36boRF0aG0kkyKeykd6EeOD3AQb5ry+vzmDyH73Dvjz76STwkz58uMBYn
mZc8G6DSZ4Js59AEWs6roVNFK3hoZ3V9azGxDHNoEIKe7SYECvPcvW4JHr/dVrFdFm68
6/MAkejZuThcT2IZQnf/347gnoyiOfXaZbs9QmrM7b2MwIHK4LfCDkcoujBCiR9o7AZA
AKpqsCtlgn0CRmoWVVPV59KUTi28oFGzCEhqFlVOIKnW8pK0Llub+CISZHP0EC6aknJM
hDK8m2kw00ZZ3eLyxJhxbIKsoWnAamafdF5eVzHiHI38Az5GHl8eqKTiLxtEvDigCD40
/O0Q==
Received: by 10.42.68.203 with SMTP id y11mr3265027ici.26.1351913740674;
Fri, 02 Nov 2012 20:35:40 -0700 (PDT)
Received: from smtp.gmail.com:587 ([2620:0:1000:fd16:224:d7ff:feae:c014])
by mx.google.com with ESMTPS id 7sm752983igh.0.2012.11.02.20.35.38
(version=TLSv1/SSLv3 cipher=OTHER);
Fri, 02 Nov 2012 20:35:39 -0700 (PDT)
Received: by smtp.gmail.com:587 (sSMTP sendmail emulation); Fri, 02 Nov 2012 20:35:38 -0700
Date: Fri, 2 Nov 2012 20:35:38 -0700
From: Brian Harring <ferringb@gmail.com>
To: Micha?? G??rny <mgorny@gentoo.org>
Cc: Mike Gilbert <floppym@gentoo.org>, gentoo-python@lists.gentoo.org
Subject: Re: [gentoo-python] Re: [PATCH eselect-python 1/2] Store per-version
interpreter preference in a file as well.
Message-ID: <20121103033538.GI3299@localhost>
References: <1351770893-1217-1-git-send-email-mgorny@gentoo.org>
<CAJ0EP41Ww9GKYto8A8gX-L+D2=3+MFhYHmUZXZNvm+Ni5ApSbw@mail.gmail.com>
<CAJ0EP42u9vNaC70ui-YPjRR1SOY9n8fCmnXA1EgijmRdsRdK8Q@mail.gmail.com>
<20121101222719.2717ff05@pomiocik.lan>
<20121101224242.GE3299@localhost>
<20121102100305.4acfc927@pomiocik.lan>
Precedence: bulk
List-Post: <mailto:gentoo-python@lists.gentoo.org>
List-Help: <mailto:gentoo-python+help@lists.gentoo.org>
List-Unsubscribe: <mailto:gentoo-python+unsubscribe@lists.gentoo.org>
List-Subscribe: <mailto:gentoo-python+subscribe@lists.gentoo.org>
List-Id: Discussions centering around the Python ecosystem in Gentoo Linux <gentoo-python.gentoo.org>
X-BeenThere: gentoo-python@gentoo.org
X-BeenThere: gentoo-python@lists.gentoo.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20121102100305.4acfc927@pomiocik.lan>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Archives-Salt: b1f97e02-9c78-43bb-bba0-847070f79933
X-Archives-Hash: ad2687b97a68951a36d28726ded4353b
On Fri, Nov 02, 2012 at 10:03:05AM +0100, Micha?? G??rny wrote:
> On Thu, 1 Nov 2012 15:42:42 -0700
> Brian Harring <ferringb@gmail.com> wrote:
>
> > On Thu, Nov 01, 2012 at 10:27:19PM +0100, Micha?? G??rny wrote:
> > > On Thu, 1 Nov 2012 14:48:55 -0400
> > > Mike Gilbert <floppym@gentoo.org> wrote:
> > >
> > > > On Thu, Nov 1, 2012 at 11:44 AM, Mike Gilbert <floppymaster@gmail.com> wrote:
> > > > > On Thu, Nov 1, 2012 at 7:54 AM, Micha?? G??rny <mgorny@gentoo.org> wrote:
> > > > >> The idea is very simple: /etc/env.d/python/python[23] with a one-line
> > > > >> value similar to the main interpreter /config file.
> > > > >>
> > > > >> That should be simpler & more reliable than reading a symlink. And at
> > > > >> some point we can replace the symlink with an $EPYTHON-aware wrapper
> > > > >> as well.
> > > > >
> > > > > I don't understand the point of this. Do we have some need to enable
> > > > > EPYTHON usages for scripts that have a python2 or python3 shebang?
> > > > >
> > > > > I also don't understand how a text file is more reliable than a
> > > > > symlink; they are basically the same thing, but the symlink has a
> > > > > different file mode.
> > >
> > > Ah, and I'd forget. I have the following dream:
> > >
> > > /etc/env.d/python/python2
> > > /etc/env.d/python/python3
> > > /etc/env.d/python/config -> python2
> > >
> > > So, python{2,3} keep the per-version interpreters, and config is just
> > > a symlink to one of them. Two bird with one stone -- readlink to get
> > > which group is enabled, cat to get the exact interpreter. I'm proud
> > > of myself!
> >
> > I too have a dream; git://pkgcore.org/eslect-python .
> >
> > That's a shebang based version of what I proposed a while back. It
> > works now and has tests. It's written to be basically a drop in for
> > existing python.eclass generation of shebangs, w/ the
> > hardlinking/de-duplication/farking-fast/fix python3.2
> > /usr/bin/sphinx-build scenario.
>
> Could you please provide an explanation of what it does? I mean, a few
> points of what happens when you run it in various scenarios. That's 600
> lines of C code, I think Python folks would appreciate not having to
> read that line by line.
Grumble. Don't think you read my commit messages at all, just
responded looking for points to claim "it sucks".
What I have written here is a python-shebang based solution. That
shebang target includes the EPYTHON's it supports.
For the usual sphinx-build example, sphinx-build-2.7 goes from
#!/usr/bin/python2.7
to
#!/usr/bin/python-shebang python2.7,python3.2
This means that sphinx-build-2.7 and sphinx-build-3.2 are now the same
content. They can be hardlinked together (space savings).
This additionally means that sphinx-build can be hardlinked to the
same underlying inode. Why? Because python-shebang is smart enough
to recognize the context it is invoked in.
This additionally means that the follow commands:
eselect python set 3.2
python2.7 /usr/bin/sphinx-build
properly work- invoking sphinx-bulid in a 2.7 context (for anyone
trying to write distutils/setup bits that aren't gentoo/EPYTHON
specific, this is fricking useful to say the least).
python-shebang does thus:
1) parse argv[1]; split it first on ' ' (everything following is split
by ' ', and used as args for the python interp invoked). The leftmost
is split on ','; this is the list of interpretter targets this script
was installed for.
2) identify the context it was invoked via. If invoked via an interp
specific pathway (aka, /usr/bin/sphinx-build-2.7; a forced python2.7
context) then it honors that, and re-execs to python2.7 $0 w/ the
additional interp args. That re-exec fail, it does the usual failure
return.
3) Making it here means it was invoked in a non forced interp context;
meaning /usr/bin/sphinx-build. If the installation required EPYTHON
to be honored always (this actually is used if you analyzed the tree),
w/in the interp targets there is a EPYTHON value- literally,
"EPYTHON". If EPYTHON env var is set, and EPYTHON is set, force that
interpretter or fail.
3) If EPYTHON is set but the script wasn't marked as "honor EPYTHON
always", check the list of interp targets, if we can the EPYTHON
target, re-exec to it.
4) Making it here means that env EPYTHON wasn't forced, nor can it be
honored. Grab the active python version, re-exec to that if it's a
support interpretter for this script.
5) If it ain't, loop over the list of supported targets, l->r, trying
to re-exec in that ordering.
> > The remaining work for that is thus:
> >
> > 1) If the EPYTHON targets aren't given via shebang arg, then it needs
> > to fallback to grabbing the targets from the file (easy enough).
> >
> > 2) Add a few helpers/wrappers to make it easier to do the
> > deduplication/shebang rewriting.
>
> How would you perform deduplication while you're at it? Could you
> measure or at least consider the benefits of it?
I described this in the original dev ml thread you responded to
bitching about symlinks; rewrite the shebang, md5 comparison, what's
the same is hardlinked; if all are the same, hardlink the wrapper
(meaning if sphinx-build-{2.7,3.2} are the same md5, hardlink
sphinx-build to the same underlying inode).
It's pretty simple.
> > Frankly, you should be looking at this imo, rather than trying
> > standalone files. Yes, files is simpler- but you'll wind up sooner or
> > later rebuilding it into what I already built out here.
>
> I think you are missing the point I'm raising here. Mostly because your
> python-wrapper doesn't handle per-version preferences like python-exec
> does, unless I'm missing something.
You're misunderstanding what was written; hell, I even referenced
that it supports EPYTHON preference in #5 (java-config cycle
breaking).
> > """
> > Add a python-shebang utility, slave python-wrapper to it.
> >
> > Going forward, the intent is to push the PYTHON_ABIs of a given
> > script down into it's shebang, pointing the shebang at python-shebang.
> >
> > In this way, all known/supported abi's are available at the time of
> > execution; further, if the target is told to respect-EPYTHON (meaning
> > no searching for something to execute, either active python version or
> > EPYTHON var), we can push this down into that list and have that code
> > handle it.
> >
> > [...]
> >
> > to the simpler form of:
> > for x in sphinx{,-{2.7,3.2}}; do echo > $x <<EOF
> > #!/usr/bin/python-shebang python2.7,python3.2
> > <code>
> > EOF
> > done
>
> Hmm, are you assuming that /usr/bin/sphinx carries the actual code?
> Or some special wrapper code?
>
> Not to mention you still bind the whole thing to installed package.
> If you need to fix those files, you need to reinstall all packages
> installing them.
This critique makes no sense... and I'm pretty sure that any point you
may manage to come up with, will apply directly to python-exec just as
equally.
Either clarify your claim, or drop the noise there.
> > The gains of this are thus:
>
> Could you please compare it to the modern solution (python-exec) rather
> than the deprecated one? You are no longer on the uncharted waters.
Respectfully, your 'modern solution' (vs deprecated) is purely in your
head; distutils/python eclasses of your making aren't the tree.
They're your playground right now- one that you're pushing, but I've
yet to see real traction on. That's just reality; you may think it's
a done deal (or will be), but the on the ground reality right now-
including the pushback you've been getting from people in certain
cases- makes me unwilling to pin my hopes on your work.
Either way, you asked, so here's the list of python-exec con's in
comparison to python-shebang:
1) Your solution actually doesn't work as a fricking wrapper for a
core usage case; `python /usr/bin/sphinx-build`. That perfectly
innocuous/valid invocation would result in the interpretter trying
to execute an elf binary. That pretty much renders your solution as a
no go right from the get go. Your only way to fix that is to have
sphinx-build be a python implementation that does wrapping, and use a
shebang of python-exec (so you can maintain speed for direct
/usr/bin/sphinx-build invocations); meaning once you spotted that
flaw, you'd evolve your attempt into something nearing what I wrote.
2) Everytime there is a new python interpretter, python-exec has to
be re-built updating it's hardcoded list of interpretters. Goes without
saying, this sucks a fair bit; your solution there likely would be
walk env.d/python/* in some fashion extracting the list of targets (or
walk /usr/bin/python* and run into problems w/ the names used there
now matching EPYTHON usage). End result, you'd evolve it into
something nearing what I did w/ shebangs.
3) It's a hardcoded lookup list that is forced for all scripts; if
that list is python3.2,python3.1,python2.7, and the active python
version doesn't intersect that (say jython-2.5), your approach forces
3.2. My solution actually allows the script, in the absense of an
enforcement via active python or EPYTHON, to use it's own preferred
ordering. Using pkgcore as an example, we prefer py2k execution over
py3k- py2k is faster since the extensions are enabled. This
capability doesn't exist in your solution, nor the existing wrapper;
it's a useful byproduct of how my solution is structured. Etc etc,
you know the response; where you'll go with this sooner or later is
where python-shebang is already.
5) Via going with a standalone package from eselect-python, you're
introducing the potential for the two to be out of sync/incompatible.
Addressable via folding it inline into eselect-python (as I did), but
it's a con against your solution in it's current claimed form. Same
thing, you'll do what I did sooner or later since it's the better
approach.
6) With N=# of python implementations, you're forcing N copies of that
script. My solution will de-dupe it down to where the content is the
same; most scripts, that means all script-* and script collapses down
to one single inode. If py3k content differs from py2k, and it's
installed for py2k/py3k, this means in that case 3 inodes. Your
solution has no way to avoid the inode/space overhead, not w/out
switching the shebang.
7) My solution doesn't hardcode /usr/bin/* pathways. User sticks a
python2.7 into /usr/local/ (for whatever reason they choose to do so),
it will honor that.
8) Your solution doesn't work in the context of being used for
/usr/bin/python wrapper; it's close, but as you yourself said, it
requires symlinks (python-python2.7).
9) No tests. Every. Fucking. Pathway. is tested with what I wrote.
Python-exec lacks that, else issue #1 would've been spotted up front.
What you've got only works for direct invocation of a target via
/usr/bin/sphinx-build; python /usr/bin/sphinx-build won't fly; nor can
it be used as the the python wrapper w/out changing how we install the
interpretter.
Python shebang however works all existing scenarios, and fixes the
core scenario that got me started here; `python2.7
/usr/bin/sphinx-build`.
The only flaws with it at this point are thus:
1) There is a 127 char limit on shebang length. This is addressable
via having python-shebang reach grab the list from the target itself
if that limit is encountered.
2) fleshing out the tools for doing de-duplication; it's easy enough,
I've just been busy with other things, and this code was done as a
side project for entertainment. I'll finish it soon enough (else if
someone wants to sort the remaining issues, I can detail exactly how).
The nice part about my solution here is that it's not something that
is hinged mgorny's python/distutils eclasses becoming the norm; it's
usable now for existing python eclass (simple enough to plug it in),
and any followup solution would be insane not to use it.
~harring