[gentoo-user] bash stopped running python scripts...

[gentoo-user] bash stopped running python scripts...

[Mike Edenfield]

I dunno what I did, but I've managed to break python shell scripts, 
which of course is playing havoc with portage.  Bash no longer wants to 
execute the scripts with python as the interpreter, but insists on 
executing them as bash scripts.  Python itself is still functioning 
properly, when invoked directly.   Since I happen to have ImageMagick on 
this machine, the result is this:

kutulu@apollo ~ $ cat test.py
#!/usr/bin/python
import sys
print "Python Ok."
kutulu@apollo ~ $ ./test.py
X connection to localhost:11.0 broken (explicit kill or server shutdown).
./test.py: line 3: print: command not found
kutulu@apollo ~ $ python ./test.py
Python Ok.
kutulu@apollo ~ $

Perl and Tcl both still work as expected, and by invoking python 
directly I can still get portage to merge things.  For example:

root@apollo ~ $ python -O /usr/lib/portage/bin/ebuild 
/usr/portage/dev-lang/python/python-2.6.2-r1.ebuild install

will re-merge python successfully, which I have now done for both python 
and bash, to no effect.

Any ideas?

--Mike

Re: [gentoo-user] bash stopped running python scripts...

[Alan McKinnon]

On Monday 03 August 2009 22:56:51 Mike Edenfield wrote:
> I dunno what I did, but I've managed to break python shell scripts,
> which of course is playing havoc with portage.  Bash no longer wants to
> execute the scripts with python as the interpreter, but insists on
> executing them as bash scripts.  Python itself is still functioning
> properly, when invoked directly.   Since I happen to have ImageMagick on
> this machine, the result is this:
>
> kutulu@apollo ~ $ cat test.py
> #!/usr/bin/python
> import sys
> print "Python Ok."
> kutulu@apollo ~ $ ./test.py
> X connection to localhost:11.0 broken (explicit kill or server shutdown).
> ./test.py: line 3: print: command not found
> kutulu@apollo ~ $ python ./test.py
> Python Ok.
> kutulu@apollo ~ $
>
> Perl and Tcl both still work as expected, and by invoking python
> directly I can still get portage to merge things.  For example:
>
> root@apollo ~ $ python -O /usr/lib/portage/bin/ebuild
> /usr/portage/dev-lang/python/python-2.6.2-r1.ebuild install
>
> will re-merge python successfully, which I have now done for both python
> and bash, to no effect.
>
> Any ideas?
>
> --Mike

Did you recently merge python-3 and were so foolish as to make it the 
default?

What is /usr/bin/python? and what version is it (-V)?

-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: bash stopped running python scripts...

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

Mike Edenfield wrote:
> I dunno what I did, but I've managed to break python shell scripts, 
> which of course is playing havoc with portage.

http://bugs.gentoo.org/show_bug.cgi?id=279915

The whole issue seems to be handled quite strangely IMO. You would think
breaking Python for all ~x86 is a major offense...

-- Remy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] bash stopped running python scripts...

[Mike Edenfield]

On 8/3/2009 5:03 PM, Alan McKinnon wrote:
> On Monday 03 August 2009 22:56:51 Mike Edenfield wrote:

>> kutulu@apollo ~ $ cat test.py
>> #!/usr/bin/python
>> import sys
>> print "Python Ok."
>> kutulu@apollo ~ $ ./test.py
>> X connection to localhost:11.0 broken (explicit kill or server shutdown).
>> ./test.py: line 3: print: command not found
>> kutulu@apollo ~ $ python ./test.py
>> Python Ok.
>> kutulu@apollo ~ $

> Did you recently merge python-3 and were so foolish as to make it the
> default?

I did emerge python-3, but then unmerged it almost immediately, and it 
was never the default.  Python was already broken when I merged python 
3.1, which I did to see if it fixed anything, which of course it didn't.

> What is /usr/bin/python? and what version is it (-V)?

root@apollo ~ # /usr/bin/python -V
Python 2.6.2
root@apollo ~ # cat /usr/bin/python
#!/bin/bash
# Gentoo Python wrapper script

[[ "${EPYTHON}" =~ (/|^python$) ]] && EPYTHON="python2.6"
"${0%/*}/${EPYTHON:-python2.6}" "$@"

Is that supposed to be that way?  I vaguely recall from my Tcl days that 
tclsh used to cause problems with the #! lines when it was a shell 
script, and that you had to use some odd exec trick to get tcl shell 
scripts to run.  Is that still true?

Looking back through my emerge.log it appears that the last thing to 
successfully run through emerge was eselect-python, if that makes a 
difference.

--Mike

Re: [gentoo-user] Re: bash stopped running python scripts...

[Mike Edenfield]

On 8/3/2009 5:14 PM, Remy Blank wrote:
> Mike Edenfield wrote:
>> I dunno what I did, but I've managed to break python shell scripts,
>> which of course is playing havoc with portage.
>
> http://bugs.gentoo.org/show_bug.cgi?id=279915
>
> The whole issue seems to be handled quite strangely IMO. You would think
> breaking Python for all ~x86 is a major offense...

Ah, thanks.  Usually my google searches pick up stuff from 
bugs.gentoo.org but this time it didn't, guess it was too new :\

And yes, reverting /usr/bin/python to a symlink instead of a shell 
script does solve the problem.  I'll just follow the bug for now and 
mask off eselect-python.

--Mike

Re: [gentoo-user] bash stopped running python scripts...

[Alan McKinnon]

On Monday 03 August 2009 23:22:08 Mike Edenfield wrote:
> On 8/3/2009 5:03 PM, Alan McKinnon wrote:
> > On Monday 03 August 2009 22:56:51 Mike Edenfield wrote:
> >> kutulu@apollo ~ $ cat test.py
> >> #!/usr/bin/python
> >> import sys
> >> print "Python Ok."
> >> kutulu@apollo ~ $ ./test.py
> >> X connection to localhost:11.0 broken (explicit kill or server
> >> shutdown). ./test.py: line 3: print: command not found
> >> kutulu@apollo ~ $ python ./test.py
> >> Python Ok.
> >> kutulu@apollo ~ $
> >
> > Did you recently merge python-3 and were so foolish as to make it the
> > default?
>
> I did emerge python-3, but then unmerged it almost immediately, and it
> was never the default.  Python was already broken when I merged python
> 3.1, which I did to see if it fixed anything, which of course it didn't.
>
> > What is /usr/bin/python? and what version is it (-V)?
>
> root@apollo ~ # /usr/bin/python -V
> Python 2.6.2
> root@apollo ~ # cat /usr/bin/python
> #!/bin/bash
> # Gentoo Python wrapper script
>
> [[ "${EPYTHON}" =~ (/|^python$) ]] && EPYTHON="python2.6"
> "${0%/*}/${EPYTHON:-python2.6}" "$@"
>
> Is that supposed to be that way?  I vaguely recall from my Tcl days that
> tclsh used to cause problems with the #! lines when it was a shell
> script, and that you had to use some odd exec trick to get tcl shell
> scripts to run.  Is that still true?

I have the identical file, it works here. 

> Looking back through my emerge.log it appears that the last thing to
> successfully run through emerge was eselect-python, if that makes a
> difference.

Your original post has a "X connection to localhost:11.0 broken" error, 
which is mighty unusual. The error is common enough, but has nothing to do 
with python.

Try some brief out-of-the-box tests:

1. Does test.py run OK from a virtual console?
2. Have you logged out and back in to X since merging python?
3. Have you re-merged python-2.6 just in case your sys lib is damaged?

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] bash stopped running python scripts...

[Mike Edenfield]

On 8/3/2009 5:48 PM, Alan McKinnon wrote:
> On Monday 03 August 2009 23:22:08 Mike Edenfield wrote:
>> On 8/3/2009 5:03 PM, Alan McKinnon wrote:
>>> On Monday 03 August 2009 22:56:51 Mike Edenfield wrote:
>>>> kutulu@apollo ~ $ cat test.py
>>>> #!/usr/bin/python
>>>> import sys
>>>> print "Python Ok."
>>>> kutulu@apollo ~ $ ./test.py
>>>> X connection to localhost:11.0 broken (explicit kill or server
>>>> shutdown). ./test.py: line 3: print: command not found
>>>> kutulu@apollo ~ $ python ./test.py
>>>> Python Ok.
>>>> kutulu@apollo ~ $
>>>
>>> Did you recently merge python-3 and were so foolish as to make it the
>>> default?
>>
>> I did emerge python-3, but then unmerged it almost immediately, and it
>> was never the default.  Python was already broken when I merged python
>> 3.1, which I did to see if it fixed anything, which of course it didn't.
>>
>>> What is /usr/bin/python? and what version is it (-V)?
>>
>> root@apollo ~ # /usr/bin/python -V
>> Python 2.6.2
>> root@apollo ~ # cat /usr/bin/python
>> #!/bin/bash
>> # Gentoo Python wrapper script
>>
>> [[ "${EPYTHON}" =~ (/|^python$) ]]&&  EPYTHON="python2.6"
>> "${0%/*}/${EPYTHON:-python2.6}" "$@"
>>
>> Is that supposed to be that way?  I vaguely recall from my Tcl days that
>> tclsh used to cause problems with the #! lines when it was a shell
>> script, and that you had to use some odd exec trick to get tcl shell
>> scripts to run.  Is that still true?
>
> I have the identical file, it works here.

That's very odd, as everything I've read over the past hour indicates 
that it's not *supposed* to work if you put a shell script in that line, 
but that's clearly not always true.

What's the #! line in your /usr/bin/emerge file?

> Your original post has a "X connection to localhost:11.0 broken" error,
> which is mighty unusual. The error is common enough, but has nothing to do
> with python.

I know where that particular error is coming from.  The first 
non-comment line in /usr/bin/emerge is "import sys".  Since bash is 
interpreting the script instead of python, when it gets to that line, it 
runs ImageMagick's "import" command.  It just indicates that bash 
ignored the #! line and kept going.

The bug Remy sent me to did have two working solutions: replacing the 
shell script with a symlink, or replacing "#!/usr/bin/python" with 
"#!/usr/bin/env python" in emerge/ebuild, both of which point the #! 
line at a real executable.  I'm now curious why that shell script works 
for anyone.

--Mike

Re: [gentoo-user] bash stopped running python scripts...

[Alan McKinnon]

On Tuesday 04 August 2009 00:08:48 Mike Edenfield wrote:
> > I have the identical file, it works here.
>
> That's very odd, as everything I've read over the past hour indicates
> that it's not supposed to work if you put a shell script in that line,
> but that's clearly not always true.

I didn't diff it, just compared yours and mine visually

> What's the #! line in your /usr/bin/emerge file?

$ head /usr/bin/emerge
#!/usr/bin/python
# Copyright 2006-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id: emerge 13131 2009-03-18 01:43:41Z zmedico $

import sys
# This block ensures that ^C interrupts are handled quietly.
try:
        import signal

>
> > Your original post has a "X connection to localhost:11.0 broken" error,
> > which is mighty unusual. The error is common enough, but has nothing to
> > do with python.
>
> I know where that particular error is coming from.  The first
> non-comment line in /usr/bin/emerge is "import sys".  Since bash is
> interpreting the script instead of python, when it gets to that line, it
> runs ImageMagick's "import" command.  It just indicates that bash
> ignored the #! line and kept going.

Ah, now stuff makes sense, like your reference to ImageMagick

> The bug Remy sent me to did have two working solutions: replacing the
> shell script with a symlink, or replacing "#!/usr/bin/python" with
> "#!/usr/bin/env python" in emerge/ebuild, both of which point the #!
> line at a real executable.  I'm now curious why that shell script works
> for anyone.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: bash stopped running python scripts...

[Graham Murray]

Remy Blank <remy.blank@pobox.com> writes:

> The whole issue seems to be handled quite strangely IMO. You would think
> breaking Python for all ~x86 is a major offense...

It did not break for all ~x86. I have 2 systems both running ~x86, both
have emerged (but not made active) python-3.1, /usr/bin/python is a bash
script on both yet emerge works with no problems. The only non-standard
things are that I have unmasked gcc-4.4.1 and am using git 2.6.31-rc
kernels. 

Re: [gentoo-user] Re: bash stopped running python scripts...

[Mike Edenfield]

On 8/4/2009 7:13 AM, Graham Murray wrote:
> Remy Blank<remy.blank@pobox.com>  writes:
>
>> The whole issue seems to be handled quite strangely IMO. You would think
>> breaking Python for all ~x86 is a major offense...
>
> It did not break for all ~x86. I have 2 systems both running ~x86, both
> have emerged (but not made active) python-3.1, /usr/bin/python is a bash
> script on both yet emerge works with no problems. The only non-standard
> things are that I have unmasked gcc-4.4.1 and am using git 2.6.31-rc
> kernels.

Aha.  I think I've discovered the "problem".  There was a change in the 
kernel execl() call between 2.6.27 and 2.6.28:

--- linux-2.6.27-hardened-r3/fs/binfmt_script.c 2008-10-09 
18:13:53.000000000 -0400
+++ linux-2.6.28-hardened/fs/binfmt_script.c    2008-12-24 
18:26:37.000000000 -0500
@@ -22,14 +22,15 @@
         char interp[BINPRM_BUF_SIZE];
         int retval;

-       if ((bprm->buf[0] != '#') || (bprm->buf[1] != '!') || 
(bprm->sh_bang))
+       if ((bprm->buf[0] != '#') || (bprm->buf[1] != '!') ||
+           (bprm->recursion_depth > BINPRM_MAX_RECURSION))
                 return -ENOEXEC;
         /*
          * This section does the #! interpretation.
          * Sorta complicated, but hopefully it will work.  -TYT
          */

-       bprm->sh_bang = 1;
+       bprm->recursion_depth++;
         allow_write_access(bprm->file);
         fput(bprm->file);
         bprm->file = NULL;


The kernel >= 2.6.28 now supports nesting up to 4 levels of script in 
the #! lines, if I'm reading that right, whereas < 2.6.28 it only 
supported 1 level.  I'll go update the b.g.o entry and try upgrading my 
kernel.  Though I dunno what that means for Gentoo/FreeBSD.

--Mike

[gentoo-user] Re: bash stopped running python scripts...

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 183 bytes --]

Mike Edenfield wrote:
> Though I dunno what that means for Gentoo/FreeBSD.

It means that relying on linux-only, non-POSIX compliant behavior is a
very bad idea ;-)

-- Remy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]