[gentoo-python] Testing dev-lang/python version bumps

[gentoo-python] Testing dev-lang/python version bumps

[Mike Gilbert]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

I have done some work this evening to integrate the latest versions of
dev-lang/python from Arfrever's Progress overlay into gentoo-x86.

He has done a nice job rebasing the patchsets and updating the ebuilds.

I took this opportunity to simplify the ebuilds a bit. Primarily this
meant stripping out the logic for pre-release snapshots. If anyone
really thinks we should keep this, please speak up.

Python-2.7.3 is already in the tree, but Arfrever indicates it has a
regression for C++ applications that link python. We should probably do
a revbump for that.

If we can get some people testing these that would be great. I would
like to add them to the tree sometime in the next week.


You may find the ebuilds here:

https://bitbucket.org/floppym/python-testing/src/default/dev-lang/python

Or you can clone the overlay for easy installation:

hg clone https://bitbucket.org/floppym/python-testing

The patchsets should be on the gentoo mirrors by the time you read this.
If not, you may find them here:

http://dev.gentoo.org/~floppym/python/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-python] Testing dev-lang/python version bumps

[Dirkjan Ochtman]

Thanks for doing this! Sorry it took so long to review them... we
should try to think of some easier review mechanism than putting up a
tarball you have to unpack.

On Sun, Apr 22, 2012 at 03:12, Mike Gilbert <floppym@gentoo.org> wrote:
> If we can get some people testing these that would be great. I would
> like to add them to the tree sometime in the next week.

I wonder, do you have a rationale for including each patch? IMO,
Arfrever has a tendency to diverge a bit further from upstream than I
like, and I note that you've taken in some patches and don't seem to
have gone in upstream. These are the  differences between my 2.7.3
patchset and your 2.7.3-0:

1. Added 08_all_regenerate_platform-specific_modules.patch, which
doesn't seem to be upstream yet.
2. Added back 22_all_turkish_locale.patch, which AFAIK isn't upstream,
nor associated with an open upstream bug?
3. Added 61_all_process_data.patch, for which the goal seems somewhat unclear.

You also removed the mention of the upstream bug from
04_all_libdir.patch, probably just by mistake?

As for 3.2.3, I'm also -1 on including 23_all_h2py_encoding.patch
after reading http://bugs.python.org/issue13032. Including
26_all_gdbm-1.9.patch in 3.1.5 is probably a good idea. For 3.1.5's
09_all_sys.platform_linux2.patch, I'd prefer if we just reuse
${FILESDIR}/linux2.patch, unless that doesn't apply for some reason.

Now, we can certainly discuss adding these patches on this list, but I
think we should try to maintain some balance on the upside of having
extra fixes in our ebuilds and the amount of maintenance we're willing
to do on carrying those patches forward (e.g. the distutils patch is a
pretty big pain, and it seems like more of a feature than a bug). I
don't think we should throw everything out on revbumps or bugfix
releases, but for new releases such as 3.3 I would personally like to
do only the bare minimum of patching.

Cheers,

Dirkjan

Re: [gentoo-python] Testing dev-lang/python version bumps

[Mike Gilbert]

[-- Attachment #1: Type: text/plain, Size: 3029 bytes --]

On 04/27/2012 09:03 AM, Dirkjan Ochtman wrote:
> Thanks for doing this! Sorry it took so long to review them... we
> should try to think of some easier review mechanism than putting up a
> tarball you have to unpack.
> 
> On Sun, Apr 22, 2012 at 03:12, Mike Gilbert <floppym@gentoo.org> wrote:
>> If we can get some people testing these that would be great. I would
>> like to add them to the tree sometime in the next week.
> 
> I wonder, do you have a rationale for including each patch? IMO,
> Arfrever has a tendency to diverge a bit further from upstream than I
> like, and I note that you've taken in some patches and don't seem to
> have gone in upstream. 

To be honest, I did not look into each patch in great detail. I really
just tested the resulting builds to make sure they did not break
anything obvious.

That said, let's dive in!

> These are the  differences between my 2.7.3
> patchset and your 2.7.3-0:
> 
> 1. Added 08_all_regenerate_platform-specific_modules.patch, which
> doesn't seem to be upstream yet.

Indeed it does not. Based on the feedback in the upstream bug, let's
drop it.

> 2. Added back 22_all_turkish_locale.patch, which AFAIK isn't upstream,
> nor associated with an open upstream bug?

I can't find a bug for this either.

> 3. Added 61_all_process_data.patch, for which the goal seems somewhat unclear.
> 

This is some logic for python-wrapper that was in the 2.7.2 patchset as
well. If you want to drop it, I'm sure that will require some
re-engineering of python-wrapper.

> You also removed the mention of the upstream bug from
> 04_all_libdir.patch, probably just by mistake?
> 

I don't see any mention of a bug in the 2.7.2, 2.7.3 or 2.7.3-0 version
of the patch, so I'm not sure what you are referring to here.

> As for 3.2.3, I'm also -1 on including 23_all_h2py_encoding.patch
> after reading http://bugs.python.org/issue13032.

Agreed.

> Including
> 26_all_gdbm-1.9.patch in 3.1.5 is probably a good idea. For 3.1.5's
> 09_all_sys.platform_linux2.patch, I'd prefer if we just reuse
> ${FILESDIR}/linux2.patch, unless that doesn't apply for some reason.

I don't really see a difference either way. I guess it is more visible
in the ebuild.

> Now, we can certainly discuss adding these patches on this list, but I
> think we should try to maintain some balance on the upside of having
> extra fixes in our ebuilds and the amount of maintenance we're willing
> to do on carrying those patches forward (e.g. the distutils patch is a
> pretty big pain, and it seems like more of a feature than a bug).

Well, that does seem to be Arfrever's baby, so as long as he keeps
rebase it, we should be ok.

> I don't think we should throw everything out on revbumps or bugfix
> releases, but for new releases such as 3.3 I would personally like to
> do only the bare minimum of patching.
> 

That makes sense. I will keep it in mind.

Would you like me to cut a new set of tarballs without 08, 22, and 23?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-python] Testing dev-lang/python version bumps

[Dirkjan Ochtman]

On Fri, Apr 27, 2012 at 17:13, Mike Gilbert <floppym@gentoo.org> wrote:
> To be honest, I did not look into each patch in great detail. I really
> just tested the resulting builds to make sure they did not break
> anything obvious.

I can imagine, I just think it's probably good if we do look at
patches in detail.

>> 3. Added 61_all_process_data.patch, for which the goal seems somewhat unclear.
> This is some logic for python-wrapper that was in the 2.7.2 patchset as
> well. If you want to drop it, I'm sure that will require some
> re-engineering of python-wrapper.

Right. Do you know what this fixes, and how/why it will be
re-engineered? As long as we don't know these things, I'd prefer to
leave the patch out rather than include it.

>> You also removed the mention of the upstream bug from
>> 04_all_libdir.patch, probably just by mistake?
>
> I don't see any mention of a bug in the 2.7.2, 2.7.3 or 2.7.3-0 version
> of the patch, so I'm not sure what you are referring to here.

Are you sure? The 04* patch I just unpacked from my 2.7.3 patch set
does have a bug in it (maybe it wasn't in 2.7.2, though).

>> Including
>> 26_all_gdbm-1.9.patch in 3.1.5 is probably a good idea. For 3.1.5's
>> 09_all_sys.platform_linux2.patch, I'd prefer if we just reuse
>> ${FILESDIR}/linux2.patch, unless that doesn't apply for some reason.
>
> I don't really see a difference either way. I guess it is more visible
> in the ebuild.

I like the fact that it's a single patch for all the versions, and
that we don't have to manage it separately in the patch sets.

>> Now, we can certainly discuss adding these patches on this list, but I
>> think we should try to maintain some balance on the upside of having
>> extra fixes in our ebuilds and the amount of maintenance we're willing
>> to do on carrying those patches forward (e.g. the distutils patch is a
>> pretty big pain, and it seems like more of a feature than a bug).
>
> Well, that does seem to be Arfrever's baby, so as long as he keeps
> rebase it, we should be ok.

Yeah, I'd really prefer to have us not depend on Arfrever for our
dev-lang/python updates. IMO we should drop this from 3.3 pending
upstream movement.

> That makes sense. I will keep it in mind.
>
> Would you like me to cut a new set of tarballs without 08, 22, and 23?

That would be perfect. I'd also still like to drop 61 unless we have a
clear picture of what/why it helps (and can document that in the
patch).

Cheers,

Dirkjan

Re: [gentoo-python] Testing dev-lang/python version bumps

[Mike Gilbert]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

On 04/27/2012 12:35 PM, Dirkjan Ochtman wrote:
>>> 3. Added 61_all_process_data.patch, for which the goal seems somewhat unclear.
>> This is some logic for python-wrapper that was in the 2.7.2 patchset as
>> well. If you want to drop it, I'm sure that will require some
>> re-engineering of python-wrapper.
> 
> Right. Do you know what this fixes, and how/why it will be
> re-engineered? As long as we don't know these things, I'd prefer to
> leave the patch out rather than include it.
> 

I have been bitten in the past by removing things that I don't fully
understand, so I'm trying to avoid doing that. I guess I can drop it and
see what happens.

>>> You also removed the mention of the upstream bug from
>>> 04_all_libdir.patch, probably just by mistake?
>>
>> I don't see any mention of a bug in the 2.7.2, 2.7.3 or 2.7.3-0 version
>> of the patch, so I'm not sure what you are referring to here.
> 
> Are you sure? The 04* patch I just unpacked from my 2.7.3 patch set
> does have a bug in it (maybe it wasn't in 2.7.2, though).
>

The words "bug" and "issue" do not appear in 04_all_libdir.patch from
the copy of python-gentoo-patches-2.7.3.tar.bz2 that I just downloaded
from distfiles.gentoo.org. I also checked the copy in distfiles-local on
dev.gentoo.org.

I think you may be working with a different tarball, or I'm missing
something really obvious.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-python] Testing dev-lang/python version bumps

[Mike Gilbert]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

On 04/27/2012 01:17 PM, Mike Gilbert wrote:
> On 04/27/2012 12:35 PM, Dirkjan Ochtman wrote:
>>>> 3. Added 61_all_process_data.patch, for which the goal seems somewhat unclear.
>>> This is some logic for python-wrapper that was in the 2.7.2 patchset as
>>> well. If you want to drop it, I'm sure that will require some
>>> re-engineering of python-wrapper.
>>
>> Right. Do you know what this fixes, and how/why it will be
>> re-engineered? As long as we don't know these things, I'd prefer to
>> leave the patch out rather than include it.
>>
> 
> I have been bitten in the past by removing things that I don't fully
> understand, so I'm trying to avoid doing that. I guess I can drop it and
> see what happens.
> 

I will add investigating this patch to my todo list. Hopefully Arfrever
can explain the rationale to me, and I can relay that.

Otherwise, I will have to figure it out the hard way.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-python] Testing dev-lang/python version bumps

[Mike Gilbert]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

On 04/27/2012 01:24 PM, Mike Gilbert wrote:
> On 04/27/2012 01:17 PM, Mike Gilbert wrote:
>> On 04/27/2012 12:35 PM, Dirkjan Ochtman wrote:
>>>>> 3. Added 61_all_process_data.patch, for which the goal seems somewhat unclear.
>>>> This is some logic for python-wrapper that was in the 2.7.2 patchset as
>>>> well. If you want to drop it, I'm sure that will require some
>>>> re-engineering of python-wrapper.
>>>
>>> Right. Do you know what this fixes, and how/why it will be
>>> re-engineered? As long as we don't know these things, I'd prefer to
>>> leave the patch out rather than include it.
>>>
>>
>> I have been bitten in the past by removing things that I don't fully
>> understand, so I'm trying to avoid doing that. I guess I can drop it and
>> see what happens.
>>
> 
> I will add investigating this patch to my todo list. Hopefully Arfrever
> can explain the rationale to me, and I can relay that.
> 
> Otherwise, I will have to figure it out the hard way.
> 

Had a chat with Arfrever last night about this.

The variables introduced in 61_all_process_data.patch are there to make
the scripts created by python_generate_wrapper_scripts as transparent as
possible. Specifically, they are used to manipulate sys.argv[0] in
python, argv[1] in C, and the "process name" as set by
prctl(PR_SET_NAME, ...).

This allows /usr/bin/A (wrapper) to call /usr/bin/A-2.7 (real script),
but still have /usr/bin/A in sys.argv[0].

This is mainly useful for supporting python scripts which vary their
behavior based on sys.argv[0]. The eclean program from gentoolkit does
this, for example. That's probably not the best example since it does
not use a python.eclass generated wrapper, but it illustrates the concept.

The process name is used by utilities like killall to find the process
to kill. I believe it is also used in top's display.

Arfrever also claims that this should help start-stop-daemon to detect
which process to kill, but I have not been able to verify this claim.
See bug 286191 for reference.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-python] Testing dev-lang/python version bumps

[Mike Gilbert]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

On 04/27/2012 12:35 PM, Dirkjan Ochtman wrote:
> On Fri, Apr 27, 2012 at 17:13, Mike Gilbert <floppym@gentoo.org> wrote:
>> Would you like me to cut a new set of tarballs without 08, 22, and 23?
> 
> That would be perfect. I'd also still like to drop 61 unless we have a
> clear picture of what/why it helps (and can document that in the
> patch).
> 

I have uploaded new tarballs to my devspace.

http://dev.gentoo.org/~floppym/python/python-gentoo-patches-2.7.3-1.tar.bz2

http://dev.gentoo.org/~floppym/python/python-gentoo-patches-3.2.3-1.tar.bz2

I removed only 08, 22, and 23.

Per my previous message, 61 has legitimate reasons for existing. I do
not believe there is a less intrusive way to accomplish what it is
doing, and removing it would cause the behavior of wrapped python
scripts to change. Do you still want to drop it?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-python] Testing dev-lang/python version bumps

[Dirkjan Ochtman]

On Sat, Apr 28, 2012 at 19:53, Mike Gilbert <floppym@gentoo.org> wrote:
> Per my previous message, 61 has legitimate reasons for existing. I do
> not believe there is a less intrusive way to accomplish what it is
> doing, and removing it would cause the behavior of wrapped python
> scripts to change. Do you still want to drop it?

Let's keep it. Thanks for doing all the work!

Cheers,

Dirkjan