[gentoo-python] The future of PYTHON_SINGLE_TARGET

[gentoo-python] The future of PYTHON_SINGLE_TARGET

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3276 bytes --]

Friends!

The current state of python-single-r1 bothered me a lot lately,
and while _AxS_ did some neat improvements to the eclass, it's still
far from perfect solution to our users.

The PYTHON_SINGLE_TARGET flags were added to make it easier to handle
single-impl packages that support both Python 2 & Python 3 which we
enable both in our profiles. It serves that purpose well -- however, it
isn't that good for single-impl packages that don't support the most
common choice of implementations.

In particular, it has two major issues:

1. We have no sane, clear and universal way of providing users with
sensible defaults. It's always one trade-off over another.

2. Matching USE dependencies against multi-impl packages introduces
a second set of USE flags (PYTHON_TARGETS) that are semi-virtual.
Sadly, they introduce confusion and cause issues.


As far as the first issue is concerned, I think enough has been said
already. Right now we are fine-tuned for two cases:

a. packages that support one impl only -- then they have a single
PYTHON_TARGET that's enabled by default,

b. packages that support python2.7 -- then we have the profile default
of python2.7 for them.

We have no good way of handling packages that support Python 3 only.


The other issue comes into game in two cases I've seen:

A. when we try to improve setting of PYTHON_SINGLE_TARGET, we force
the user to enable matching PYTHON_TARGET anyway -- so in the end
manual intervention is required,

B. they cause issues with REQUIRED_USE on other packages (like libpeas
that supports only one version of python2 and one of python3). Even
though PYTHON_SINGLE_TARGET results in effective use of a single impl,
PYTHON_TARGET USE-dep requests full PYTHON_TARGETS match on the dep.
Therefore, the user needs to disable other implementations anyway to
get the expected result.


To be honest, I don't see a good way out of this. Possible bad ways:

I. disable PYTHON_SINGLE_TARGET and use PYTHON_REQUIRED_USE="^^ ..."
when python2_7 is not in PYTHON_COMPAT. In other words, add another
workaround that gets things rolling with the default profile setup.

II. Remove PYTHON_SINGLE_TARGET completely and use
PYTHON_REQUIRED_USE="^^ ...".

Option (1) is 'uglier' since it introduces another special case
and adds to the complexity. However, it supposedly impacts the lowest
number of users -- the default profile settings should pass, and only
people enabling more implementations would have to disable the flags.

Option (2) is cleaner but it imposes a major shock to our users. It
specifically involves manipulating USE flags on a lot of packages even
with the default profiles.

Both options come with another major drawback -- exactly-one-if deps
force manipulating the flags. Once manipulation is done, it may make
Python upgrades much harder -- esp. if portage would end up enabling
older impls via p.use.

Of course, this wouldn't be a problem if users only added -flags to
disable the old implementations. Then the issue will reapply when a new
implementation is enabled globally, making the effective of 2 enabled
implementations.


So, what are your thoughts? What are your solutions?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-python] Re: The future of PYTHON_SINGLE_TARGET

[Mike Gilbert]

On Fri, Nov 28, 2014 at 5:59 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Friends!
>
> The current state of python-single-r1 bothered me a lot lately,
> and while _AxS_ did some neat improvements to the eclass, it's still
> far from perfect solution to our users.
>
> The PYTHON_SINGLE_TARGET flags were added to make it easier to handle
> single-impl packages that support both Python 2 & Python 3 which we
> enable both in our profiles. It serves that purpose well -- however, it
> isn't that good for single-impl packages that don't support the most
> common choice of implementations.
>
> In particular, it has two major issues:
>
> 1. We have no sane, clear and universal way of providing users with
> sensible defaults. It's always one trade-off over another.
>
> 2. Matching USE dependencies against multi-impl packages introduces
> a second set of USE flags (PYTHON_TARGETS) that are semi-virtual.
> Sadly, they introduce confusion and cause issues.
>
>
> As far as the first issue is concerned, I think enough has been said
> already. Right now we are fine-tuned for two cases:
>
> a. packages that support one impl only -- then they have a single
> PYTHON_TARGET that's enabled by default,
>
> b. packages that support python2.7 -- then we have the profile default
> of python2.7 for them.
>
> We have no good way of handling packages that support Python 3 only.
>

I'm thinking we need to move the default PYTHON_SINGLE_TARGET setting
from profiles to IUSE defaults computed in the eclass.

What are the problems associated with that?

>
> The other issue comes into game in two cases I've seen:
>
> A. when we try to improve setting of PYTHON_SINGLE_TARGET, we force
> the user to enable matching PYTHON_TARGET anyway -- so in the end
> manual intervention is required,

That would seem to be the exception rather than the norm. This would
only happen if we somehow defaulted PYTHON_SINGLE_TARGET to something
that is not in PYTHON_TARGETS by default. How would that ever happen?

> B. they cause issues with REQUIRED_USE on other packages (like libpeas
> that supports only one version of python2 and one of python3). Even
> though PYTHON_SINGLE_TARGET results in effective use of a single impl,
> PYTHON_TARGET USE-dep requests full PYTHON_TARGETS match on the dep.
> Therefore, the user needs to disable other implementations anyway to
> get the expected result.

I don't understand this part. Can you please explain it? Maybe an example?

> So, what are your thoughts? What are your solutions?

I don't really understand the problems well enough to propose
solutions. Hopefully answering my questions above will help with that.

Re: [gentoo-python] Re: The future of PYTHON_SINGLE_TARGET

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 5967 bytes --]

Dnia 2014-11-30, o godz. 13:12:18
Mike Gilbert <floppym@gentoo.org> napisał(a):

> On Fri, Nov 28, 2014 at 5:59 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > As far as the first issue is concerned, I think enough has been said
> > already. Right now we are fine-tuned for two cases:
> >
> > a. packages that support one impl only -- then they have a single
> > PYTHON_TARGET that's enabled by default,
> >
> > b. packages that support python2.7 -- then we have the profile default
> > of python2.7 for them.
> >
> > We have no good way of handling packages that support Python 3 only.
> >
> 
> I'm thinking we need to move the default PYTHON_SINGLE_TARGET setting
> from profiles to IUSE defaults computed in the eclass.
> 
> What are the problems associated with that?

Well, one issue I see is that it means we will have a 'moving' default.
That is, different packages will have different (non-matching) defaults
depending on supported implementations. Worse than that, adding a new
implementation will change the default in one package which may conflict
with its dependencies.

Please consider the following example. We have two packages, AA and AB.
Both are python-single-r1 and in version 1 both support 3.2+3.3, with
3.3 being the IUSE default. AB depends on AA[${PYTHON_USEDEP}].

Now, A-2 gets 3.4 support and 3.4 becomes the IUSE default. Now emerge
-1vDtuN ab gives:

emerge: there are no ebuilds built with USE flags to satisfy
"dev-python/aa[python_targets_python3_3(-),python_single_target_python3_3(+)]". !!!
One of the following packages is required to complete your request:
- dev-python/aa-2::mgorny (Change USE: +python_single_target_python3_3,
this change violates use flag constraints defined by dev-python/aa-2:
'exactly-one-of ( python_single_target_python3_2
python_single_target_python3_3 python_single_target_python3_4 )
python_single_target_python3_2? ( python_targets_python3_2 )
python_single_target_python3_3? ( python_targets_python3_3 )
python_single_target_python3_4? ( python_targets_python3_4 )')
(dependency required by "dev-python/ab-1::mgorny" [installed])
(dependency required by "ab" [argument])

This happens both when PST USE-deps are '?' and '='.

Now, assuming that user actually adds 3.3, he gets regular REQUIRED_USE
conflict. He can solve it by removing 3.4 *explicitly*. Now he can
upgrade. Note that PYTHON_TARGETS has 3.3+3.4 now (see issue below).

Now good news, everyone! AB just gained 3.4 support, and therefore
switched default again. Now portage at least suggests the right thing
to do:

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by dev-python/ab-2::mgorny
# required by ab (argument)
>=dev-python/aa-2 python_single_target_python3_4 -python_single_target_python3_3

This is good so far. Now, let's imagine we have three packages -- AA,
AB and AC. AC also depends on AA and supports 3.2+3.3. This means that
once AB and AA again 3.4 support, Portage ends up having a real
conflict:

!!! Multiple package instances within a single package slot have been
pulled !!! into the dependency graph, resulting in a slot conflict:

dev-python/aa:0

  (dev-python/aa-2:0/0::mgorny, installed) pulled in by
    dev-python/aa[python_targets_python3_3(-),-python_single_target_python3_2(+),python_single_target_python3_3(+)]
  required by (dev-python/ac-1:0/0::mgorny, installed) 

  (dev-python/aa-2:0/0::mgorny, ebuild scheduled for merge) pulled in by
    dev-python/aa[python_targets_python3_2(-)?,python_targets_python3_3(-)?,python_targets_python3_4(-)?,python_single_target_python3_2(+)=,python_single_target_python3_3(+)=,python_single_target_python3_4(+)=]
  required by (dev-python/ab-2:0/0::mgorny, ebuild scheduled for merge)


Not good at all. Portage still suggests enabling 3.4 on AA but that
only makes things worse. If you force 3.3 on AB, it lets you continue.

Once AC gets 3.4, Portage says to switch AA to 3.4. Once you do that,
it tells you to switch it to 3.3 (because of AB). Guess what it tells
you to do next :).

> > The other issue comes into game in two cases I've seen:
> >
> > A. when we try to improve setting of PYTHON_SINGLE_TARGET, we force
> > the user to enable matching PYTHON_TARGET anyway -- so in the end
> > manual intervention is required,
> 
> That would seem to be the exception rather than the norm. This would
> only happen if we somehow defaulted PYTHON_SINGLE_TARGET to something
> that is not in PYTHON_TARGETS by default. How would that ever happen?

Forget about this. I guess we can enable the same PYTHON_TARGET
and PYTHON_SINGLE_TARGET via IUSE defaults.

> > B. they cause issues with REQUIRED_USE on other packages (like libpeas
> > that supports only one version of python2 and one of python3). Even
> > though PYTHON_SINGLE_TARGET results in effective use of a single impl,
> > PYTHON_TARGET USE-dep requests full PYTHON_TARGETS match on the dep.
> > Therefore, the user needs to disable other implementations anyway to
> > get the expected result.
> 
> I don't understand this part. Can you please explain it? Maybe an example?

gedit and libpeas, with gedit depending on libpeas. libpeas supports
one version of Python 2 and one of Python 3, and gedit just one
in general:

libpeas is REQUIRED_USE=?? ( 2.* ) ?? ( 3.* )
gedit is REQUIRED_USE=^^ ( 3.* )

So libpeas needs python-r1 with specific REQUIRED_USE. gedit would be
a candidate for python-single-r1 but...

p-s-r1 enforces ^^ only on PYTHON_SINGLE_TARGET. So I end up with
something like:

  PYTHON_TARGETS="3.3 3.4" PYTHON_SINGLE_TARGET="3.4"

and this enforces dependency on libpeas[3.3,3.4] which contradicts its
REQUIRED_USE. So we need also REQUIRED_USE=^^ ( 3.* ). But basically,
the whole mess with the second variable doesn't really help here.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-python] Re: The future of PYTHON_SINGLE_TARGET

[Mike Gilbert]

On Sun, Nov 30, 2014 at 3:50 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> > B. they cause issues with REQUIRED_USE on other packages (like libpeas
>> > that supports only one version of python2 and one of python3). Even
>> > though PYTHON_SINGLE_TARGET results in effective use of a single impl,
>> > PYTHON_TARGET USE-dep requests full PYTHON_TARGETS match on the dep.
>> > Therefore, the user needs to disable other implementations anyway to
>> > get the expected result.
>>
>> I don't understand this part. Can you please explain it? Maybe an example?
>
> gedit and libpeas, with gedit depending on libpeas. libpeas supports
> one version of Python 2 and one of Python 3, and gedit just one
> in general:
>
> libpeas is REQUIRED_USE=?? ( 2.* ) ?? ( 3.* )
> gedit is REQUIRED_USE=^^ ( 3.* )
>
> So libpeas needs python-r1 with specific REQUIRED_USE. gedit would be
> a candidate for python-single-r1 but...
>
> p-s-r1 enforces ^^ only on PYTHON_SINGLE_TARGET. So I end up with
> something like:
>
>   PYTHON_TARGETS="3.3 3.4" PYTHON_SINGLE_TARGET="3.4"
>
> and this enforces dependency on libpeas[3.3,3.4] which contradicts its
> REQUIRED_USE. So we need also REQUIRED_USE=^^ ( 3.* ). But basically,
> the whole mess with the second variable doesn't really help here.
>

What I take away from this is that the libpeas ebuild is really weird:
supporting exactly one python2 and one python3 sounds like a horrible
hack that should be fixed.

I don't think this crazy REQUIRED_USE constraint is something we
should design for in the eclasses. Let's fix libpeas to work with
multiple python3 versions instead.

Re: [gentoo-python] Re: The future of PYTHON_SINGLE_TARGET

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2249 bytes --]

Dnia 2014-11-30, o godz. 16:02:35
Mike Gilbert <floppym@gentoo.org> napisał(a):

> On Sun, Nov 30, 2014 at 3:50 PM, Michał Górny <mgorny@gentoo.org> wrote:
> >> > B. they cause issues with REQUIRED_USE on other packages (like libpeas
> >> > that supports only one version of python2 and one of python3). Even
> >> > though PYTHON_SINGLE_TARGET results in effective use of a single impl,
> >> > PYTHON_TARGET USE-dep requests full PYTHON_TARGETS match on the dep.
> >> > Therefore, the user needs to disable other implementations anyway to
> >> > get the expected result.
> >>
> >> I don't understand this part. Can you please explain it? Maybe an example?
> >
> > gedit and libpeas, with gedit depending on libpeas. libpeas supports
> > one version of Python 2 and one of Python 3, and gedit just one
> > in general:
> >
> > libpeas is REQUIRED_USE=?? ( 2.* ) ?? ( 3.* )
> > gedit is REQUIRED_USE=^^ ( 3.* )
> >
> > So libpeas needs python-r1 with specific REQUIRED_USE. gedit would be
> > a candidate for python-single-r1 but...
> >
> > p-s-r1 enforces ^^ only on PYTHON_SINGLE_TARGET. So I end up with
> > something like:
> >
> >   PYTHON_TARGETS="3.3 3.4" PYTHON_SINGLE_TARGET="3.4"
> >
> > and this enforces dependency on libpeas[3.3,3.4] which contradicts its
> > REQUIRED_USE. So we need also REQUIRED_USE=^^ ( 3.* ). But basically,
> > the whole mess with the second variable doesn't really help here.
> >
> 
> What I take away from this is that the libpeas ebuild is really weird:
> supporting exactly one python2 and one python3 sounds like a horrible
> hack that should be fixed.

No, supporting multiple Python versions actually is a horrible hack
that causes us much more pain than we need. Any sane binary distro
limits itself to Python 2 + Python 3, and that's what libpeas addresses
-- the strong incompatibility between P2 and P3 that forces it to have
two loaders that link to different Python versions.

Of course, we could make it have up to five loaders of which at most
two will be practically used but is it worth the effort? We'll
introduce incompatibility with plugins that will in turn require
modifying every reverse dependency of libpeas.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]