[gentoo-dev] [RFC] Perspectives on improving (dis-hacking) python-single-r1

[gentoo-dev] [RFC] Perspectives on improving (dis-hacking) python-single-r1

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4421 bytes --]

Hello,

Some aspects of the current design of python-single-r1 are gross hack. 
I'd like to discuss potential alternatives to them.


Preamble
========
For the purpose of this mail, let's establish two terms.

'Single' will refer to packages allowing the user to select 1 out of N
supported Python implementations.  This generally means programs linking
to libpython, foreign build systems and end-user apps where installing
multiple copies of the code makes no sense.  Those are the packages
using python-single-r1.

'Multi' will refer to packages allowing the user to select M out of N
supported Python implementations.  This generally means ebuilds for
Python packages (modules, extensions).  Those are the packages using
python-r1 (usually via distutils-r1).

'Single' packages can PYTHON_USEDEP-end either on other 'single'
or on 'multi' packages.  However, 'multi' packages can only
PYTHON_USEDEP-end on other 'multi' packages (because 'single' wouldn't
ever be able to satisfy >1 impl).


Current status
==============
Currently, 'single' packages use two sets of USE flags: PYTHON_TARGETS
and PYTHON_SINGLE_TARGET.  The latter is used to select
the implementation used, and the former is used only for technical
reasons.  Selected PST is required to be also enabled in PT.

All dependencies enforce both PST and PT match.  The PYTHON_USEDEP is
generally a giant hack that's meant to work both for depending
on 'single' and 'multi' packages.  I'm not convinced this was the best
choice anymore.

In single-to-single deps, PYTHON_USEDEP enforces both PST and PT flags
to match.  This means that if two packages have matching PST but for
some reason were built with different PT, users end up having to switch
PT and rebuild the package without any real change.

In single-to-multi deps, PYTHON_USEDEP enforces PT flags.  For this
reason, we need to enforce that selected PST is always present in PT,
and users always have to put both flags in package.use.

There's also a gross hack in PYTHON_USEDEP in 'multi' packages that
means to prevent multi-to-single deps.  However, the dep mismatch it
causes is not very readable.


Alternative 1: PYTHON_TARGETS only
==================================
The first alternative I'd like to explore is removing
PYTHON_SINGLE_TARGET flags.  Why were they added in the first place? 
The primary goal was to be able to set PT to 2.7+3.x without requiring
people to adjust flags for every 'single' package out there.  Plus, it
turned out very convenient for Python team members who want to enable
all PT but obviously can't do the same for PST.

Originally I brought this proposal in context of automated REQUIRED_USE
conflict resolution.  However, that was rejected by the Council.  Still,
it may start making sense again in the near future when we start
removing py2.7 and pypy2.7 support.  If we can limit PT to one
implementation, and handle the remaining packages via IUSE defaults
and package.use, this may just work.

The inconvenience to people enabling multiple PT will remain though.


Alternative 2: new dependency API
=================================
If PST is going to stay, we may look into removing PT from 'single'
packages instead.  The idea is to provide new method of generating
cross-package deps that doesn't require fake flags.

PYTHON_USEDEP would continue working through a transitional period. 
When it's entirely gone, we can remove PYTHON_TARGETS from 'single'
packages.

Single-to-single deps would switch to PYTHON_SINGLE_USEDEP, that only
enforces PST and disregards PT entirely.

Single-to-multi deps would have to use a new generator function,
$(python_gen_multi_dep ...) that would create appropriate USE-mapping
from PST to PT.

Example ebuild would have:

RDEPEND="app-foo/singlepkg[${PYTHON_SINGLE_USEDEP}]
    $(python_gen_multi_dep '
         dev-python/foo[${PYTHON_USEDEP}]
         dev-python/bar[${PYTHON_USEDEP}]
    ')"

This will generate something like:

    pst_python2_7? (
        dev-python/foo[pt_python2_7(-)]
        dev-python/bar[pt_python2_7(-)]
    )
    pst_python3_7? (
        dev-python/foo[pt_python3_7(-)]
        dev-python/bar[pt_python3_7(-)]
    )


Your opinions
=============
So, WDYT?  Do you think this approach is worthwhile?  Do you see other
options?


-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

[gentoo-dev] Re: [RFC] Perspectives on improving (dis-hacking) python-single-r1

[Luca Barbato]

On 23/11/2019 16:48, Michał Górny wrote:
> Hello,
> 
> Some aspects of the current design of python-single-r1 are gross hack.
> I'd like to discuss potential alternatives to them.
> 
> 
> Preamble
> ========
> For the purpose of this mail, let's establish two terms.
> 
> 'Single' will refer to packages allowing the user to select 1 out of N
> supported Python implementations.  This generally means programs linking
> to libpython, foreign build systems and end-user apps where installing
> multiple copies of the code makes no sense.  Those are the packages
> using python-single-r1.
> 
> 'Multi' will refer to packages allowing the user to select M out of N
> supported Python implementations.  This generally means ebuilds for
> Python packages (modules, extensions).  Those are the packages using
> python-r1 (usually via distutils-r1).
> 
> 'Single' packages can PYTHON_USEDEP-end either on other 'single'
> or on 'multi' packages.  However, 'multi' packages can only
> PYTHON_USEDEP-end on other 'multi' packages (because 'single' wouldn't
> ever be able to satisfy >1 impl).
> 
> 
> Current status
> ==============
> Currently, 'single' packages use two sets of USE flags: PYTHON_TARGETS
> and PYTHON_SINGLE_TARGET.  The latter is used to select
> the implementation used, and the former is used only for technical
> reasons.  Selected PST is required to be also enabled in PT.
> 
> All dependencies enforce both PST and PT match.  The PYTHON_USEDEP is
> generally a giant hack that's meant to work both for depending
> on 'single' and 'multi' packages.  I'm not convinced this was the best
> choice anymore.
> 
> In single-to-single deps, PYTHON_USEDEP enforces both PST and PT flags
> to match.  This means that if two packages have matching PST but for
> some reason were built with different PT, users end up having to switch
> PT and rebuild the package without any real change.
> 
> In single-to-multi deps, PYTHON_USEDEP enforces PT flags.  For this
> reason, we need to enforce that selected PST is always present in PT,
> and users always have to put both flags in package.use.
> 
> There's also a gross hack in PYTHON_USEDEP in 'multi' packages that
> means to prevent multi-to-single deps.  However, the dep mismatch it
> causes is not very readable.
> 
> 
> Alternative 1: PYTHON_TARGETS only
> ==================================
> The first alternative I'd like to explore is removing
> PYTHON_SINGLE_TARGET flags.  Why were they added in the first place?
> The primary goal was to be able to set PT to 2.7+3.x without requiring
> people to adjust flags for every 'single' package out there.  Plus, it
> turned out very convenient for Python team members who want to enable
> all PT but obviously can't do the same for PST.
> 
> Originally I brought this proposal in context of automated REQUIRED_USE
> conflict resolution.  However, that was rejected by the Council.  Still,
> it may start making sense again in the near future when we start
> removing py2.7 and pypy2.7 support.  If we can limit PT to one
> implementation, and handle the remaining packages via IUSE defaults
> and package.use, this may just work.
> 
> The inconvenience to people enabling multiple PT will remain though.
> 
> 
> Alternative 2: new dependency API
> =================================
> If PST is going to stay, we may look into removing PT from 'single'
> packages instead.  The idea is to provide new method of generating
> cross-package deps that doesn't require fake flags.
> 
> PYTHON_USEDEP would continue working through a transitional period.
> When it's entirely gone, we can remove PYTHON_TARGETS from 'single'
> packages.
> 
> Single-to-single deps would switch to PYTHON_SINGLE_USEDEP, that only
> enforces PST and disregards PT entirely.
> 
> Single-to-multi deps would have to use a new generator function,
> $(python_gen_multi_dep ...) that would create appropriate USE-mapping
> from PST to PT.
> 
> Example ebuild would have:
> 
> RDEPEND="app-foo/singlepkg[${PYTHON_SINGLE_USEDEP}]
>      $(python_gen_multi_dep '
>           dev-python/foo[${PYTHON_USEDEP}]
>           dev-python/bar[${PYTHON_USEDEP}]
>      ')"
> 
> This will generate something like:
> 
>      pst_python2_7? (
>          dev-python/foo[pt_python2_7(-)]
>          dev-python/bar[pt_python2_7(-)]
>      )
>      pst_python3_7? (
>          dev-python/foo[pt_python3_7(-)]
>          dev-python/bar[pt_python3_7(-)]
>      )
> 
> 
> Your opinions
> =============
> So, WDYT?  Do you think this approach is worthwhile?  Do you see other
> options?
> 
> 

The new dependency API proposal looks nicer to me.

lu

Re: [gentoo-dev] [RFC] Perspectives on improving (dis-hacking) python-single-r1

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

Since we are here ...
I'd still like to see some way to adopt latest python version if user wants
to.
One way it could work is that we add a "LATEST" to PYTHON_TARGETS that
would always build against best version of python.
To avoid complications if a new version of python is emerged together with
other packages it must always include also a numeric version like 3_8

Regarding your proposal getting rid of PYTHON_SINGLE_TARGET would be nice
but being able to have multiple version of python installed is nicer and
differentiate gentoo from most other distro

Alternative 2 is also nice, the only thing that make me dubious is that it
looks like it will be a gigantic work, but you have a better sense of the
situation and will be one of those doing the actual work, so go for it!

[-- Attachment #2: Type: text/html, Size: 864 bytes --]

Re: [gentoo-dev] [RFC] Perspectives on improving (dis-hacking) python-single-r1

[Michael 'veremitz' Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 1330 bytes --]

On 24/11/19 16:06, Francesco Riosa wrote:
> Since we are here ...
> I'd still like to see some way to adopt latest python version if user
> wants to.
> One way it could work is that we add a "LATEST" to PYTHON_TARGETS that
> would always build against best version of python.
> To avoid complications if a new version of python is emerged together
> with other packages it must always include also a numeric version like 3_8
>
> Regarding your proposal getting rid of PYTHON_SINGLE_TARGET would be nice
> but being able to have multiple version of python installed is nicer and
> differentiate gentoo from most other distro
>
> Alternative 2 is also nice, the only thing that make me dubious is that
> it looks like it will be a gigantic work, but you have a better sense of
> the situation and will be one of those doing the actual work, so go for it!
Most distros have (or haD) .. python2 and python3 (not necessarily each
with their suffix) in their repo's .. but being able to have multiple
branches of either is certainly a feature IMHO.

I'm certainly for re-working the PYTHON_[SINGLE_]TARGETS dependencies, as
they can often get 'screwed up' when upgrading between versions, and
updating packages in between, and all quickly gets out of sync, and
requires a lot of hacking to fix up properly.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 801 bytes --]