[gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Patrick McLean]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

This patch splits the definition of _PYTHON_ALL_IMPLS and
_python_impl_supported to a separate eclass, this allows overlays
to easily support a different set of python implementations than
::gentoo without having to fork the entire suite of eclasses.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: python-utils-r1-split-impls.patch --]
[-- Type: text/x-patch, Size: 5323 bytes --]

diff --git a/eclass/python-impls-r1.eclass b/eclass/python-impls-r1.eclass
new file mode 100644
index 00000000000..0ae6e4e84a1
--- /dev/null
+++ b/eclass/python-impls-r1.eclass
@@ -0,0 +1,90 @@
+# Copyright 1999-2020 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: python-impls-r1.eclass
+# @MAINTAINER:
+# Python team <python@gentoo.org>
+# @AUTHOR:
+# Author: Michał Górny <mgorny@gentoo.org>
+# Split to separate eclass by: Patrick McLean <chutzpah@gentoo.org>
+# Based on work of: Krzysztof Pawlik <nelchael@gentoo.org>
+# @SUPPORTED_EAPIS: 5 6 7
+# @BLURB: Definitions of supported eclasses for python-utils-r1
+# @DESCRIPTION:
+# A helper eclass defining the supported python implementations for
+# the python-r1 suite of eclasses.
+#
+# This eclass is meant to be inherited by python-utils-r1, inheriting
+# it separately is very unlikely to be useful.
+#
+# For more information, please see the Python Guide:
+# https://dev.gentoo.org/~mgorny/python-guide/
+
+case "${EAPI:-0}" in
+	[0-4]) die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}" ;;
+	[5-7]) ;;
+	*)     die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}" ;;
+esac
+
+if [[ ${_PYTHON_ECLASS_INHERITED} ]]; then
+	die 'python-r1 suite eclasses can not be used with python.eclass.'
+fi
+
+if [[ ! ${_PYTHON_IMPLS_R1} ]]; then
+
+# @ECLASS-VARIABLE: _PYTHON_ALL_IMPLS
+# @INTERNAL
+# @DESCRIPTION:
+# All supported Python implementations, most preferred last.
+_PYTHON_ALL_IMPLS=(
+	pypy3
+	python2_7
+	python3_6 python3_7 python3_8
+)
+readonly _PYTHON_ALL_IMPLS
+
+# @ECLASS-VARIABLE: PYTHON_COMPAT_NO_STRICT
+# @INTERNAL
+# @DESCRIPTION:
+# Set to a non-empty value in order to make eclass tolerate (ignore)
+# unknown implementations in PYTHON_COMPAT.
+#
+# This is intended to be set by the user when using ebuilds that may
+# have unknown (newer) implementations in PYTHON_COMPAT. The assumption
+# is that the ebuilds are intended to be used within multiple contexts
+# which can involve revisions of this eclass that support a different
+# set of Python implementations.
+
+# @FUNCTION: _python_impl_supported
+# @USAGE: <impl>
+# @INTERNAL
+# @DESCRIPTION:
+# Check whether the implementation <impl> (PYTHON_COMPAT-form)
+# is still supported.
+#
+# Returns 0 if the implementation is valid and supported. If it is
+# unsupported, returns 1 -- and the caller should ignore the entry.
+# If it is invalid, dies with an appopriate error messages.
+_python_impl_supported() {
+	debug-print-function ${FUNCNAME} "${@}"
+
+	[[ ${#} -eq 1 ]] || die "${FUNCNAME}: takes exactly 1 argument (impl)."
+
+	local impl=${1}
+
+	# keep in sync with _PYTHON_ALL_IMPLS!
+	# (not using that list because inline patterns shall be faster)
+	case "${impl}" in
+		python2_7|python3_[678]|pypy3)
+			return 0
+			;;
+		jython2_7|pypy|pypy1_[89]|pypy2_0|python2_[56]|python3_[12345])
+			return 1
+			;;
+		*)
+			[[ ${PYTHON_COMPAT_NO_STRICT} ]] && return 1
+			die "Invalid implementation in PYTHON_COMPAT: ${impl}"
+	esac
+}
+_PYTHON_IMPLS_R1=1
+fi
diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index aacee5ac35a..28df410d5a1 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -32,62 +32,7 @@ fi
 if [[ ! ${_PYTHON_UTILS_R1} ]]; then
 
 [[ ${EAPI} == 5 ]] && inherit eutils multilib
-inherit toolchain-funcs
-
-# @ECLASS-VARIABLE: _PYTHON_ALL_IMPLS
-# @INTERNAL
-# @DESCRIPTION:
-# All supported Python implementations, most preferred last.
-_PYTHON_ALL_IMPLS=(
-	pypy3
-	python2_7
-	python3_6 python3_7 python3_8
-)
-readonly _PYTHON_ALL_IMPLS
-
-# @ECLASS-VARIABLE: PYTHON_COMPAT_NO_STRICT
-# @INTERNAL
-# @DESCRIPTION:
-# Set to a non-empty value in order to make eclass tolerate (ignore)
-# unknown implementations in PYTHON_COMPAT.
-#
-# This is intended to be set by the user when using ebuilds that may
-# have unknown (newer) implementations in PYTHON_COMPAT. The assumption
-# is that the ebuilds are intended to be used within multiple contexts
-# which can involve revisions of this eclass that support a different
-# set of Python implementations.
-
-# @FUNCTION: _python_impl_supported
-# @USAGE: <impl>
-# @INTERNAL
-# @DESCRIPTION:
-# Check whether the implementation <impl> (PYTHON_COMPAT-form)
-# is still supported.
-#
-# Returns 0 if the implementation is valid and supported. If it is
-# unsupported, returns 1 -- and the caller should ignore the entry.
-# If it is invalid, dies with an appopriate error messages.
-_python_impl_supported() {
-	debug-print-function ${FUNCNAME} "${@}"
-
-	[[ ${#} -eq 1 ]] || die "${FUNCNAME}: takes exactly 1 argument (impl)."
-
-	local impl=${1}
-
-	# keep in sync with _PYTHON_ALL_IMPLS!
-	# (not using that list because inline patterns shall be faster)
-	case "${impl}" in
-		python2_7|python3_[678]|pypy3)
-			return 0
-			;;
-		jython2_7|pypy|pypy1_[89]|pypy2_0|python2_[56]|python3_[12345])
-			return 1
-			;;
-		*)
-			[[ ${PYTHON_COMPAT_NO_STRICT} ]] && return 1
-			die "Invalid implementation in PYTHON_COMPAT: ${impl}"
-	esac
-}
+inherit toolchain-funcs python-impls-r1
 
 # @FUNCTION: _python_set_impls
 # @INTERNAL

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

On Thu, 2020-03-26 at 14:03 -0700, Patrick McLean wrote:
> This patch splits the definition of _PYTHON_ALL_IMPLS and
> _python_impl_supported to a separate eclass, this allows overlays
> to easily support a different set of python implementations than
> ::gentoo without having to fork the entire suite of eclasses.

NAK.  This increases the maintenance effort (even if it means having to
open yet another file) for *zero* gain to Gentoo users.  Your policy is
entirely broken by design and I'm against supporting it officially.

The existing number of eclasses is already causing confusion and added
maintenance effort, and it has strong justification in *a lot of shared
code*.

To say it bluntly: if you want to do stupid things, do them yourselves
and don't expect others to help.  You get paid for that.  We just waste
our time.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Patrick McLean]

On Fri, 27 Mar 2020 06:53:13 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Thu, 2020-03-26 at 14:03 -0700, Patrick McLean wrote:
> > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > _python_impl_supported to a separate eclass, this allows overlays
> > to easily support a different set of python implementations than
> > ::gentoo without having to fork the entire suite of eclasses.  
> 
> NAK.  This increases the maintenance effort (even if it means having to
> open yet another file) for *zero* gain to Gentoo users.  Your policy is
> entirely broken by design and I'm against supporting it officially.
> 
> The existing number of eclasses is already causing confusion and added
> maintenance effort, and it has strong justification in *a lot of shared
> code*.
> 
> To say it bluntly: if you want to do stupid things, do them yourselves
> and don't expect others to help.  You get paid for that.  We just waste
> our time.
> 
It can hard to keep the size of network we have in sync with upstream,
and we have quite a large number of internal repositories. The
approaches we are using are trying to strike a balance between
backwards compatibility and moving forward.

We get paid to do our job, which sometimes coincides with doing
upstream Gentoo work, and often doesn't. We have a policy to work
upstream wherever possible, this often makes certain types of tasks
take significantly more effort than if we just decided to fork
everything and work an internal overlay. These small changes are an
attempt to reduce the extra work that working upstream can create (we
generally work to support the general use case of all Gentoo users, not
just our limited use cases).

Would Gentoo as a whole be better served if we just forked everything
and stopped trying to contribute as much as possible upstream? Are some
small changes to some shared code to help us worth the amount of
contributions that we push back upstream.

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2573 bytes --]

On Fri, 2020-03-27 at 13:22 -0700, Patrick McLean wrote:
> On Fri, 27 Mar 2020 06:53:13 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > On Thu, 2020-03-26 at 14:03 -0700, Patrick McLean wrote:
> > > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > > _python_impl_supported to a separate eclass, this allows overlays
> > > to easily support a different set of python implementations than
> > > ::gentoo without having to fork the entire suite of eclasses.  
> > 
> > NAK.  This increases the maintenance effort (even if it means having to
> > open yet another file) for *zero* gain to Gentoo users.  Your policy is
> > entirely broken by design and I'm against supporting it officially.
> > 
> > The existing number of eclasses is already causing confusion and added
> > maintenance effort, and it has strong justification in *a lot of shared
> > code*.
> > 
> > To say it bluntly: if you want to do stupid things, do them yourselves
> > and don't expect others to help.  You get paid for that.  We just waste
> > our time.
> > 
> It can hard to keep the size of network we have in sync with upstream,
> and we have quite a large number of internal repositories. The
> approaches we are using are trying to strike a balance between
> backwards compatibility and moving forward.
> 
> We get paid to do our job, which sometimes coincides with doing
> upstream Gentoo work, and often doesn't. We have a policy to work
> upstream wherever possible, this often makes certain types of tasks
> take significantly more effort than if we just decided to fork
> everything and work an internal overlay. These small changes are an
> attempt to reduce the extra work that working upstream can create (we
> generally work to support the general use case of all Gentoo users, not
> just our limited use cases).
> 
> Would Gentoo as a whole be better served if we just forked everything
> and stopped trying to contribute as much as possible upstream? Are some
> small changes to some shared code to help us worth the amount of
> contributions that we push back upstream.

I was thinking about it, and how about you ask Portage people to provide
an option to silently ignore dying ebuilds, rather than trying hard to
prevent the eclass from exploding when it's supposed to explode?  That
would certainly be better than turning the eclasses into a minefield,
and expecting me to consider 'will this cause some eclass fork to
explode?' every time I change some internal eclass bit.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

On Fri, 2020-03-27 at 06:53 +0100, Michał Górny wrote:
> 
> To say it bluntly: if you want to do stupid things, do them yourselves
> and don't expect others to help.  You get paid for that.  We just waste
> our time.

I'm sorry, this paragraph turned out to be aggressive.  I didn't mean to
insult anyone.  I retract what I said here, and apologize to anyone
offended by it.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Matt Turner]

On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean <chutzpah@gentoo.org> wrote:
>
> This patch splits the definition of _PYTHON_ALL_IMPLS and
> _python_impl_supported to a separate eclass, this allows overlays
> to easily support a different set of python implementations than
> ::gentoo without having to fork the entire suite of eclasses.

(I think the issue is that you have some packages that still need
Python 3.4. Correct me if I'm wrong)

How many packages do you need to work with Python 3.4? Presumably just
a couple and then a pile of dependencies in ::gentoo?

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Patrick McLean]

On Fri, 27 Mar 2020 14:48:53 -0700
Matt Turner <mattst88@gentoo.org> wrote:

> On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean <chutzpah@gentoo.org> wrote:
> >
> > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > _python_impl_supported to a separate eclass, this allows overlays
> > to easily support a different set of python implementations than
> > ::gentoo without having to fork the entire suite of eclasses.  
> 
> (I think the issue is that you have some packages that still need
> Python 3.4. Correct me if I'm wrong)
> 
> How many packages do you need to work with Python 3.4? Presumably just
> a couple and then a pile of dependencies in ::gentoo?
> 

For our particular purpose, it's more complicated than that. We do not
expect or want ::gentoo to try support Python 3.4 in any way. We have an
overlay that is shared on a lot of machines, some of those machines are
older than others. As such we still have ebuilds that only support
python3_4 that still exist in the overlay. We would like it if the
existence of these packages in the overlay, do not cause metadata
generation or dependency calculation to explode on the more up-to-date
machines.

We do not need Python 3.4 packages to be installable on the newer
machines, we just need them not to explode.

I am trying to come up with a solution that *any* overlay can use, I
am happy to do work towards that end. Basically, it would be very
nice if there was a minimal eclass that handles all the python 
version compatibility. Almost everything in the eclasses does not care
about versions.

This is not meant to be something just for our usage, we want this to
be usable for *any* overlay, and are more than happy to make things as
generic as possible. If some overlay wants to support Python 3.10 alpha,
resurrect jython support, or add Ironpython support, without forking
all the eclasses, I would like to think that could be doable as well.

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2409 bytes --]

On Fri, Mar 27, 2020 at 3:11 PM Patrick McLean <chutzpah@gentoo.org> wrote:

> On Fri, 27 Mar 2020 14:48:53 -0700
> Matt Turner <mattst88@gentoo.org> wrote:
>
> > On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean <chutzpah@gentoo.org>
> wrote:
> > >
> > > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > > _python_impl_supported to a separate eclass, this allows overlays
> > > to easily support a different set of python implementations than
> > > ::gentoo without having to fork the entire suite of eclasses.
> >
> > (I think the issue is that you have some packages that still need
> > Python 3.4. Correct me if I'm wrong)
> >
> > How many packages do you need to work with Python 3.4? Presumably just
> > a couple and then a pile of dependencies in ::gentoo?
> >
>
> For our particular purpose, it's more complicated than that. We do not
> expect or want ::gentoo to try support Python 3.4 in any way. We have an
> overlay that is shared on a lot of machines, some of those machines are
> older than others. As such we still have ebuilds that only support
> python3_4 that still exist in the overlay. We would like it if the
> existence of these packages in the overlay, do not cause metadata
> generation or dependency calculation to explode on the more up-to-date
> machines.
>

> We do not need Python 3.4 packages to be installable on the newer
> machines, we just need them not to explode.
>

Couldn't you simply remove the ebuilds from the overlay entirely in this
case? It's my understanding that on the machines with the packages
installed, the merged package metadata is being used (which is why those
machines work) and since the newer machines don't have this merged package
metadata, they don't work properly.

-A


>
> I am trying to come up with a solution that *any* overlay can use, I
> am happy to do work towards that end. Basically, it would be very
> nice if there was a minimal eclass that handles all the python
> version compatibility. Almost everything in the eclasses does not care
> about versions.
>
> This is not meant to be something just for our usage, we want this to
> be usable for *any* overlay, and are more than happy to make things as
> generic as possible. If some overlay wants to support Python 3.10 alpha,
> resurrect jython support, or add Ironpython support, without forking
> all the eclasses, I would like to think that could be doable as well.
>
>

[-- Attachment #2: Type: text/html, Size: 3373 bytes --]

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Patrick McLean]

On Fri, 27 Mar 2020 15:51:35 -0700
Alec Warner <antarus@gentoo.org> wrote:

> On Fri, Mar 27, 2020 at 3:11 PM Patrick McLean <chutzpah@gentoo.org> wrote:
> 
> > On Fri, 27 Mar 2020 14:48:53 -0700
> > Matt Turner <mattst88@gentoo.org> wrote:
> >  
> > > On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean <chutzpah@gentoo.org>  
> > wrote:  
> > > >
> > > > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > > > _python_impl_supported to a separate eclass, this allows overlays
> > > > to easily support a different set of python implementations than
> > > > ::gentoo without having to fork the entire suite of eclasses.  
> > >
> > > (I think the issue is that you have some packages that still need
> > > Python 3.4. Correct me if I'm wrong)
> > >
> > > How many packages do you need to work with Python 3.4? Presumably just
> > > a couple and then a pile of dependencies in ::gentoo?
> > >  
> >
> > For our particular purpose, it's more complicated than that. We do not
> > expect or want ::gentoo to try support Python 3.4 in any way. We have an
> > overlay that is shared on a lot of machines, some of those machines are
> > older than others. As such we still have ebuilds that only support
> > python3_4 that still exist in the overlay. We would like it if the
> > existence of these packages in the overlay, do not cause metadata
> > generation or dependency calculation to explode on the more up-to-date
> > machines.
> >  
> 
> > We do not need Python 3.4 packages to be installable on the newer
> > machines, we just need them not to explode.
> >  
> 
> Couldn't you simply remove the ebuilds from the overlay entirely in this
> case? It's my understanding that on the machines with the packages
> installed, the merged package metadata is being used (which is why those
> machines work) and since the newer machines don't have this merged package
> metadata, they don't work properly.
> 

Sometimes we have to fix the older machines, so we have to keep
everything around until none of our machines are using it any more.

> 
> >
> > I am trying to come up with a solution that *any* overlay can use, I
> > am happy to do work towards that end. Basically, it would be very
> > nice if there was a minimal eclass that handles all the python
> > version compatibility. Almost everything in the eclasses does not care
> > about versions.
> >
> > This is not meant to be something just for our usage, we want this to
> > be usable for *any* overlay, and are more than happy to make things as
> > generic as possible. If some overlay wants to support Python 3.10 alpha,
> > resurrect jython support, or add Ironpython support, without forking
> > all the eclasses, I would like to think that could be doable as well.
> >
> >  

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 4856 bytes --]

On Fri, Mar 27, 2020 at 3:54 PM Patrick McLean <chutzpah@gentoo.org> wrote:

> On Fri, 27 Mar 2020 15:51:35 -0700
> Alec Warner <antarus@gentoo.org> wrote:
>
> > On Fri, Mar 27, 2020 at 3:11 PM Patrick McLean <chutzpah@gentoo.org>
> wrote:
> >
> > > On Fri, 27 Mar 2020 14:48:53 -0700
> > > Matt Turner <mattst88@gentoo.org> wrote:
> > >
> > > > On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean <chutzpah@gentoo.org>
>
> > > wrote:
> > > > >
> > > > > This patch splits the definition of _PYTHON_ALL_IMPLS and
> > > > > _python_impl_supported to a separate eclass, this allows overlays
> > > > > to easily support a different set of python implementations than
> > > > > ::gentoo without having to fork the entire suite of eclasses.
> > > >
> > > > (I think the issue is that you have some packages that still need
> > > > Python 3.4. Correct me if I'm wrong)
> > > >
> > > > How many packages do you need to work with Python 3.4? Presumably
> just
> > > > a couple and then a pile of dependencies in ::gentoo?
> > > >
> > >
> > > For our particular purpose, it's more complicated than that. We do not
> > > expect or want ::gentoo to try support Python 3.4 in any way. We have
> an
> > > overlay that is shared on a lot of machines, some of those machines are
> > > older than others. As such we still have ebuilds that only support
> > > python3_4 that still exist in the overlay. We would like it if the
> > > existence of these packages in the overlay, do not cause metadata
> > > generation or dependency calculation to explode on the more up-to-date
> > > machines.
> > >
> >
> > > We do not need Python 3.4 packages to be installable on the newer
> > > machines, we just need them not to explode.
> > >
> >
> > Couldn't you simply remove the ebuilds from the overlay entirely in this
> > case? It's my understanding that on the machines with the packages
> > installed, the merged package metadata is being used (which is why those
> > machines work) and since the newer machines don't have this merged
> package
> > metadata, they don't work properly.
> >
>
> Sometimes we have to fix the older machines, so we have to keep
> everything around until none of our machines are using it any more.
>

+Zac Medico <zmedico@gentoo.org>

I'm not following something then. One of the proposals earlier was "do not
generate metadata for these ebuilds" which to me reads as "keep these
ebuilds in the repo, but the ebuilds themselves will not be usable[0]."
Then you go on to say that old machines need to be fixed occasionally, so
either you need to reinstall a package or update it. The reinstall might be
strictly possible from the vdb metadata, but the upgrade would require
working repository metadata, which we said earlier we didn't want to
generate.

(There is a different question of whether you could use a binpkg binhost to
basically build versions of these packages and use those for reinstalls,
because at least the binpkg metadata *is* nominally fixed in time and can
be re-used easily and doesn't require eclass magic in theory, as the
eclasses are bound in the environment.tar?) I suspect in essence it might
be easier to just do what Robin suggested and use a frozen ::gentoo on the
older machines.

-A

[0] Perhaps the earlier proposals were ..slightly off. With more
information it seems like what you want is to be able to easily maintain
some python-3.4 ebuilds in your overlay, even though Gentoo is on to 3.6
(and soon 3.7). I personally think the conversation would have gone much
better had you just come out and said "hey we have a bunch of internal
overlays with 3.4 and we need to keep the packages for another N months,
how can we do this easily?" Instead of whatever happened. I tend to agree
with mgorny that it's not simply carrying this single patch, but in reality
it's a stronger commitment to build some kind of method to let you continue
to support older python versions. Future changes might impact your ability
to support your ebuilds and we have no real way of knowing if we broke
you..so I can see why (irrespective of the tone of the conversation) he
would be reticent to pick that up.


>
> > >
> > > I am trying to come up with a solution that *any* overlay can use, I
> > > am happy to do work towards that end. Basically, it would be very
> > > nice if there was a minimal eclass that handles all the python
> > > version compatibility. Almost everything in the eclasses does not care
> > > about versions.
> > >
> > > This is not meant to be something just for our usage, we want this to
> > > be usable for *any* overlay, and are more than happy to make things as
> > > generic as possible. If some overlay wants to support Python 3.10
> alpha,
> > > resurrect jython support, or add Ironpython support, without forking
> > > all the eclasses, I would like to think that could be doable as well.
> > >
> > >
>
>

[-- Attachment #2: Type: text/html, Size: 6360 bytes --]

Re: [gentoo-dev] [PATCH] Split python implementations definition to separate eclass

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 5442 bytes --]

On 3/27/20 4:12 PM, Alec Warner wrote:
> On Fri, Mar 27, 2020 at 3:54 PM Patrick McLean <chutzpah@gentoo.org
> <mailto:chutzpah@gentoo.org>> wrote:
> 
>     On Fri, 27 Mar 2020 15:51:35 -0700
>     Alec Warner <antarus@gentoo.org <mailto:antarus@gentoo.org>> wrote:
> 
>     > On Fri, Mar 27, 2020 at 3:11 PM Patrick McLean
>     <chutzpah@gentoo.org <mailto:chutzpah@gentoo.org>> wrote:
>     >
>     > > On Fri, 27 Mar 2020 14:48:53 -0700
>     > > Matt Turner <mattst88@gentoo.org <mailto:mattst88@gentoo.org>>
>     wrote:
>     > > 
>     > > > On Thu, Mar 26, 2020 at 2:03 PM Patrick McLean
>     <chutzpah@gentoo.org <mailto:chutzpah@gentoo.org>> 
>     > > wrote: 
>     > > > >
>     > > > > This patch splits the definition of _PYTHON_ALL_IMPLS and
>     > > > > _python_impl_supported to a separate eclass, this allows
>     overlays
>     > > > > to easily support a different set of python implementations than
>     > > > > ::gentoo without having to fork the entire suite of eclasses. 
>     > > >
>     > > > (I think the issue is that you have some packages that still need
>     > > > Python 3.4. Correct me if I'm wrong)
>     > > >
>     > > > How many packages do you need to work with Python 3.4?
>     Presumably just
>     > > > a couple and then a pile of dependencies in ::gentoo?
>     > > > 
>     > >
>     > > For our particular purpose, it's more complicated than that. We
>     do not
>     > > expect or want ::gentoo to try support Python 3.4 in any way. We
>     have an
>     > > overlay that is shared on a lot of machines, some of those
>     machines are
>     > > older than others. As such we still have ebuilds that only support
>     > > python3_4 that still exist in the overlay. We would like it if the
>     > > existence of these packages in the overlay, do not cause metadata
>     > > generation or dependency calculation to explode on the more
>     up-to-date
>     > > machines.
>     > > 
>     >
>     > > We do not need Python 3.4 packages to be installable on the newer
>     > > machines, we just need them not to explode.
>     > > 
>     >
>     > Couldn't you simply remove the ebuilds from the overlay entirely
>     in this
>     > case? It's my understanding that on the machines with the packages
>     > installed, the merged package metadata is being used (which is why
>     those
>     > machines work) and since the newer machines don't have this merged
>     package
>     > metadata, they don't work properly.
>     >
> 
>     Sometimes we have to fix the older machines, so we have to keep
>     everything around until none of our machines are using it any more.
> 
> 
> +Zac Medico <mailto:zmedico@gentoo.org> 
> 
> I'm not following something then. One of the proposals earlier was "do
> not generate metadata for these ebuilds" which to me reads as "keep
> these ebuilds in the repo, but the ebuilds themselves will not be
> usable[0]." Then you go on to say that old machines need to be fixed
> occasionally, so either you need to reinstall a package or update it.
> The reinstall might be strictly possible from the vdb metadata, but the
> upgrade would require working repository metadata, which we said earlier
> we didn't want to generate.

The ebuilds may or may not be usable, depending on the snapshot of
::gentoo eclasses that they're paired with.

> (There is a different question of whether you could use a binpkg binhost
> to basically build versions of these packages and use those for
> reinstalls, because at least the binpkg metadata *is* nominally fixed in
> time and can be re-used easily and doesn't require eclass magic in
> theory, as the eclasses are bound in the environment.tar?) I suspect in
> essence it might be easier to just do what Robin suggested and use a
> frozen ::gentoo on the older machines.

We do use a frozen snapshot of ::gentoo on older machines. The issue is
that we've got a repository of ebuilds that can be used with many
different snapshots of ::gentoo, but we'd like to be able to support
python implementations that may not be officially supported by a
particular snapshot of ::gentoo eclasses.

> -A
> 
> [0] Perhaps the earlier proposals were ..slightly off. With more
> information it seems like what you want is to be able to easily maintain
> some python-3.4 ebuilds in your overlay, even though Gentoo is on to 3.6
> (and soon 3.7). I personally think the conversation would have gone much
> better had you just come out and said "hey we have a bunch of internal
> overlays with 3.4 and we need to keep the packages for another N months,
> how can we do this easily?" Instead of whatever happened. I tend to
> agree with mgorny that it's not simply carrying this single patch, but
> in reality it's a stronger commitment to build some kind of method to
> let you continue to support older python versions. Future changes might
> impact your ability to support your ebuilds and we have no real way of
> knowing if we broke you..so I can see why (irrespective of the tone of
> the conversation) he would be reticent to pick that up.

If you break us then that's fine, we'll deal with it. We're just asking
for a way to tweak the set supported of python implementations, and
whether or not those implementations actually work in practice is not an
upstream gentoo problem.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 343 bytes --]