[gentoo-python] RFC: Redesign for 'best implementation' in python-r1

[gentoo-python] RFC: Redesign for 'best implementation' in python-r1

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

Hello,

The current design for getting the 'best implementation' is far from
pretty. It pretty much relies on constant preference order, setting
Python 2.7 over other implementations for practical reasons. As a side
effect, various ebuilds rely on that particular order, e.g.:

  DEPEND="doc? ( dev-python/epydoc[$(python_gen_usedep 'python2*')] )"
  REQUIRED_USE="doc? ( $(python_gen_useflags 'python2*') )"

which pretty much assumes that *if python2 is enabled*, then any
version of it will be the best impl. I find this disgusting, and I'd
like to replace it with something more explicit.


Specifically:

1. python_export_best becomes deprecated for good,

2. python_setup changes API to:

  python_setup [<impl-wildcard>...]

alike python_gen_* functions, defaulting to '*'. Now it chooses
the best impl from implementations matching the specified patterns,
e.g.:

  python_setup 'python2*'

would get the best version of CPython 2.

3. we introduce extra variable for distutils-r1 (oh cruel world, how
should we name it?!) that applies wildcards to python_*_all() phases.


As for the choice within the list the algo needs to stay as-is for now.
However, in the future we could either:

a. respect EPYTHON and fallback to the other algo,

b. respect pre-defined order i.e. 3.4 > 2.7,

c. respect the order in PYTHON_COMPAT -- however that could be a little
surprising to devs.


What are your thoughts?

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-python] RFC: Redesign for 'best implementation' in python-r1

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 466 bytes --]

Dnia 2014-12-27, o godz. 14:50:43
Michał Górny <mgorny@gentoo.org> napisał(a):

> 2. python_setup changes API to:
> 
>   python_setup [<impl-wildcard>...]
> 
> alike python_gen_* functions, defaulting to '*'. Now it chooses
> the best impl from implementations matching the specified patterns,
> e.g.:
> 
>   python_setup 'python2*'
> 
> would get the best version of CPython 2.

Patch for this one attached.

-- 
Best regards,
Michał Górny

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: python-r1.eclass.diff --]
[-- Type: text/x-patch, Size: 2235 bytes --]

Index: python-r1.eclass
===================================================================
RCS file: /var/cvsroot/gentoo-x86/eclass/python-r1.eclass,v
retrieving revision 1.81
diff -u -B -r1.81 python-r1.eclass
--- python-r1.eclass	28 Dec 2014 10:56:55 -0000	1.81
+++ python-r1.eclass	28 Dec 2014 12:00:02 -0000
@@ -737,17 +737,56 @@
 }
 
 # @FUNCTION: python_setup
+# @USAGE: [<impl-pattern>...]
 # @DESCRIPTION:
-# Find the best (most preferred) Python implementation enabled
-# and set the Python build environment up for it.
+# Find the best (most preferred) Python implementation that is enabled
+# and matches at least one of the patterns passed (or '*' if no patterns
+# passed). Set the Python build environment up for that implementation.
 #
 # This function needs to be used when Python is being called outside
 # of python_foreach_impl calls (e.g. for shared processes like doc
 # building). python_foreach_impl sets up the build environment itself.
+#
+# If the specific commands support only a subset of Python
+# implementations, patterns need to be passed to restrict the allowed
+# implementations.
+#
+# Example:
+# @CODE
+# DEPEND="doc? ( dev-python/epydoc[$(python_gen_usedep 'python2*')] )"
+#
+# src_compile() {
+#   #...
+#   if use doc; then
+#     python_setup 'python2*'
+#     make doc
+#   fi
+# }
+# @CODE
 python_setup() {
 	debug-print-function ${FUNCNAME} "${@}"
 
-	python_export_best
+	local best_impl patterns=( "${@-*}" )
+	_python_try_impl() {
+		local pattern
+		for pattern in "${patterns[@]}"; do
+			if [[ ${EPYTHON} == ${pattern} ]]; then
+				best_impl=${EPYTHON}
+			fi
+		done
+	}
+	python_foreach_impl _python_try_impl
+
+	if [[ ! ${best_impl} ]]; then
+		eerror "${FUNCNAME}: none of the enabled implementation matched the patterns."
+		eerror "  patterns: ${@-'(*)'}"
+		eerror "Likely a REQUIRED_USE constraint (possibly USE-conditional) is missing."
+		eerror "  suggested: || ( \$(python_gen_useflags ${@}) )"
+		eerror "(remember to quote all the patterns with '')"
+		die "${FUNCNAME}: no enabled implementation satisfy requirements"
+	fi
+
+	python_export "${best_impl}" EPYTHON PYTHON
 	python_wrapper_setup
 }
 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-python] RFC: Redesign for 'best implementation' in python-r1

[Mike Gilbert]

On Sat, Dec 27, 2014 at 8:50 AM, Michał Górny <mgorny@gentoo.org> wrote:
> Hello,
>
> The current design for getting the 'best implementation' is far from
> pretty. It pretty much relies on constant preference order, setting
> Python 2.7 over other implementations for practical reasons. As a side
> effect, various ebuilds rely on that particular order, e.g.:
>
>   DEPEND="doc? ( dev-python/epydoc[$(python_gen_usedep 'python2*')] )"
>   REQUIRED_USE="doc? ( $(python_gen_useflags 'python2*') )"
>
> which pretty much assumes that *if python2 is enabled*, then any
> version of it will be the best impl. I find this disgusting, and I'd
> like to replace it with something more explicit.
>

Yeah, relying python_compile_all running under python2.7 is no good.

>
> Specifically:
>
> 1. python_export_best becomes deprecated for good,
>
> 2. python_setup changes API to:
>
>   python_setup [<impl-wildcard>...]
>
> alike python_gen_* functions, defaulting to '*'. Now it chooses
> the best impl from implementations matching the specified patterns,
> e.g.:
>
>   python_setup 'python2*'
>
> would get the best version of CPython 2.
>
> 3. we introduce extra variable for distutils-r1 (oh cruel world, how
> should we name it?!) that applies wildcards to python_*_all() phases.
>

This sounds like a reasonable idea, and a natural extension of the
python_setup function.

>
> As for the choice within the list the algo needs to stay as-is for now.
> However, in the future we could either:
>
> a. respect EPYTHON and fallback to the other algo,
>
> b. respect pre-defined order i.e. 3.4 > 2.7,
>
> c. respect the order in PYTHON_COMPAT -- however that could be a little
> surprising to devs.
>
>
> What are your thoughts?

Starting/ending with EPYTHON from the environment seems wrong, and
doesn't seem to add value anyway.

I would prefer to stick with a predefined order unless someone can
present a case where the ebuild author really needs to control it.

Re: [gentoo-python] RFC: Redesign for 'best implementation' in python-r1

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

Dnia 2014-12-31, o godz. 11:41:45
Mike Gilbert <floppym@gentoo.org> napisał(a):

> >
> > As for the choice within the list the algo needs to stay as-is for now.
> > However, in the future we could either:
> >
> > a. respect EPYTHON and fallback to the other algo,
> >
> > b. respect pre-defined order i.e. 3.4 > 2.7,
> >
> > c. respect the order in PYTHON_COMPAT -- however that could be a little
> > surprising to devs.
> >
> >
> > What are your thoughts?
> 
> Starting/ending with EPYTHON from the environment seems wrong, and
> doesn't seem to add value anyway.
> 
> I would prefer to stick with a predefined order unless someone can
> present a case where the ebuild author really needs to control it.

Well, it isn't really about the ebuild author but about the user.
EPYTHON idea is pretty much a reference to the python-any-r1 behavior.
IOW, having both eclasses use the same preference handling.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-python] RFC: Redesign for 'best implementation' in python-r1

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]

Dnia 2014-12-27, o godz. 14:50:43
Michał Górny <mgorny@gentoo.org> napisał(a):

> 3. we introduce extra variable for distutils-r1 (oh cruel world, how
> should we name it?!) that applies wildcards to python_*_all() phases.

Ok, I see a problem with this one. Consider the common use:

  DEPEND="doc? ( dev-python/epydoc[$(python_gen_usedep 'python2*')] )"
  REQUIRED_USE="doc? ( $(python_gen_useflags 'python2*') )"

  python_compile_all() {
    use doc && esetup.py doc # runs epydoc
  }

So we'd like the _all() phase to be run with python2* because of epydoc.
But OTOH we don't need any python2* if USE=-doc.

If we used a simple variable to control impl for _all(), it would
unnecessarily force python2* with USE=-doc, and python2* may not be
enabled at all then. So we either set it conditionally (+ extra phase,
added complexity), or make it non-fatal. Non-fatal is ugly since it
means unpredictable behavior.

So we can use a complex variable instead. But this way, we're in
python.eclass territory soon. Alternatively, add some function that's
called random-early by other python stuff, and that can be used to set
this variable -- but this doesn't look like good API design.

Maybe calling python_setup inside _all() phases -- but this implies
setting at least two environments, and we'd have to handle extra stuff
that distutils-r1 sets like PYTHONPATH. Still doesn't sound good.

So any ideas? :P

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]