[gentoo-dev] [PATCHES] python-r1 suite clean up & impl reorder

[gentoo-dev] [PATCHES] python-r1 suite clean up & impl reorder

[Michał Górny]

Hi,

Here's another patchset. Mostly fixups inspired by multilib-build.eclass
changes with a few little additions.

Changes:

1. eclass-set variables are now read-only,
2. 'unset -f' is used to unset temporary & local functions,
3. implementations are reordered for sane order.

I've tested this with a few dozen random distutils-r1, python-r1,
python-any-r1 and python-single-r1 packages. However, for
the implementation reorder a larger tinderbox run would be appreciated.

As explained in the commit, the reorder may influence files installed by
a package, and implementation selected by python_setup(). This should
not cause issues for correctly written ebuilds, and should help us find
those that are not correctly written ;-).

In other words, we're finally considering Python 3.x preferred over
Python 2.x.

Please review.

[gentoo-dev] [PATCH 01/13] python-utils-r1.eclass: Mark _PYTHON_ALL_IMPLS read-only

[Michał Górny]

---
 eclass/python-utils-r1.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 024e093..0bce6a9 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -40,7 +40,7 @@ inherit toolchain-funcs
 # @INTERNAL
 # @DESCRIPTION:
 # All supported Python implementations, most preferred last.
-_PYTHON_ALL_IMPLS=(
+declare -g -r _PYTHON_ALL_IMPLS=(
 	jython2_5 jython2_7
 	pypy pypy3
 	python3_3 python3_4 python3_5
-- 
2.6.3

[gentoo-dev] [PATCH 02/13] python-utils-r1.eclass: Reorder implementations to semi-ascending order

[Michał Górny]

Reorder the Python implementations to ascending version order, with
CPython listed first and other implementations in descending preference.

The previous ordering has been used for two reasons:

1. There were packages which supported Python 3.x or PyPy partially but
their documentation builds or test functions required CPython 2.x.
The specific ordering caused python_export_best (the predecessor of
python_setup) to use CPython 2.x for those tasks. This is now replaced
by explicit implementation restrictions in python_setup.

2. PyPy setup runs were usually slower than CPython, and CPython 3.x
runs were often slower due to 2to3 calls. Combined with parallel build
runs, this ordering caused slower builds to start earlier and sometimes
resulted in more efficient use of threads. However, nowadays we no
longer do parallel builds.

Therefore, it seems reasonable to finally reorder the implementations
into a more intuitive order.
---
 eclass/python-utils-r1.eclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 0bce6a9..3ea23a8 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -41,10 +41,10 @@ inherit toolchain-funcs
 # @DESCRIPTION:
 # All supported Python implementations, most preferred last.
 declare -g -r _PYTHON_ALL_IMPLS=(
-	jython2_5 jython2_7
-	pypy pypy3
-	python3_3 python3_4 python3_5
 	python2_7
+	python3_3 python3_4 python3_5
+	pypy pypy3
+	jython2_5 jython2_7
 )
 
 # @FUNCTION: _python_impl_supported
-- 
2.6.3

Re: [gentoo-dev] [PATCH 02/13] python-utils-r1.eclass: Reorder implementations to semi-ascending order

[Patrice Clement]

Sunday 06 Dec 2015 20:03:21, Michał Górny wrote :
> Reorder the Python implementations to ascending version order, with
> CPython listed first and other implementations in descending preference.
> 
> The previous ordering has been used for two reasons:
> 
> 1. There were packages which supported Python 3.x or PyPy partially but
> their documentation builds or test functions required CPython 2.x.
> The specific ordering caused python_export_best (the predecessor of
> python_setup) to use CPython 2.x for those tasks. This is now replaced
> by explicit implementation restrictions in python_setup.
> 
> 2. PyPy setup runs were usually slower than CPython, and CPython 3.x
> runs were often slower due to 2to3 calls. Combined with parallel build
> runs, this ordering caused slower builds to start earlier and sometimes
> resulted in more efficient use of threads. However, nowadays we no
> longer do parallel builds.
> 
> Therefore, it seems reasonable to finally reorder the implementations
> into a more intuitive order.
> ---
>  eclass/python-utils-r1.eclass | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
> index 0bce6a9..3ea23a8 100644
> --- a/eclass/python-utils-r1.eclass
> +++ b/eclass/python-utils-r1.eclass
> @@ -41,10 +41,10 @@ inherit toolchain-funcs
>  # @DESCRIPTION:
>  # All supported Python implementations, most preferred last.
>  declare -g -r _PYTHON_ALL_IMPLS=(
> -	jython2_5 jython2_7
> -	pypy pypy3
> -	python3_3 python3_4 python3_5
>  	python2_7
> +	python3_3 python3_4 python3_5
> +	pypy pypy3
> +	jython2_5 jython2_7
>  )
>  
>  # @FUNCTION: _python_impl_supported
> -- 
> 2.6.3
> 
> 

Michal,

While at it, please delete jython2_5 from this list since jython versions < 2.7 are in
the process of being purged from Portage. 

See https://bugs.gentoo.org/show_bug.cgi?id=552452

Cheers,

-- 
Patrice Clement
Gentoo Linux developer
http://www.gentoo.org

Re: [gentoo-dev] [PATCH 02/13] python-utils-r1.eclass: Reorder implementations to semi-ascending order

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2252 bytes --]

On Tue, 8 Dec 2015 09:19:57 +0100
Patrice Clement <monsieurp@gentoo.org> wrote:

> Sunday 06 Dec 2015 20:03:21, Michał Górny wrote :
> > Reorder the Python implementations to ascending version order, with
> > CPython listed first and other implementations in descending preference.
> > 
> > The previous ordering has been used for two reasons:
> > 
> > 1. There were packages which supported Python 3.x or PyPy partially but
> > their documentation builds or test functions required CPython 2.x.
> > The specific ordering caused python_export_best (the predecessor of
> > python_setup) to use CPython 2.x for those tasks. This is now replaced
> > by explicit implementation restrictions in python_setup.
> > 
> > 2. PyPy setup runs were usually slower than CPython, and CPython 3.x
> > runs were often slower due to 2to3 calls. Combined with parallel build
> > runs, this ordering caused slower builds to start earlier and sometimes
> > resulted in more efficient use of threads. However, nowadays we no
> > longer do parallel builds.
> > 
> > Therefore, it seems reasonable to finally reorder the implementations
> > into a more intuitive order.
> > ---
> >  eclass/python-utils-r1.eclass | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> > 
> > diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
> > index 0bce6a9..3ea23a8 100644
> > --- a/eclass/python-utils-r1.eclass
> > +++ b/eclass/python-utils-r1.eclass
> > @@ -41,10 +41,10 @@ inherit toolchain-funcs
> >  # @DESCRIPTION:
> >  # All supported Python implementations, most preferred last.
> >  declare -g -r _PYTHON_ALL_IMPLS=(
> > -	jython2_5 jython2_7
> > -	pypy pypy3
> > -	python3_3 python3_4 python3_5
> >  	python2_7
> > +	python3_3 python3_4 python3_5
> > +	pypy pypy3
> > +	jython2_5 jython2_7
> >  )
> >  
> >  # @FUNCTION: _python_impl_supported
> > -- 
> > 2.6.3
> > 
> >   
> 
> Michal,
> 
> While at it, please delete jython2_5 from this list since jython versions < 2.7 are in
> the process of being purged from Portage. 
> 
> See https://bugs.gentoo.org/show_bug.cgi?id=552452

Will do, also from profiles.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] [PATCH 03/13] python-any-r1.eclass: Rename global-setting func to match eclass name

[Michał Górny]

---
 eclass/python-any-r1.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/python-any-r1.eclass b/eclass/python-any-r1.eclass
index b6c2258..390ce90 100644
--- a/eclass/python-any-r1.eclass
+++ b/eclass/python-any-r1.eclass
@@ -115,7 +115,7 @@ fi
 # 	dev-lang/python:2.6[gdbm] )
 # @CODE
 
-_python_build_set_globals() {
+_python_any_set_globals() {
 	local usestr i PYTHON_PKG_DEP
 	[[ ${PYTHON_REQ_USE} ]] && usestr="[${PYTHON_REQ_USE}]"
 
@@ -135,7 +135,7 @@ _python_build_set_globals() {
 	done
 	PYTHON_DEPS="|| ( ${PYTHON_DEPS})"
 }
-_python_build_set_globals
+_python_any_set_globals
 
 # @ECLASS-VARIABLE: PYTHON_USEDEP
 # @DESCRIPTION:
-- 
2.6.3

[gentoo-dev] [PATCH 04/13] python-any-r1.eclass: Unset global-setting function after use

[Michał Górny]

---
 eclass/python-any-r1.eclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/eclass/python-any-r1.eclass b/eclass/python-any-r1.eclass
index 390ce90..7d86b9a 100644
--- a/eclass/python-any-r1.eclass
+++ b/eclass/python-any-r1.eclass
@@ -136,6 +136,7 @@ _python_any_set_globals() {
 	PYTHON_DEPS="|| ( ${PYTHON_DEPS})"
 }
 _python_any_set_globals
+unset -f _python_any_set_globals
 
 # @ECLASS-VARIABLE: PYTHON_USEDEP
 # @DESCRIPTION:
-- 
2.6.3

[gentoo-dev] [PATCH 05/13] python-any-r1.eclass: Mark eclass-generated variables read-only

[Michał Górny]

---
 eclass/python-any-r1.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/python-any-r1.eclass b/eclass/python-any-r1.eclass
index 7d86b9a..82ead76 100644
--- a/eclass/python-any-r1.eclass
+++ b/eclass/python-any-r1.eclass
@@ -133,7 +133,7 @@ _python_any_set_globals() {
 
 		PYTHON_DEPS="${PYTHON_PKG_DEP} ${PYTHON_DEPS}"
 	done
-	PYTHON_DEPS="|| ( ${PYTHON_DEPS})"
+	declare -g -r PYTHON_DEPS="|| ( ${PYTHON_DEPS})"
 }
 _python_any_set_globals
 unset -f _python_any_set_globals
-- 
2.6.3

[gentoo-dev] [PATCH 06/13] python-single-r1.eclass: Unset global-setting function after use

[Michał Górny]

---
 eclass/python-single-r1.eclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/eclass/python-single-r1.eclass b/eclass/python-single-r1.eclass
index d9fc34b..a89926d 100644
--- a/eclass/python-single-r1.eclass
+++ b/eclass/python-single-r1.eclass
@@ -255,6 +255,7 @@ _python_single_set_globals() {
 	fi
 }
 _python_single_set_globals
+unset -f _python_single_set_globals
 
 # @FUNCTION: python_gen_usedep
 # @USAGE: <pattern> [...]
-- 
2.6.3

[gentoo-dev] [PATCH 07/13] python-single-r1.eclass: python_gen_cond_dep(), rename local PYTHON_USEDEP

[Michał Górny]

Rename local PYTHON_USEDEP variable to avoid collisions with read-only
global variable.
---
 eclass/python-single-r1.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/python-single-r1.eclass b/eclass/python-single-r1.eclass
index a89926d..8021501 100644
--- a/eclass/python-single-r1.eclass
+++ b/eclass/python-single-r1.eclass
@@ -388,8 +388,8 @@ python_gen_cond_dep() {
 				# (since python_gen_usedep() will not return ${PYTHON_USEDEP}
 				#  the code is run at most once)
 				if [[ ${dep} == *'${PYTHON_USEDEP}'* ]]; then
-					local PYTHON_USEDEP=$(python_gen_usedep "${@}")
-					dep=${dep//\$\{PYTHON_USEDEP\}/${PYTHON_USEDEP}}
+					local usedep=$(python_gen_usedep "${@}")
+					dep=${dep//\$\{PYTHON_USEDEP\}/${usedep}}
 				fi
 
 				matches+=( "python_single_target_${impl}? ( ${dep} )" )
-- 
2.6.3

[gentoo-dev] [PATCH 08/13] python-single-r1.eclass: Mark eclass-generated variables read-only

[Michał Górny]

---
 eclass/python-single-r1.eclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/eclass/python-single-r1.eclass b/eclass/python-single-r1.eclass
index 8021501..1582295 100644
--- a/eclass/python-single-r1.eclass
+++ b/eclass/python-single-r1.eclass
@@ -238,7 +238,7 @@ _python_single_set_globals() {
 			PYTHON_DEPS+="python_single_target_${i}? ( ${PYTHON_PKG_DEP} ) "
 		done
 	fi
-	PYTHON_USEDEP=${optflags// /,}
+	declare -g -r PYTHON_USEDEP=${optflags// /,}
 
 	# 1) well, python-exec would suffice as an RDEP
 	# but no point in making this overcomplex, BDEP doesn't hurt anyone
@@ -253,6 +253,7 @@ _python_single_set_globals() {
 	else
 		PYTHON_DEPS+="dev-lang/python-exec:2[${PYTHON_USEDEP}]"
 	fi
+	readonly PYTHON_DEPS PYTHON_REQUIRED_USE
 }
 _python_single_set_globals
 unset -f _python_single_set_globals
-- 
2.6.3

[gentoo-dev] [PATCH 09/13] python-r1.eclass: Unset global-setting function after use

[Michał Górny]

---
 eclass/python-r1.eclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/eclass/python-r1.eclass b/eclass/python-r1.eclass
index 91f0436..6e7b6e1 100644
--- a/eclass/python-r1.eclass
+++ b/eclass/python-r1.eclass
@@ -234,6 +234,7 @@ _python_set_globals() {
 	fi
 }
 _python_set_globals
+unset -f _python_set_globals
 
 # @FUNCTION: _python_validate_useflags
 # @INTERNAL
-- 
2.6.3

[gentoo-dev] [PATCH 10/13] python-r1.eclass: Unset local functions after use

[Michał Górny]

---
 eclass/python-r1.eclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/eclass/python-r1.eclass b/eclass/python-r1.eclass
index 6e7b6e1..22030b9 100644
--- a/eclass/python-r1.eclass
+++ b/eclass/python-r1.eclass
@@ -582,6 +582,7 @@ python_setup() {
 		done
 	}
 	python_foreach_impl _python_try_impl
+	unset -f _python_try_impl
 
 	if [[ ! ${best_impl} ]]; then
 		eerror "${FUNCNAME}: none of the enabled implementation matched the patterns."
@@ -619,6 +620,7 @@ python_export_best() {
 		best=${MULTIBUILD_VARIANT}
 	}
 	multibuild_for_best_variant _python_set_best
+	unset -f _python_set_best
 
 	debug-print "${FUNCNAME}: Best implementation is: ${best}"
 	python_export "${best}" "${@}"
@@ -653,6 +655,7 @@ python_replicate_script() {
 
 	local files=( "${@}" )
 	python_foreach_impl _python_replicate_script
+	unset -f _python_replicate_script
 
 	# install the wrappers
 	local f
-- 
2.6.3

[gentoo-dev] [PATCH 11/13] python-r1.eclass: python_gen_cond_dep(), rename local PYTHON_USEDEP

[Michał Górny]

Rename local PYTHON_USEDEP variable to avoid collisions with read-only
global variable.
---
 eclass/python-r1.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/python-r1.eclass b/eclass/python-r1.eclass
index 22030b9..8c58e31 100644
--- a/eclass/python-r1.eclass
+++ b/eclass/python-r1.eclass
@@ -390,8 +390,8 @@ python_gen_cond_dep() {
 				# (since python_gen_usedep() will not return ${PYTHON_USEDEP}
 				#  the code is run at most once)
 				if [[ ${dep} == *'${PYTHON_USEDEP}'* ]]; then
-					local PYTHON_USEDEP=$(python_gen_usedep "${@}")
-					dep=${dep//\$\{PYTHON_USEDEP\}/${PYTHON_USEDEP}}
+					local usedep=$(python_gen_usedep "${@}")
+					dep=${dep//\$\{PYTHON_USEDEP\}/${usedep}}
 				fi
 
 				matches+=( "python_targets_${impl}? ( ${dep} )" )
-- 
2.6.3

[gentoo-dev] [PATCH 12/13] python-r1.eclass: Mark eclass-generated variables read-only

[Michał Górny]

---
 eclass/python-r1.eclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/eclass/python-r1.eclass b/eclass/python-r1.eclass
index 8c58e31..4b9c6f3 100644
--- a/eclass/python-r1.eclass
+++ b/eclass/python-r1.eclass
@@ -216,8 +216,8 @@ _python_set_globals() {
 	optflags+=,${flags_st[@]/%/(-)}
 
 	IUSE=${flags[*]}
-	PYTHON_REQUIRED_USE="|| ( ${flags[*]} )"
-	PYTHON_USEDEP=${optflags// /,}
+	declare -g -r PYTHON_REQUIRED_USE="|| ( ${flags[*]} )"
+	declare -g -r PYTHON_USEDEP=${optflags// /,}
 
 	# 1) well, python-exec would suffice as an RDEP
 	# but no point in making this overcomplex, BDEP doesn't hurt anyone
@@ -232,6 +232,7 @@ _python_set_globals() {
 	else
 		PYTHON_DEPS+="dev-lang/python-exec:2[${PYTHON_USEDEP}]"
 	fi
+	readonly PYTHON_DEPS
 }
 _python_set_globals
 unset -f _python_set_globals
-- 
2.6.3

[gentoo-dev] [PATCH 13/13] distutils-r1.eclass: Unset local functions after use

[Michał Górny]

---
 eclass/distutils-r1.eclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 36b3436..be94c46 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -678,6 +678,7 @@ _distutils-r1_run_common_phase() {
 			done
 		}
 		python_foreach_impl _distutils_try_impl
+		unset -f _distutils_try_impl
 
 		local PYTHON_COMPAT=( "${best_impl}" )
 	fi
-- 
2.6.3

Re: [gentoo-dev] [PATCHES] python-r1 suite clean up & impl reorder

[Mike Gilbert]

On Sun, Dec 6, 2015 at 2:03 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Hi,
>
> Here's another patchset. Mostly fixups inspired by multilib-build.eclass
> changes with a few little additions.
>
> Changes:
>
> 1. eclass-set variables are now read-only,

Sounds ok, but might break some ebuilds.

> 2. 'unset -f' is used to unset temporary & local functions,
> 3. implementations are reordered for sane order.
>
> I've tested this with a few dozen random distutils-r1, python-r1,
> python-any-r1 and python-single-r1 packages. However, for
> the implementation reorder a larger tinderbox run would be appreciated.
>
> As explained in the commit, the reorder may influence files installed by
> a package, and implementation selected by python_setup(). This should
> not cause issues for correctly written ebuilds, and should help us find
> those that are not correctly written ;-).
>
> In other words, we're finally considering Python 3.x preferred over
> Python 2.x.

That could be a significant change, and I do expect that this will
break some ebuilds.

Any easy way to test this first? Or should we just be ready for the bug reports?

Re: [gentoo-dev] [PATCHES] python-r1 suite clean up & impl reorder

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1788 bytes --]

On Mon, 7 Dec 2015 15:36:30 -0500
Mike Gilbert <floppym@gentoo.org> wrote:

> On Sun, Dec 6, 2015 at 2:03 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > Hi,
> >
> > Here's another patchset. Mostly fixups inspired by multilib-build.eclass
> > changes with a few little additions.
> >
> > Changes:
> >
> > 1. eclass-set variables are now read-only,  
> 
> Sounds ok, but might break some ebuilds.
> 
> > 2. 'unset -f' is used to unset temporary & local functions,
> > 3. implementations are reordered for sane order.
> >
> > I've tested this with a few dozen random distutils-r1, python-r1,
> > python-any-r1 and python-single-r1 packages. However, for
> > the implementation reorder a larger tinderbox run would be appreciated.
> >
> > As explained in the commit, the reorder may influence files installed by
> > a package, and implementation selected by python_setup(). This should
> > not cause issues for correctly written ebuilds, and should help us find
> > those that are not correctly written ;-).
> >
> > In other words, we're finally considering Python 3.x preferred over
> > Python 2.x.  
> 
> That could be a significant change, and I do expect that this will
> break some ebuilds.
> 
> Any easy way to test this first? Or should we just be ready for the bug reports?

Not an easy. We could try some tinderboxing with USE='doc python'. Or
just grep ebuilds for python_export_best calls and plain python_setup
calls, and see if someone didn't assume it will always prefer python2.
REQUIRED_USE with python_gen_useflags may be a suggestion too.

So I'd go for waiting for bug reports. Or maybe doing the change
conditionally to EAPI 6 for improved confusion ;-).

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] [PATCHES] python-r1 suite clean up & impl reorder

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

On Sun,  6 Dec 2015 20:03:19 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Hi,
> 
> Here's another patchset. Mostly fixups inspired by multilib-build.eclass
> changes with a few little additions.
> 
> Changes:
> 
> 1. eclass-set variables are now read-only,
> 2. 'unset -f' is used to unset temporary & local functions,
> 3. implementations are reordered for sane order.
> 
> I've tested this with a few dozen random distutils-r1, python-r1,
> python-any-r1 and python-single-r1 packages. However, for
> the implementation reorder a larger tinderbox run would be appreciated.
> 
> As explained in the commit, the reorder may influence files installed by
> a package, and implementation selected by python_setup(). This should
> not cause issues for correctly written ebuilds, and should help us find
> those that are not correctly written ;-).
> 
> In other words, we're finally considering Python 3.x preferred over
> Python 2.x.

Merged now.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]