[gentoo-dev] [PATCH 00/11] distutils-r1.eclass: Support for gpep517-9+ --optimize + minor changes

[gentoo-dev] [PATCH 00/11] distutils-r1.eclass: Support for gpep517-9+ --optimize + minor changes

[Michał Górny]

Hi,

Here's another medium-sized patchset for Python eclasses.  The primary
feature is support for the new gpep517 9+ option to byte-compile
installed .pyc files.  As a result, rather than delaying until
python_install(), we get optimized .pyc already in python_compile()
and the test phase should be faster.

This implies a minor change to DISTUTILS_USE_PEP517=no behavior -- we're
no longer calling python_optimize implicitly, so if the ebuild does not
compile bytecode, it needs to call python_optimize explicitly now.
This mode is used only by a few ebuilds right now, so I'm including
updates to them as well.  On the plus side, it now avoids compiling .py
files twice in the remaining ebuilds.

The other major change included is rewriting the .pyc install-qa-check.d
to use the new "gpep517 verify-pyc" command rather than implementing
the checks in bash.  Since gpep517 is a dep of all ebuilds inheriting
distutils-r1, this should be a relatively safe place to put
the functionality, and so the check should work reliably in most
of the environments.  This also includes more correct checking of
whether .pyc files are up-to-date that couldn't be done in bash because
it impacted performance *a lot*.

The other changes include a fix for obsolete pyproject.toml
recommendation, the usual build system minimal version updates,
disabling pytest-sugar plugin in epytest (that makes test output awful)
and fixing EPREFIX in addpredict invocations.


Michał Górny (11):
  distutils-r1.eclass: Update pyproject.toml advice for PEP517 mode
  distutils-r1.eclass: Require gpep517 >= 8
  install-qa-check.d: Rewrite 60python-pyc to use gpep517
  distutils-r1.eclass: Bump dev-python/sip dependency
  distutils-r1.eclass: Pass EPREFIX to addpredict
  distutils-r1.eclass: Let gpep517 compile bytecode when 9+ is used
  dev-python/installer: Call python_optimize explicitly
  dev-python/tomli: Call python_optimize explicitly
  net-wireless/blueman: Invoke eautomake to fix py-compile script
  dev-python/pygobject: Call python_optimize explicitly
  python-utils-r1.eclass: Sterilize pytest-sugar plugin

 .../installer/installer-0.5.1-r1.ebuild       |   5 +
 dev-python/pygobject/pygobject-3.42.2.ebuild  |   1 +
 dev-python/tomli/tomli-2.0.1-r1.ebuild        |   5 +
 eclass/distutils-r1.eclass                    |  52 ++++--
 eclass/python-utils-r1.eclass                 |   3 +
 metadata/install-qa-check.d/60python-pyc      | 168 ++++++++++--------
 net-wireless/blueman/blueman-2.3.1.ebuild     |   2 +-
 net-wireless/blueman/blueman-9999.ebuild      |   2 +-
 8 files changed, 146 insertions(+), 92 deletions(-)

-- 
2.35.1

[gentoo-dev] [PATCH 01/11] distutils-r1.eclass: Update pyproject.toml advice for PEP517 mode

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index e64eedec5fd3..9909ba70178c 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -884,10 +884,10 @@ _distutils-r1_handle_pyproject_toml() {
 
 	if [[ ! -f setup.py && -f pyproject.toml ]]; then
 		if [[ ${DISTUTILS_USE_SETUPTOOLS} != pyproject.toml ]]; then
-			eerror "No setup.py found but pyproject.toml is present.  In order to enable"
-			eerror "pyproject.toml support in distutils-r1, set:"
-			eerror "  DISTUTILS_USE_SETUPTOOLS=pyproject.toml"
-			die "No setup.py found and DISTUTILS_USE_SETUPTOOLS!=pyproject.toml"
+			eerror "No setup.py found but pyproject.toml is present.  Please migrate"
+			eerror "the package to use DISTUTILS_USE_PEP517. See:"
+			eerror "  https://projects.gentoo.org/python/guide/distutils.html"
+			die "No setup.py found and PEP517 mode not enabled"
 		fi
 	fi
 }
-- 
2.35.1

[gentoo-dev] [PATCH 02/11] distutils-r1.eclass: Require gpep517 >= 8

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 9909ba70178c..85a5c8496d49 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -205,7 +205,7 @@ _distutils_set_globals() {
 		fi
 
 		bdep='
-			>=dev-python/gpep517-6[${PYTHON_USEDEP}]
+			>=dev-python/gpep517-8[${PYTHON_USEDEP}]
 		'
 		case ${DISTUTILS_USE_PEP517} in
 			flit)
-- 
2.35.1

[gentoo-dev] [PATCH 03/11] install-qa-check.d: Rewrite 60python-pyc to use gpep517

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 metadata/install-qa-check.d/60python-pyc | 168 +++++++++++++----------
 1 file changed, 96 insertions(+), 72 deletions(-)

diff --git a/metadata/install-qa-check.d/60python-pyc b/metadata/install-qa-check.d/60python-pyc
index 47e9a3eea908..fd9434994f2f 100644
--- a/metadata/install-qa-check.d/60python-pyc
+++ b/metadata/install-qa-check.d/60python-pyc
@@ -4,86 +4,110 @@
 # QA check: ensure that Python modules are compiled after installing
 # Maintainer: Python project <python@gentoo.org>
 
-# EAPI guard to prevent errors from trying to import python-utils-r1
-# in unsupported EAPIs.  Please keep the list in sync with the eclass!
-if [[ ${EAPI} == [6-8] ]]; then
-	inherit python-utils-r1
+python_pyc_check() {
+	local save=$(shopt -p nullglob)
+	shopt -s nullglob
+	local progs=( "${EPREFIX}"/usr/lib/python-exec/*/gpep517 )
+	${save}
 
-	python_pyc_check() {
-		local impl missing=() outdated=()
-		for impl in "${_PYTHON_SUPPORTED_IMPLS[@]}"; do
-			_python_export "${impl}" EPYTHON PYTHON
-			[[ -x ${PYTHON} ]] || continue
-			local sitedir=$(python_get_sitedir "${impl}")
+	local invalid=()
+	local mismatched_timestamp=()
+	local mismatched_data=()
+	local missing=()
+	local stray=()
 
-			if [[ -d ${D}${sitedir} ]]; then
-				local suffixes=() subdir=
-				case ${EPYTHON} in
-					python2*)
-						suffixes=( .py{c,o} )
-						;;
-					pypy)
-						suffixes=( .pyc )
-						;;
-					python3*|pypy3*)
-						local tag=$("${PYTHON}" -c 'import sys; print(sys.implementation.cache_tag)')
-						suffixes=( ".${tag}"{,.opt-{1,2}}.pyc )
-						subdir=__pycache__/
-						;;
-					*)
-						# skip testing unknown impl
-						continue
-						;;
-				esac
+	for prog in "${progs[@]}"; do
+		local impl=${prog%/*}
+		impl=${impl##*/}
+		einfo "Verifying compiled files for ${impl}"
+		local kind pyc py
+		while IFS=: read -r kind pyc py extra; do
+			case ${kind} in
+				invalid)
+					invalid+=( "${pyc}" )
+					;;
+				mismatched)
+					case ${extra} in
+						timestamp)
+							mismatched_timestamp+=( "${pyc}" )
+							;;
+						*)
+							mismatched_data+=( "${pyc}" )
+							;;
+					esac
+					;;
+				missing)
+					missing+=( "${pyc}" )
+					;;
+				older)
+					# older warnings were produced by earlier version
+					# of gpep517 but the check was incorrect, so we just
+					# ignore them
+					;;
+				stray)
+					stray+=( "${pyc}" )
+					;;
+			esac
+		done < <("${prog}" verify-pyc --destdir "${D}" --prefix "${EPREFIX}"/usr)
+	done
 
-				einfo "Verifying compiled files in ${sitedir}"
-				local f s
-				while read -d $'\0' -r f; do
-					local dir=${f%/*}
-					local basename=${f##*/}
-					basename=${basename%.py}
+	local found=
+	if [[ ${missing[@]} ]]; then
+		eqawarn
+		eqawarn "QA Notice: This package installs one or more Python modules that are"
+		eqawarn "not byte-compiled."
+		eqawarn "The following files are missing:"
+		eqawarn
+		eqatag -v python-pyc.missing "${missing[@]}"
+		found=1
+	fi
 
-					for s in "${suffixes[@]}"; do
-						local cache=${dir}/${subdir}${basename}${s}
-						if [[ ! -f ${cache} ]]; then
-							missing+=( "${cache}" )
-						elif [[ ${f} -nt ${cache} ]]; then
-							outdated+=( "${cache}" )
-						fi
-					done
-				done < <(find "${D}${sitedir}" -name '*.py' -print0)
-			fi
-		done
+	if [[ ${invalid[@]} ]]; then
+		eqawarn
+		eqawarn "QA Notice: This package installs one or more compiled Python modules"
+		eqawarn "that seem to be invalid (do not have the correct header)."
+		eqawarn "The following files are invalid:"
+		eqawarn
+		eqatag -v python-pyc.invalid "${invalid[@]}"
+		found=1
+	fi
 
-		if [[ ${missing[@]} ]]; then
-			eqawarn
-			eqawarn "QA Notice: This package installs one or more Python modules that are"
-			eqawarn "not byte-compiled."
-			eqawarn "The following files are missing:"
-			eqawarn
-			eqatag -v python-pyc.missing "${missing[@]#${D}}"
-		fi
+	if [[ ${mismatched_data[@]} ]]; then
+		eqawarn
+		eqawarn "QA Notice: This package installs one or more compiled Python modules whose"
+		eqawarn ".py files have different content (size or hash) than recorded:"
+		eqawarn
+		eqatag -v python-pyc.mismatched.data "${mismatched_data[@]}"
+		found=1
+	fi
 
-		if [[ ${outdated[@]} ]]; then
-			eqawarn
-			eqawarn "QA Notice: This package installs one or more compiled Python modules that have"
-			eqawarn "older timestamps than the corresponding source files:"
-			eqawarn
-			eqatag -v python-pyc.outdated "${outdated[@]#${D}}"
-		fi
+	if [[ ${mismatched_timestamp[@]} ]]; then
+		eqawarn
+		eqawarn "QA Notice: This package installs one or more compiled Python modules whose"
+		eqawarn ".py files have different timestamps than recorded:"
+		eqawarn
+		eqatag -v python-pyc.mismatched.timestamp "${mismatched_timestamp[@]}"
+		found=1
+	fi
 
-		if [[ ${missing[@]} || ${outdated[@]} ]]; then
-			eqawarn
-			eqawarn "Please either fix the upstream build system to byte-compile Python modules"
-			eqawarn "correctly, or call python_optimize after installing them.  For more"
-			eqawarn "information, see:"
-			eqawarn "https://projects.gentoo.org/python/guide/helper.html#byte-compiling-python-modules"
-			eqawarn
-		fi
-	}
+	if [[ ${stray[@]} ]]; then
+		eqawarn
+		eqawarn "QA Notice: This package installs one or more compiled Python modules"
+		eqawarn "that do not match installed modules (or their implementation)."
+		eqawarn "The following files are stray:"
+		eqawarn
+		eqatag -v python-pyc.stray "${stray[@]}"
+		found=1
+	fi
 
-	python_pyc_check
-fi
+	if [[ ${found} ]]; then
+		eqawarn
+		eqawarn "For more information on bytecode files and related issues, please see:"
+		eqawarn "  https://projects.gentoo.org/python/guide/qawarn.html#compiled-bytecode-related-warnings"
+	fi
+}
+
+python_pyc_check
 
 : # guarantee successful exit
 
-- 
2.35.1

[gentoo-dev] [PATCH 04/11] distutils-r1.eclass: Bump dev-python/sip dependency

[Michał Górny]

Bug: https://bugs.gentoo.org/851840
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 85a5c8496d49..e4ef1919bc7d 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -265,7 +265,7 @@ _distutils_set_globals() {
 				;;
 			sip)
 				bdep+='
-					>=dev-python/sip-6.6.1[${PYTHON_USEDEP}]
+					>=dev-python/sip-6.6.2[${PYTHON_USEDEP}]
 				'
 				;;
 			standalone)
-- 
2.35.1

[gentoo-dev] [PATCH 05/11] distutils-r1.eclass: Pass EPREFIX to addpredict

[Michał Górny]

Pointed out by Arfrever.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index e4ef1919bc7d..495eead1d5fd 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1647,8 +1647,8 @@ distutils-r1_python_install() {
 		# python likes to compile any module it sees, which triggers sandbox
 		# failures if some packages haven't compiled their modules yet.
 		addpredict "${EPREFIX}/usr/lib/${EPYTHON}"
-		addpredict /usr/lib/pypy3.9
-		addpredict /usr/local # bug 498232
+		addpredict "${EPREFIX}/usr/lib/pypy3.9"
+		addpredict "${EPREFIX}/usr/local" # bug 498232
 
 		if [[ ! ${DISTUTILS_SINGLE_IMPL} ]]; then
 			merge_root=1
-- 
2.35.1

[gentoo-dev] [PATCH 06/11] distutils-r1.eclass: Let gpep517 compile bytecode when 9+ is used

[Michał Górny]

Use the new --optimize option of gpep517 >= 9 when it is installed,
and remove the python_optimize call then.  This conditional logic is
meant to give some additional testing while gpep517-9 is still in ~arch.

Note that this also removes python_optimize call for "no" mode.
However, this mode is used only by a handful of ebuilds, so if any of
them needs an explicit python_optimize call, we can quickly fix them.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 36 ++++++++++++++++++++++++++----------
 1 file changed, 26 insertions(+), 10 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 495eead1d5fd..60f81473c0a6 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1270,13 +1270,25 @@ distutils_wheel_install() {
 	local wheel=${2}
 
 	einfo "  Installing ${wheel##*/} to ${root}"
-	local cmd=(
-		gpep517 install-wheel
-			--destdir="${root}"
-			--interpreter="${PYTHON}"
-			--prefix="${EPREFIX}/usr"
-			"${wheel}"
-	)
+	if has_version -b ">=dev-python/gpep517-9"; then
+		# TODO: inline when we dep on >=9
+		local cmd=(
+			gpep517 install-wheel
+				--destdir="${root}"
+				--interpreter="${PYTHON}"
+				--prefix="${EPREFIX}/usr"
+				--optimize=all
+				"${wheel}"
+		)
+	else
+		local cmd=(
+			gpep517 install-wheel
+				--destdir="${root}"
+				--interpreter="${PYTHON}"
+				--prefix="${EPREFIX}/usr"
+				"${wheel}"
+		)
+	fi
 	printf '%s\n' "${cmd[*]}"
 	"${cmd[@]}" || die "Wheel install failed"
 
@@ -1994,9 +2006,13 @@ _distutils-r1_post_python_install() {
 		done
 
 		if [[ ${DISTUTILS_USE_PEP517} ]]; then
-			# we need to recompile everything here in order to embed
-			# the correct paths
-			python_optimize "${sitedir}"
+			if ! has_version -b ">=dev-python/gpep517-9"
+			then
+				# TODO: remove when we dep on >=9
+				# we need to recompile everything here in order to embed
+				# the correct paths
+				python_optimize "${sitedir}"
+			fi
 		fi
 	fi
 }
-- 
2.35.1

[gentoo-dev] [PATCH 07/11] dev-python/installer: Call python_optimize explicitly

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 dev-python/installer/installer-0.5.1-r1.ebuild | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/dev-python/installer/installer-0.5.1-r1.ebuild b/dev-python/installer/installer-0.5.1-r1.ebuild
index a29b236242ca..39b7336c9b89 100644
--- a/dev-python/installer/installer-0.5.1-r1.ebuild
+++ b/dev-python/installer/installer-0.5.1-r1.ebuild
@@ -35,3 +35,8 @@ distutils_enable_tests pytest
 python_compile() {
 	python_domodule src/installer "${WORKDIR}"/*.dist-info
 }
+
+python_install() {
+	distutils-r1_python_install
+	python_optimize
+}
-- 
2.35.1

[gentoo-dev] [PATCH 08/11] dev-python/tomli: Call python_optimize explicitly

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 dev-python/tomli/tomli-2.0.1-r1.ebuild | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/dev-python/tomli/tomli-2.0.1-r1.ebuild b/dev-python/tomli/tomli-2.0.1-r1.ebuild
index f24193c0c9fe..4a4c0bc1ea17 100644
--- a/dev-python/tomli/tomli-2.0.1-r1.ebuild
+++ b/dev-python/tomli/tomli-2.0.1-r1.ebuild
@@ -34,3 +34,8 @@ distutils_enable_tests unittest
 python_compile() {
 	python_domodule src/tomli "${WORKDIR}"/*.dist-info
 }
+
+python_install() {
+	distutils-r1_python_install
+	python_optimize
+}
-- 
2.35.1

[gentoo-dev] [PATCH 09/11] net-wireless/blueman: Invoke eautomake to fix py-compile script

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 net-wireless/blueman/blueman-2.3.1.ebuild | 2 +-
 net-wireless/blueman/blueman-9999.ebuild  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/net-wireless/blueman/blueman-2.3.1.ebuild b/net-wireless/blueman/blueman-2.3.1.ebuild
index dbedcfc7eae6..997963609ec2 100644
--- a/net-wireless/blueman/blueman-2.3.1.ebuild
+++ b/net-wireless/blueman/blueman-2.3.1.ebuild
@@ -97,7 +97,7 @@ pkg_setup() {
 }
 
 src_prepare() {
-	[[ ${PV} == 9999 ]] && eautoreconf
+	[[ ${PV} == 9999 ]] && eautoreconf || eautomake
 	distutils-r1_src_prepare
 }
 
diff --git a/net-wireless/blueman/blueman-9999.ebuild b/net-wireless/blueman/blueman-9999.ebuild
index dbedcfc7eae6..997963609ec2 100644
--- a/net-wireless/blueman/blueman-9999.ebuild
+++ b/net-wireless/blueman/blueman-9999.ebuild
@@ -97,7 +97,7 @@ pkg_setup() {
 }
 
 src_prepare() {
-	[[ ${PV} == 9999 ]] && eautoreconf
+	[[ ${PV} == 9999 ]] && eautoreconf || eautomake
 	distutils-r1_src_prepare
 }
 
-- 
2.35.1

[gentoo-dev] [PATCH 10/11] dev-python/pygobject: Call python_optimize explicitly

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 dev-python/pygobject/pygobject-3.42.2.ebuild | 1 +
 1 file changed, 1 insertion(+)

diff --git a/dev-python/pygobject/pygobject-3.42.2.ebuild b/dev-python/pygobject/pygobject-3.42.2.ebuild
index 217ae21f2c4a..d411290d7520 100644
--- a/dev-python/pygobject/pygobject-3.42.2.ebuild
+++ b/dev-python/pygobject/pygobject-3.42.2.ebuild
@@ -65,6 +65,7 @@ python_test() {
 
 python_install() {
 	meson_src_install
+	python_optimize
 }
 
 python_install_all() {
-- 
2.35.1

[gentoo-dev] [PATCH 11/11] python-utils-r1.eclass: Sterilize pytest-sugar plugin

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/python-utils-r1.eclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index b793a1f13e0f..89013591c31b 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -1331,6 +1331,9 @@ epytest() {
 		# sterilize pytest-markdown as it runs code snippets from all
 		# *.md files found without any warning
 		-p no:markdown
+		# pytest-sugar undoes everything that's good about pytest output
+		# and makes it hard to read logs
+		-p no:sugar
 	)
 	local x
 	for x in "${EPYTEST_DESELECT[@]}"; do
-- 
2.35.1

Re: [gentoo-dev] [PATCH 09/11] net-wireless/blueman: Invoke eautomake to fix py-compile script

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 614 bytes --]

>>>>> On Fri, 29 Jul 2022, Michał Górny wrote:

> --- a/net-wireless/blueman/blueman-2.3.1.ebuild
> +++ b/net-wireless/blueman/blueman-2.3.1.ebuild
> @@ -97,7 +97,7 @@ pkg_setup() {
>  }
>  
>  src_prepare() {
> -	[[ ${PV} == 9999 ]] && eautoreconf
> +	[[ ${PV} == 9999 ]] && eautoreconf || eautomake

So if eautoreconf returns with nonzero status, eautomake will be called?
Or does eautoreconf always return zero (I believe it does)?

However, I don't want to think about that when reading the code, so
could you make it an if-then-else statement instead?

Same for the live ebuild.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]