[gentoo-dev] [PATCH 0/7] distutils-r1.eclass: wheel reuse optimization, EPYTEST_FLAGS and scikit-build-core updates

[gentoo-dev] [PATCH 0/7] distutils-r1.eclass: wheel reuse optimization, EPYTEST_FLAGS and scikit-build-core updates

[Michał Górny]

Hello,

Here's a small batch of patches that:

1. Add support for reusing prior wheels if they are compatible to avoid
invoking the (slow) build system multiple times when building for
multiple targes.  This is currently opt-in (via make.conf variable)
and can benefit us in two cases:

a. in pure Python packages to avoid repeatedly building wheel with
the same files, and

b. in packages using the stable API to avoid building identical
extensions separately for every impl.

2. Add support for EPYTEST_FLAGS variable that can be used by the user
to quickly append additional flags to pytest invocation (e.g. `-x` to
make them stop on first failure, or `-s` to disable output capture).

3. Support passing ninja options to scikit-build-core, and update
invocation for bugfixes from 0.9.4.



Michał Górny (7):
  distutils-r1.eclass: Set DISTUTILS_WHEEL_PATH in PEP517 install
  distutils-r1.eclass: Store created wheels in DISTUTILS_WHEELS
  distutils-r1.eclass: Add a QA warning for pure Python file mismatch
  distutils-r1.eclass: Support reusing prior wheels when compatible
  python-utils-r1.eclass: Support passing EPYTEST_FLAGS
  distutils-r1.eclass: Update scikit-build-core to 0.9.4
  distutils-r1.eclass: Pass ninja options to scikit-build-core

 eclass/distutils-r1.eclass    | 106 ++++++++++++++++++++++++++++++++--
 eclass/python-utils-r1.eclass |  11 +++-
 2 files changed, 110 insertions(+), 7 deletions(-)

-- 
2.45.1

[gentoo-dev] [PATCH 1/7] distutils-r1.eclass: Set DISTUTILS_WHEEL_PATH in PEP517 install

[Michał Górny]

Store the created wheel path in DISTUTILS_WHEEL_PATH when returning
from distutils_pep517_install.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index a67122a59a33..1037c0abe239 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1343,6 +1343,10 @@ distutils_wheel_install() {
 		\) -delete || die
 }
 
+# @VARIABLE: DISTUTILS_WHEEL_PATH
+# @DESCRIPTION:
+# Path to the wheel created by distutils_pep517_install.
+
 # @FUNCTION: distutils_pep517_install
 # @USAGE: <root>
 # @DESCRIPTION:
@@ -1350,7 +1354,8 @@ distutils_wheel_install() {
 # backend and install it into <root>.
 #
 # This function is intended for expert use only.  It does not handle
-# wrapping executables.
+# wrapping executables.  The wheel path is returned
+# in DISTUTILS_WHEEL_PATH variable.
 distutils_pep517_install() {
 	debug-print-function ${FUNCNAME} "${@}"
 	[[ ${#} -eq 1 ]] || die "${FUNCNAME} takes exactly one argument: root"
@@ -1523,6 +1528,8 @@ distutils_pep517_install() {
 	[[ -n ${wheel} ]] || die "No wheel name returned"
 
 	distutils_wheel_install "${root}" "${WHEEL_BUILD_DIR}/${wheel}"
+
+	DISTUTILS_WHEEL_PATH=${WHEEL_BUILD_DIR}/${wheel}
 }
 
 # @FUNCTION: distutils-r1_python_compile
-- 
2.45.1

[gentoo-dev] [PATCH 2/7] distutils-r1.eclass: Store created wheels in DISTUTILS_WHEELS

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 1037c0abe239..89223b248157 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1532,6 +1532,14 @@ distutils_pep517_install() {
 	DISTUTILS_WHEEL_PATH=${WHEEL_BUILD_DIR}/${wheel}
 }
 
+# @VARIABLE: DISTUTILS_WHEELS
+# @DESCRIPTION:
+# An array of wheels created as a result of distutils-r1_python_compile.
+# Note that this includes only wheels implicitly created by the eclass,
+# and not wheels created as a result of direct distutils_pep517_install
+# calls in the ebuild.
+DISTUTILS_WHEELS=()
+
 # @FUNCTION: distutils-r1_python_compile
 # @USAGE: [additional-args...]
 # @DESCRIPTION:
@@ -1541,6 +1549,7 @@ distutils_pep517_install() {
 #
 # If DISTUTILS_USE_PEP517 is set to any other value, builds a wheel
 # using the PEP517 backend and installs it into ${BUILD_DIR}/install.
+# Path to the wheel is then added to DISTUTILS_WHEELS array.
 #
 # In legacy mode, runs 'esetup.py build'. Any parameters passed to this
 # function will be appended to setup.py invocation, i.e. passed
@@ -1576,6 +1585,7 @@ distutils-r1_python_compile() {
 
 	if [[ ${DISTUTILS_USE_PEP517} ]]; then
 		distutils_pep517_install "${BUILD_DIR}/install"
+		DISTUTILS_WHEELS+=( "${DISTUTILS_WHEEL_PATH}" )
 	fi
 }
 
-- 
2.45.1

[gentoo-dev] [PATCH 3/7] distutils-r1.eclass: Add a QA warning for pure Python file mismatch

[Michał Górny]

If the package is creating at least one pure Python wheel, check whether
the baseline package contents (i.e. everything but compiled Python
modules, extensions and .dist-info) match between implementations.
This is meant to ensure that we can safely optimize builds by reusing
pure Python wheels from previous builds.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 40 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 89223b248157..f014a184885a 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -2021,6 +2021,44 @@ distutils-r1_src_configure() {
 	return ${ret}
 }
 
+# @FUNCTION: _distutils-r1_compare_installed_files
+# @INTERNAL
+# @DESCRIPTION:
+# Verify the the match between files installed between this and previous
+# implementation.
+_distutils-r1_compare_installed_files() {
+	debug-print-function ${FUNCNAME} "${@}"
+
+	# QA check requires diff(1).
+	if ! type -P diff &>/dev/null; then
+		return
+	fi
+
+	# Perform the check only if at least one potentially reusable wheel
+	# has been produced.  Nonpure packages (e.g. NumPy) may install
+	# interpreter configuration details into sitedir.
+	if [[ ${DISTUTILS_WHEELS[*]} != *-none-any.whl* &&
+			${DISTUTILS_WHEELS[*]} != *-abi3-*.whl ]]; then
+		return
+	fi
+
+	local sitedir=${BUILD_DIR}/install$(python_get_sitedir)
+	if [[ -n ${_DISTUTILS_PREVIOUS_SITE} ]]; then
+		diff -dur \
+			--exclude=__pycache__ \
+			--exclude='*.dist-info' \
+			--exclude="*$(get_modname)" \
+			"${_DISTUTILS_PREVIOUS_SITE}" "${sitedir}"
+		if [[ ${?} -ne 0 ]]; then
+			eqawarn "Package creating at least one pure Python wheel installs different"
+			eqawarn "Python files between implementations.  See diff in build log, above"
+			eqawarn "this message."
+		fi
+	fi
+
+	_DISTUTILS_PREVIOUS_SITE=${sitedir}
+}
+
 # @FUNCTION: _distutils-r1_post_python_compile
 # @INTERNAL
 # @DESCRIPTION:
@@ -2055,6 +2093,8 @@ _distutils-r1_post_python_compile() {
 		find "${bindir}" -type f -exec sed -i \
 			-e "1s@^#!\(${EPREFIX}/usr/bin/\(python\|pypy\)\)@#!${root}\1@" \
 			{} + || die
+
+		_distutils-r1_compare_installed_files
 	fi
 }
 
-- 
2.45.1

[gentoo-dev] [PATCH 4/7] distutils-r1.eclass: Support reusing prior wheels when compatible

[Michał Górny]

Support reusing the wheels built for earlier Python implementations
if they are compatible with the subsequent implementations being built.
This includes pure Python wheels in packages that do not set
DISTUTILS_EXT, and stable ABI wheels.

Closes: https://bugs.gentoo.org/931689
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index f014a184885a..e4d53083124e 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -189,6 +189,18 @@ esac
 #     ${DISTUTILS_DEPS}"
 # @CODE
 
+# @ECLASS_VARIABLE: DISTUTILS_ALLOW_WHEEL_REUSE
+# @DEFAULT_UNSET
+# @USER_VARIABLE
+# @DESCRIPTION:
+# If set to a non-empty value, the eclass is allowed to reuse a wheel
+# that was built for a prior Python implementation, provided that it is
+# compatible with the current one, rather than building a new one.
+#
+# This is an optimization that can avoid the overhead of calling into
+# the build system in pure Python packages and packages using the stable
+# Python ABI.
+
 if [[ -z ${_DISTUTILS_R1_ECLASS} ]]; then
 _DISTUTILS_R1_ECLASS=1
 
@@ -1584,6 +1596,27 @@ distutils-r1_python_compile() {
 	esac
 
 	if [[ ${DISTUTILS_USE_PEP517} ]]; then
+		if [[ ${DISTUTILS_ALLOW_WHEEL_REUSE} ]]; then
+			local whl
+			for whl in "${DISTUTILS_WHEELS[@]}"; do
+				# 1. Use pure Python wheels only if we're not expected
+				# to build extensions.  Otherwise, we may end up
+				# not building the extension at all when e.g. PyPy3
+				# is built without one.
+				#
+				# 2. For CPython, we can reuse stable ABI wheels.  Note
+				# that this relies on the assumption that we're building
+				# from the oldest to the newest implementation,
+				# and the wheels are forward-compatible.
+				if [[ ( ! ${DISTUTILS_EXT} && ${whl} == *py3-none-any* ) ||
+					( ${EPYTHON} == python* && ${whl} == *-abi3-* ) ]]
+				then
+					distutils_wheel_install "${BUILD_DIR}/install" "${whl}"
+					return
+				fi
+			done
+		fi
+
 		distutils_pep517_install "${BUILD_DIR}/install"
 		DISTUTILS_WHEELS+=( "${DISTUTILS_WHEEL_PATH}" )
 	fi
-- 
2.45.1

[gentoo-dev] [PATCH 5/7] python-utils-r1.eclass: Support passing EPYTEST_FLAGS

[Michał Górny]

Closes: https://bugs.gentoo.org/905863
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/python-utils-r1.eclass | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 275ac3a96523..584ed831f816 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -1321,6 +1321,15 @@ _python_check_occluded_packages() {
 # Specifies the number of jobs for parallel (pytest-xdist) test runs.
 # When unset, defaults to -j from MAKEOPTS, or the current nproc.
 
+# @ECLASS_VARIABLE: EPYTEST_FLAGS
+# @USER_VARIABLE
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# Additional flags to pass to pytest.  This is intended to be set
+# in the environment when debugging packages (options such as -x or -s
+# are useful here), rather than globally.  It must not be set
+# in ebuilds.
+
 # @FUNCTION: epytest
 # @USAGE: [<args>...]
 # @DESCRIPTION:
@@ -1432,7 +1441,7 @@ epytest() {
 	for x in "${EPYTEST_IGNORE[@]}"; do
 		args+=( --ignore "${x}" )
 	done
-	set -- "${EPYTHON}" -m pytest "${args[@]}" "${@}"
+	set -- "${EPYTHON}" -m pytest "${args[@]}" "${@}" ${EPYTEST_FLAGS}
 
 	echo "${@}" >&2
 	"${@}" || die -n "pytest failed with ${EPYTHON}"
-- 
2.45.1

[gentoo-dev] [PATCH 6/7] distutils-r1.eclass: Update scikit-build-core to 0.9.4

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index e4d53083124e..02921919c7ef 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -279,7 +279,7 @@ _distutils_set_globals() {
 				;;
 			scikit-build-core)
 				bdep+='
-					>=dev-python/scikit-build-core-0.8.2[${PYTHON_USEDEP}]
+					>=dev-python/scikit-build-core-0.9.4[${PYTHON_USEDEP}]
 				'
 				;;
 			setuptools)
@@ -1457,8 +1457,6 @@ distutils_pep517_install() {
 				"${DISTUTILS_ARGS[@]}"
 			)
 
-			# NB: we need to pass strings for boolean fields
-			# https://github.com/scikit-build/scikit-build-core/issues/707
 			config_settings=$(
 				"${EPYTHON}" - "${cmake_args[@]}" <<-EOF || die
 					import json
@@ -1466,8 +1464,8 @@ distutils_pep517_install() {
 					print(json.dumps({
 						"cmake.args": ";".join(sys.argv[1:]),
 						"cmake.build-type": "${CMAKE_BUILD_TYPE}",
-						"cmake.verbose": "true",
-						"install.strip": "false",
+						"cmake.verbose": True,
+						"install.strip": False,
 					}))
 				EOF
 			)
-- 
2.45.1

[gentoo-dev] [PATCH 7/7] distutils-r1.eclass: Pass ninja options to scikit-build-core

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 02921919c7ef..9b2fc0583149 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1457,11 +1457,17 @@ distutils_pep517_install() {
 				"${DISTUTILS_ARGS[@]}"
 			)
 
+			local -x NINJAOPTS=$(get_NINJAOPTS)
 			config_settings=$(
 				"${EPYTHON}" - "${cmake_args[@]}" <<-EOF || die
 					import json
+					import os
+					import shlex
 					import sys
+
+					ninjaopts = shlex.split(os.environ["NINJAOPTS"])
 					print(json.dumps({
+						"build.tool-args": ninjaopts,
 						"cmake.args": ";".join(sys.argv[1:]),
 						"cmake.build-type": "${CMAKE_BUILD_TYPE}",
 						"cmake.verbose": True,
-- 
2.45.1