[gentoo-dev] [PATCH v2 0/7] wheel reuse optimization, EPYTEST_FLAGS and scikit-build-core

[gentoo-dev] [PATCH v2 0/7] wheel reuse optimization, EPYTEST_FLAGS and scikit-build-core

[Michał Górny]

Hi,

Changed in v2:

Made DISTUTILS_WHEELS an associative array, and included source
directory in it.  This ensures that if an ebuild calls
distutils-r1_python_compile multiple times, we don't end up reusing
wheels created for another source directory.


Michał Górny (7):
  distutils-r1.eclass: Set DISTUTILS_WHEEL_PATH in PEP517 install
  distutils-r1.eclass: Store created wheels in DISTUTILS_WHEELS
  distutils-r1.eclass: Add a QA warning for pure Python file mismatch
  distutils-r1.eclass: Support reusing prior wheels when compatible
  python-utils-r1.eclass: Support passing EPYTEST_FLAGS
  distutils-r1.eclass: Update scikit-build-core to 0.9.4
  distutils-r1.eclass: Pass ninja options to scikit-build-core

 eclass/distutils-r1.eclass    | 112 ++++++++++++++++++++++++++++++++--
 eclass/python-utils-r1.eclass |  11 +++-
 2 files changed, 116 insertions(+), 7 deletions(-)

-- 
2.45.1

[gentoo-dev] [PATCH v2 1/7] distutils-r1.eclass: Set DISTUTILS_WHEEL_PATH in PEP517 install

[Michał Górny]

Store the created wheel path in DISTUTILS_WHEEL_PATH when returning
from distutils_pep517_install.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index a67122a59a33..1037c0abe239 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1343,6 +1343,10 @@ distutils_wheel_install() {
 		\) -delete || die
 }
 
+# @VARIABLE: DISTUTILS_WHEEL_PATH
+# @DESCRIPTION:
+# Path to the wheel created by distutils_pep517_install.
+
 # @FUNCTION: distutils_pep517_install
 # @USAGE: <root>
 # @DESCRIPTION:
@@ -1350,7 +1354,8 @@ distutils_wheel_install() {
 # backend and install it into <root>.
 #
 # This function is intended for expert use only.  It does not handle
-# wrapping executables.
+# wrapping executables.  The wheel path is returned
+# in DISTUTILS_WHEEL_PATH variable.
 distutils_pep517_install() {
 	debug-print-function ${FUNCNAME} "${@}"
 	[[ ${#} -eq 1 ]] || die "${FUNCNAME} takes exactly one argument: root"
@@ -1523,6 +1528,8 @@ distutils_pep517_install() {
 	[[ -n ${wheel} ]] || die "No wheel name returned"
 
 	distutils_wheel_install "${root}" "${WHEEL_BUILD_DIR}/${wheel}"
+
+	DISTUTILS_WHEEL_PATH=${WHEEL_BUILD_DIR}/${wheel}
 }
 
 # @FUNCTION: distutils-r1_python_compile
-- 
2.45.1

[gentoo-dev] [PATCH v2 2/7] distutils-r1.eclass: Store created wheels in DISTUTILS_WHEELS

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 1037c0abe239..955c41fe4e2d 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1532,6 +1532,15 @@ distutils_pep517_install() {
 	DISTUTILS_WHEEL_PATH=${WHEEL_BUILD_DIR}/${wheel}
 }
 
+# @VARIABLE: DISTUTILS_WHEELS
+# @DESCRIPTION:
+# An associative array of wheels created as a result
+# of distutils-r1_python_compile invocations, mapped to the source
+# directories.  Note that this includes only wheels implicitly created
+# by the eclass, and not wheels created as a result of direct
+# distutils_pep517_install calls in the ebuild.
+declare -g -A DISTUTILS_WHEELS=()
+
 # @FUNCTION: distutils-r1_python_compile
 # @USAGE: [additional-args...]
 # @DESCRIPTION:
@@ -1541,6 +1550,7 @@ distutils_pep517_install() {
 #
 # If DISTUTILS_USE_PEP517 is set to any other value, builds a wheel
 # using the PEP517 backend and installs it into ${BUILD_DIR}/install.
+# Path to the wheel is then added to DISTUTILS_WHEELS array.
 #
 # In legacy mode, runs 'esetup.py build'. Any parameters passed to this
 # function will be appended to setup.py invocation, i.e. passed
@@ -1576,6 +1586,7 @@ distutils-r1_python_compile() {
 
 	if [[ ${DISTUTILS_USE_PEP517} ]]; then
 		distutils_pep517_install "${BUILD_DIR}/install"
+		DISTUTILS_WHEELS+=( "${DISTUTILS_WHEEL_PATH}" "${PWD}" )
 	fi
 }
 
-- 
2.45.1

[gentoo-dev] [PATCH v2 3/7] distutils-r1.eclass: Add a QA warning for pure Python file mismatch

[Michał Górny]

If the package is creating at least one pure Python wheel, check whether
the baseline package contents (i.e. everything but compiled Python
modules, extensions and .dist-info) match between implementations.
This is meant to ensure that we can safely optimize builds by reusing
pure Python wheels from previous builds.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 40 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 955c41fe4e2d..29e901720e6c 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -2022,6 +2022,44 @@ distutils-r1_src_configure() {
 	return ${ret}
 }
 
+# @FUNCTION: _distutils-r1_compare_installed_files
+# @INTERNAL
+# @DESCRIPTION:
+# Verify the the match between files installed between this and previous
+# implementation.
+_distutils-r1_compare_installed_files() {
+	debug-print-function ${FUNCNAME} "${@}"
+
+	# QA check requires diff(1).
+	if ! type -P diff &>/dev/null; then
+		return
+	fi
+
+	# Perform the check only if at least one potentially reusable wheel
+	# has been produced.  Nonpure packages (e.g. NumPy) may install
+	# interpreter configuration details into sitedir.
+	if [[ ${!DISTUTILS_WHEELS[*]} != *-none-any.whl* &&
+			${!DISTUTILS_WHEELS[*]} != *-abi3-*.whl ]]; then
+		return
+	fi
+
+	local sitedir=${BUILD_DIR}/install$(python_get_sitedir)
+	if [[ -n ${_DISTUTILS_PREVIOUS_SITE} ]]; then
+		diff -dur \
+			--exclude=__pycache__ \
+			--exclude='*.dist-info' \
+			--exclude="*$(get_modname)" \
+			"${_DISTUTILS_PREVIOUS_SITE}" "${sitedir}"
+		if [[ ${?} -ne 0 ]]; then
+			eqawarn "Package creating at least one pure Python wheel installs different"
+			eqawarn "Python files between implementations.  See diff in build log, above"
+			eqawarn "this message."
+		fi
+	fi
+
+	_DISTUTILS_PREVIOUS_SITE=${sitedir}
+}
+
 # @FUNCTION: _distutils-r1_post_python_compile
 # @INTERNAL
 # @DESCRIPTION:
@@ -2056,6 +2094,8 @@ _distutils-r1_post_python_compile() {
 		find "${bindir}" -type f -exec sed -i \
 			-e "1s@^#!\(${EPREFIX}/usr/bin/\(python\|pypy\)\)@#!${root}\1@" \
 			{} + || die
+
+		_distutils-r1_compare_installed_files
 	fi
 }
 
-- 
2.45.1

[gentoo-dev] [PATCH v2 4/7] distutils-r1.eclass: Support reusing prior wheels when compatible

[Michał Górny]

Support reusing the wheels built for earlier Python implementations
if they are compatible with the subsequent implementations being built.
This includes pure Python wheels in packages that do not set
DISTUTILS_EXT, and stable ABI wheels.

Closes: https://bugs.gentoo.org/931689
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 29e901720e6c..a8f9817a3cf0 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -189,6 +189,18 @@ esac
 #     ${DISTUTILS_DEPS}"
 # @CODE
 
+# @ECLASS_VARIABLE: DISTUTILS_ALLOW_WHEEL_REUSE
+# @DEFAULT_UNSET
+# @USER_VARIABLE
+# @DESCRIPTION:
+# If set to a non-empty value, the eclass is allowed to reuse a wheel
+# that was built for a prior Python implementation, provided that it is
+# compatible with the current one, rather than building a new one.
+#
+# This is an optimization that can avoid the overhead of calling into
+# the build system in pure Python packages and packages using the stable
+# Python ABI.
+
 if [[ -z ${_DISTUTILS_R1_ECLASS} ]]; then
 _DISTUTILS_R1_ECLASS=1
 
@@ -1585,6 +1597,32 @@ distutils-r1_python_compile() {
 	esac
 
 	if [[ ${DISTUTILS_USE_PEP517} ]]; then
+		if [[ ${DISTUTILS_ALLOW_WHEEL_REUSE} ]]; then
+			local whl
+			for whl in "${!DISTUTILS_WHEELS[@]}"; do
+				# use only wheels corresponding to the current directory
+				if [[ ${PWD} != ${DISTUTILS_WHEELS["${whl}"]} ]]; then
+					continue
+				fi
+
+				# 1. Use pure Python wheels only if we're not expected
+				# to build extensions.  Otherwise, we may end up
+				# not building the extension at all when e.g. PyPy3
+				# is built without one.
+				#
+				# 2. For CPython, we can reuse stable ABI wheels.  Note
+				# that this relies on the assumption that we're building
+				# from the oldest to the newest implementation,
+				# and the wheels are forward-compatible.
+				if [[ ( ! ${DISTUTILS_EXT} && ${whl} == *py3-none-any* ) ||
+					( ${EPYTHON} == python* && ${whl} == *-abi3-* ) ]]
+				then
+					distutils_wheel_install "${BUILD_DIR}/install" "${whl}"
+					return
+				fi
+			done
+		fi
+
 		distutils_pep517_install "${BUILD_DIR}/install"
 		DISTUTILS_WHEELS+=( "${DISTUTILS_WHEEL_PATH}" "${PWD}" )
 	fi
-- 
2.45.1

[gentoo-dev] [PATCH v2 5/7] python-utils-r1.eclass: Support passing EPYTEST_FLAGS

[Michał Górny]

Closes: https://bugs.gentoo.org/905863
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/python-utils-r1.eclass | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index 275ac3a96523..584ed831f816 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -1321,6 +1321,15 @@ _python_check_occluded_packages() {
 # Specifies the number of jobs for parallel (pytest-xdist) test runs.
 # When unset, defaults to -j from MAKEOPTS, or the current nproc.
 
+# @ECLASS_VARIABLE: EPYTEST_FLAGS
+# @USER_VARIABLE
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# Additional flags to pass to pytest.  This is intended to be set
+# in the environment when debugging packages (options such as -x or -s
+# are useful here), rather than globally.  It must not be set
+# in ebuilds.
+
 # @FUNCTION: epytest
 # @USAGE: [<args>...]
 # @DESCRIPTION:
@@ -1432,7 +1441,7 @@ epytest() {
 	for x in "${EPYTEST_IGNORE[@]}"; do
 		args+=( --ignore "${x}" )
 	done
-	set -- "${EPYTHON}" -m pytest "${args[@]}" "${@}"
+	set -- "${EPYTHON}" -m pytest "${args[@]}" "${@}" ${EPYTEST_FLAGS}
 
 	echo "${@}" >&2
 	"${@}" || die -n "pytest failed with ${EPYTHON}"
-- 
2.45.1

[gentoo-dev] [PATCH v2 6/7] distutils-r1.eclass: Update scikit-build-core to 0.9.4

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index a8f9817a3cf0..3aa2c8984ab2 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -279,7 +279,7 @@ _distutils_set_globals() {
 				;;
 			scikit-build-core)
 				bdep+='
-					>=dev-python/scikit-build-core-0.8.2[${PYTHON_USEDEP}]
+					>=dev-python/scikit-build-core-0.9.4[${PYTHON_USEDEP}]
 				'
 				;;
 			setuptools)
@@ -1457,8 +1457,6 @@ distutils_pep517_install() {
 				"${DISTUTILS_ARGS[@]}"
 			)
 
-			# NB: we need to pass strings for boolean fields
-			# https://github.com/scikit-build/scikit-build-core/issues/707
 			config_settings=$(
 				"${EPYTHON}" - "${cmake_args[@]}" <<-EOF || die
 					import json
@@ -1466,8 +1464,8 @@ distutils_pep517_install() {
 					print(json.dumps({
 						"cmake.args": ";".join(sys.argv[1:]),
 						"cmake.build-type": "${CMAKE_BUILD_TYPE}",
-						"cmake.verbose": "true",
-						"install.strip": "false",
+						"cmake.verbose": True,
+						"install.strip": False,
 					}))
 				EOF
 			)
-- 
2.45.1

[gentoo-dev] [PATCH v2 7/7] distutils-r1.eclass: Pass ninja options to scikit-build-core

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 3aa2c8984ab2..71b80fafe1a5 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1457,11 +1457,17 @@ distutils_pep517_install() {
 				"${DISTUTILS_ARGS[@]}"
 			)
 
+			local -x NINJAOPTS=$(get_NINJAOPTS)
 			config_settings=$(
 				"${EPYTHON}" - "${cmake_args[@]}" <<-EOF || die
 					import json
+					import os
+					import shlex
 					import sys
+
+					ninjaopts = shlex.split(os.environ["NINJAOPTS"])
 					print(json.dumps({
+						"build.tool-args": ninjaopts,
 						"cmake.args": ";".join(sys.argv[1:]),
 						"cmake.build-type": "${CMAKE_BUILD_TYPE}",
 						"cmake.verbose": True,
-- 
2.45.1