[gentoo-dev] [PATCH 0/4] distutils-r1.eclass: symlinking support

[gentoo-dev] [PATCH 0/4] distutils-r1.eclass: symlinking support

[Michał Górny]

Hi,

This batch includes some minor refactoring, plus two major experimental
features (to be enabled via `make.conf` by people ready to test them):

1. DISTUTILS_ALLOW_PYC_SYMLINKS to symlink matching .pyc files across
   optimization levels, e.g.:

     __init__.opt-2.pyc -> __init__.opt-1.pyc
     __init__.opt-1.pyc -> __init__.pyc

2. DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS to symlink matching files across
   different Python implementations and save lots of space, e.g.:

     python3.11/site-packages/frobnicate.py
     python3.12/site-packages/frobnicate.py -> ../../python3.11/site-packages/frobnicate.py
     python3.13/site-packages/frobnicate.py -> ../../python3.11/site-packages/frobnicate.py

   This can specifically help with large .py files, lots of data
   installed in package directory and stable ABI extensions.

Both features are opt-in, and at this point considered experimental
and dangerous.  We may consider making the first one opt-out
in the future, but I want to test both extensively first.

Note that we may independently of this start moving package data
out of site-packages and into /usr/share for some large packages.


Michał Górny (4):
  python-utils-r1.eclass: Simplify _python_impl_matches
  distutils-r1.eclass: Quote DISTUTILS_ALLOW_WHEEL_REUSE
  distutils-r1.eclass: Support .pyc symlinking
  distutils-r1.eclass: Support cross-impl symlinks

 eclass/distutils-r1.eclass    | 81 ++++++++++++++++++++++++++++++++++-
 eclass/python-utils-r1.eclass |  4 +-
 2 files changed, 81 insertions(+), 4 deletions(-)

[gentoo-dev] [PATCH 1/4] python-utils-r1.eclass: Simplify _python_impl_matches

[Michał Górny]

Remove special case for '3.10' in _python_impl_matches -- it was only
necessary because we needed to handle 'pypy3' target specially,
and that is no longer the case.  Technically, the code checks for
'pypy3_10' but that's not a problem, since there is no such a thing.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/python-utils-r1.eclass | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index ef05a58e1b13..3a4a3f19a1be 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -230,9 +230,7 @@ _python_impl_matches() {
 				fi
 				return 0
 				;;
-			3.10)
-				;;
-			3.8|3.9|3.1[1-3])
+			3.[89]|3.1[0-3])
 				[[ ${impl%t} == python${pattern/./_} || ${impl} == pypy${pattern/./_} ]] &&
 					return 0
 				;;

[gentoo-dev] [PATCH 2/4] distutils-r1.eclass: Quote DISTUTILS_ALLOW_WHEEL_REUSE

[Michał Górny]

Reported-by: Ionen Wolkens <ionen@gentoo.org>
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 880e33cd741f..799206f7eb03 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -215,7 +215,7 @@
 # This is an optimization that can avoid the overhead of calling into
 # the build system in pure Python packages and packages using the stable
 # Python ABI.
-: ${DISTUTILS_ALLOW_WHEEL_REUSE=1}
+: "${DISTUTILS_ALLOW_WHEEL_REUSE=1}"
 
 # @ECLASS_VARIABLE: BUILD_DIR
 # @OUTPUT_VARIABLE

[gentoo-dev] [PATCH 3/4] distutils-r1.eclass: Support .pyc symlinking

[Michał Górny]

Support replacing identical .pyc files with symlinks, using gpep517-19
option `--symlink-pyc`.  Opt in via `DISTUTILS_ALLOW_PYC_SYMLINKS`
variable.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 799206f7eb03..32cff457e996 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -217,6 +217,16 @@
 # Python ABI.
 : "${DISTUTILS_ALLOW_WHEEL_REUSE=1}"
 
+# @ECLASS_VARIABLE: DISTUTILS_ALLOW_PYC_SYMLINKS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# If set to a non-empty value, the eclass is allowed to create symlinks
+# between different optimization levels of compiled .pyc files.
+#
+# This is an optimization that can slightly reduce disk space usage.
+# Note that it requires >=dev-python/gpep517-19.
+: "${DISTUTILS_ALLOW_PYC_SYMLINKS=}"
+
 # @ECLASS_VARIABLE: BUILD_DIR
 # @OUTPUT_VARIABLE
 # @DEFAULT_UNSET
@@ -1220,6 +1230,9 @@ distutils_wheel_install() {
 			--optimize=all
 			"${wheel}"
 	)
+	if [[ ${DISTUTILS_ALLOW_PYC_SYMLINKS} ]]; then
+		cmd+=( --symlink-pyc )
+	fi
 	printf '%s\n' "${cmd[*]}"
 	"${cmd[@]}" || die "Wheel install failed"
 

[gentoo-dev] [PATCH 4/4] distutils-r1.eclass: Support cross-impl symlinks

[Michał Górny]

Support replacing installed files with cross-implementation symlinks
to save space.  Opt-in via `DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS`
variable.

Closes: https://bugs.gentoo.org/954762
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 66 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 32cff457e996..d2e4f9ce7ba0 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -227,6 +227,18 @@
 # Note that it requires >=dev-python/gpep517-19.
 : "${DISTUTILS_ALLOW_PYC_SYMLINKS=}"
 
+# @ECLASS_VARIABLE: DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# If set to a non-empty value, the eclass will replace identical files
+# from installed wheels with symlinks, across different Python
+# implementations.  This may include .py files, variety of data files
+# and stable ABI extensions.
+#
+# This is an optimization that can reduce disk space usage when packages
+# are built for multiple Python implementations.
+: "${DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS=}"
+
 # @ECLASS_VARIABLE: BUILD_DIR
 # @OUTPUT_VARIABLE
 # @DEFAULT_UNSET
@@ -1233,6 +1245,33 @@ distutils_wheel_install() {
 	if [[ ${DISTUTILS_ALLOW_PYC_SYMLINKS} ]]; then
 		cmd+=( --symlink-pyc )
 	fi
+	if [[ ${DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS} && ${_DISTUTILS_SITES[@]} ]]
+	then
+		set -- "${_DISTUTILS_SITES[@]}"
+		while [[ ${#} -gt 0 ]]; do
+			local full_site=${1}
+			local site=${2}
+			shift 2
+
+			mkdir -p "${BUILD_DIR}/install${site%/*}" || die
+			ln -s "${full_site}" "${BUILD_DIR}/install${site}" || die
+		done
+
+		# construct relative path from new sitedir to previous sitedir
+		local new_site=$(python_get_sitedir)
+		local relative_path=
+		while [[ ${site%%/*} == ${new_site%%/*} ]]; do
+			site=${site#*/}
+			new_site=${new_site#*/}
+		done
+		while [[ ${new_site} == */* ]]; do
+			relative_path+=../
+			new_site=${new_site%/*}
+		done
+		relative_path+=../${site}
+
+		cmd+=( --symlink-to "${relative_path}" )
+	fi
 	printf '%s\n' "${cmd[*]}"
 	"${cmd[@]}" || die "Wheel install failed"
 
@@ -1475,6 +1514,15 @@ distutils_pep517_install() {
 # distutils_pep517_install calls in the ebuild.
 declare -g -A DISTUTILS_WHEELS=()
 
+# @VARIABLE: _DISTUTILS_SITES
+# @INTERNAL
+# @DESCRIPTION:
+# A list of site-packages directories for Python implementations
+# previously built.  Each directory is represented by a pair of values:
+# absolute path to it in the build directory, and the final path
+# returned by python_get_sitedir.
+declare -g _DISTUTILS_SITES=()
+
 # @FUNCTION: distutils-r1_python_compile
 # @USAGE: [additional-args...]
 # @DESCRIPTION:
@@ -1705,6 +1753,20 @@ distutils-r1_python_install() {
 				mv "${wrapped_scriptdir}" "${reg_scriptdir}" || die
 			fi
 		fi
+
+		# remove site-packages symlinks if we created them
+		if [[ ${DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS} ]]; then
+			set -- "${_DISTUTILS_SITES[@]}"
+			while [[ ${#} -gt 0 ]]; do
+				local site=${2}
+				shift 2
+
+				if [[ -L "${BUILD_DIR}/install${site}" ]]; then
+					rm "${BUILD_DIR}/install${site}" || die
+				fi
+			done
+		fi
+
 		# prune empty directories to see if ${root} contains anything
 		# to merge
 		find "${BUILD_DIR}"/install -type d -empty -delete || die
@@ -2054,6 +2116,10 @@ _distutils-r1_post_python_compile() {
 
 		_distutils-r1_compare_installed_files
 	fi
+
+	local site=$(python_get_sitedir)
+	local full_site="${BUILD_DIR}/install${site}"
+	_DISTUTILS_SITES+=( "${full_site}" "${site}" )
 }
 
 distutils-r1_src_compile() {