[gentoo-dev] [PATCH 0/5] distutils-r1.eclass: DISTUTILS_EXT to enable features specific to building extensions

[gentoo-dev] [PATCH 0/5] distutils-r1.eclass: DISTUTILS_EXT to enable features specific to building extensions

[Michał Górny]

Hi,

Here's a patch series that adds a DISTUTILS_EXT variable that indicates
that the package in question builds Python extensions (i.e. loadable
.so modules) and enables a few related features:

1. adds DEPEND=${PYTHON_DEPS} to fix cross-compiling

2. adds IUSE=debug that controls passing -DNDEBUG/-UNDEBUG to builds
   (right now unlike most distros out there we are always building
   with assertions enabled)

3. limits our parallel build_ext optimization to DISTUTILS_EXT ebuilds
   (i.e. eliminates the few false positives triggering it right now)

The patches also include logic to automatically detect and report
(as a QA warning) missing DISTUTILS_EXT if a package is installing .so
files into site-packages.

Note that technically none of the above applies to ebuilds building Rust
extensions rather than C/Cython.  However, there's no trivial way to
distinguish one from the other for the purpose of QA check and none
of the above points do any real harm.  Especially that cargo.eclass
adds IUSE=debug anyway.

DISTUTILS_EXT should also be set if the extensions are built
conditionally to USE flags.  The only ill effect of doing that is that
PYTHON_DEPS are added unconditionally (unless you're using
DISTUTILS_OPTIONAL) rather than conditionally to the flag responsible
for the extension but that's a minor point anyway.

-- 
Best regards,
Michał Górny

Michał Górny (5):
  distutils-r1.eclass: Add DISTUTILS_EXT control variable
  distutils-r1.eclass: Add PYTHON_DEPS to DEPEND when DISTUTILS_EXT
  distutils-r1.eclass: Support IUSE=debug for DISTUTILS_EXT
  distutils-r1.eclass: Issue build_ext only w/ DISTUTILS_EXT
  distutils-r1.eclass: Detect missing DISTUTILS_EXT

 eclass/distutils-r1.eclass | 50 +++++++++++++++++++++++++++++++++-----
 1 file changed, 44 insertions(+), 6 deletions(-)

-- 
2.40.0

[gentoo-dev] [PATCH 1/5] distutils-r1.eclass: Add DISTUTILS_EXT control variable

[Michał Górny]

Add a control variable that should be set in ebuilds that build
C, Cython, Rust extensions, etc.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 09684781de2a..f283aa90cfaa 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -49,6 +49,13 @@ case ${EAPI} in
 	*) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
 esac
 
+# @ECLASS_VARIABLE: DISTUTILS_EXT
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# Set this variable to a non-null value if the package (possibly
+# optionally) builds Python extensions (loadable modules written in C,
+# Cython, Rust, etc.).
+
 # @ECLASS_VARIABLE: DISTUTILS_OPTIONAL
 # @DEFAULT_UNSET
 # @DESCRIPTION:
-- 
2.40.0

[gentoo-dev] [PATCH 2/5] distutils-r1.eclass: Add PYTHON_DEPS to DEPEND when DISTUTILS_EXT

[Michał Górny]

Add `${PYTHON_DEPS}` to `DEPEND` (i.e. `SYSROOT` dependencies) when
building extensions.  This is necessary to support cross-compilations
correctly.

Originally proposed by Raul E Rangel <rrangel@chromium.org>.

Closes: https://github.com/gentoo/gentoo/pull/30469
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index f283aa90cfaa..f56fa5a51efa 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -55,6 +55,11 @@ esac
 # Set this variable to a non-null value if the package (possibly
 # optionally) builds Python extensions (loadable modules written in C,
 # Cython, Rust, etc.).
+#
+# When enabled, the eclass:
+#
+# - adds PYTHON_DEPS to DEPEND (for cross-compilation support), unless
+#   DISTUTILS_OPTIONAL is used
 
 # @ECLASS_VARIABLE: DISTUTILS_OPTIONAL
 # @DEFAULT_UNSET
@@ -317,6 +322,10 @@ _distutils_set_globals() {
 		RDEPEND="${PYTHON_DEPS} ${rdep}"
 		BDEPEND="${PYTHON_DEPS} ${bdep}"
 		REQUIRED_USE=${PYTHON_REQUIRED_USE}
+
+		if [[ ${DISTUTILS_EXT} ]]; then
+			DEPEND="${PYTHON_DEPS}"
+		fi
 	fi
 }
 _distutils_set_globals
-- 
2.40.0

[gentoo-dev] [PATCH 3/5] distutils-r1.eclass: Support IUSE=debug for DISTUTILS_EXT

[Michał Górny]

Add IUSE=debug when DISTUTILS_EXT is set, and use it to control
assertions via appending -DNDEBUG or -UNDEBUG to CPPFLAGS.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index f56fa5a51efa..257e032a6c83 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -60,6 +60,8 @@ esac
 #
 # - adds PYTHON_DEPS to DEPEND (for cross-compilation support), unless
 #   DISTUTILS_OPTIONAL is used
+#
+# - adds debug flag to IUSE that controls assertions (i.e. -DNDEBUG)
 
 # @ECLASS_VARIABLE: DISTUTILS_OPTIONAL
 # @DEFAULT_UNSET
@@ -325,6 +327,7 @@ _distutils_set_globals() {
 
 		if [[ ${DISTUTILS_EXT} ]]; then
 			DEPEND="${PYTHON_DEPS}"
+			IUSE="debug"
 		fi
 	fi
 }
@@ -1766,6 +1769,10 @@ distutils-r1_run_phase() {
 	local -x AR=${AR} CC=${CC} CPP=${CPP} CXX=${CXX}
 	tc-export AR CC CPP CXX
 
+	if [[ ${DISTUTILS_EXT} ]]; then
+		local -x CPPFLAGS="${CPPFLAGS} $(usex debug '-UNDEBUG' '-DNDEBUG')"
+	fi
+
 	# How to build Python modules in different worlds...
 	local ldopts
 	case "${CHOST}" in
-- 
2.40.0

[gentoo-dev] [PATCH 4/5] distutils-r1.eclass: Issue build_ext only w/ DISTUTILS_EXT

[Michał Górny]

Issue the explicit `build_ext` command call only if `DISTUTILS_EXT`
is set.  This should avoid a few false positives where the command
was called due to irrelevant .c files being present in the source tree.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 257e032a6c83..f7510a4640cb 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -62,6 +62,9 @@ esac
 #   DISTUTILS_OPTIONAL is used
 #
 # - adds debug flag to IUSE that controls assertions (i.e. -DNDEBUG)
+#
+# - calls `build_ext` command if setuptools build backend is used
+#   and there is potential benefit from parallel builds
 
 # @ECLASS_VARIABLE: DISTUTILS_OPTIONAL
 # @DEFAULT_UNSET
@@ -1452,12 +1455,14 @@ distutils-r1_python_compile() {
 				# .pyx is added for Cython
 				#
 				# esetup.py does not respect SYSROOT, so skip it there
-				if [[ -z ${SYSROOT} && 1 -ne ${jobs} && 2 -eq $(
-					find '(' -name '*.c' -o -name '*.cc' -o -name '*.cpp' \
-						-o -name '*.cxx' -o -name '*.c++' -o -name '*.m' \
-						-o -name '*.mm' -o -name '*.pyx' ')' -printf '\n' |
-						head -n 2 | wc -l
-				) ]]; then
+				if [[ -z ${SYSROOT} && ${DISTUTILS_EXT} && 1 -ne ${jobs}
+					&& 2 -eq $(
+						find '(' -name '*.c' -o -name '*.cc' -o -name '*.cpp' \
+							-o -name '*.cxx' -o -name '*.c++' -o -name '*.m' \
+							-o -name '*.mm' -o -name '*.pyx' ')' -printf '\n' |
+							head -n 2 | wc -l
+					)
+				]]; then
 					esetup.py build_ext -j "${jobs}" "${@}"
 				fi
 			else
-- 
2.40.0

[gentoo-dev] [PATCH 5/5] distutils-r1.eclass: Detect missing DISTUTILS_EXT

[Michał Górny]

Detect when Python packages install extensions and warn about missing
DISTUTILS_EXT appropriately.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index f7510a4640cb..9b31e5db0d2f 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -2069,6 +2069,16 @@ _distutils-r1_post_python_install() {
 			eerror "https://projects.gentoo.org/python/guide/qawarn.html#stray-top-level-files-in-site-packages"
 			die "Failing install because of stray top-level files in site-packages"
 		fi
+
+		if [[ ! ${DISTUTILS_EXT} && ! ${_DISTUTILS_EXT_WARNED} ]]; then
+			if [[ $(find "${sitedir}" -name "*$(get_modname)" | head -n 1) ]]
+			then
+				eqawarn "Python extension modules (*$(get_modname)) found installed. Please set:"
+				eqawarn "  DISTUTILS_EXT=1"
+				eqawarn "in the ebuild."
+				_DISTUTILS_EXT_WARNED=1
+			fi
+		fi
 	fi
 }
 
-- 
2.40.0