[gentoo-dev] [PATCH 0/6] Another batch of distutils-r1.eclass fixes

[gentoo-dev] [PATCH 0/6] Another batch of distutils-r1.eclass fixes

[Michał Górny]

Hi,

This series includes some long overdue fixes + some recent PEP517 fixes.
In order:

1. Disabling post-test egg-info cleaning code in PEP517 -- this was
   needed when distutils_install_for_testing was used, and we no longer
   use it in PEP517 mode.

2. Fix a regression that `esetup.py` would successfully do nothing
   when run in a directory where there's no setuptools config.  Now it
   requires the current directory to contain either `setup.py`
   or `setup.cfg`.

3. Make distutils sub-phases pass the return value through.  This
   primarily fixes a somewhat rare bug involving:

       src_test() {
           virtx distutils-r1_src_test
       }

       python_test() {
           epytest
       }

   This code would not die on failure because virtx runs its arguments
   nonfatal, and epytest respects nonfatal.  Ideally, you'd use explicit
   `|| die` in python_test() but if you don't, the unsuccessful return
   code from epytest will eventually reach virtx and cause it to fail.

4. Refactoring.

5. Add DISTUTILS_DEPS containing BDEPEND in PEP517 mode.  This is needed
   to populate BDEPEND in DISTUTILS_OPTIONAL ebuilds.  For non-PEP517
   mode, we assumed that the dev will just inline the setuptools dep
   but with PEP517 we have more deps, and the tools used by eclass
   (tomli, installer) are not guaranteed to be fixed forever.

6. Add a QA warning when you declare DISTUTILS_USE_SETUPTOOLS along
   with DISTUTILS_OPTIONAL (it doesn't work).  This warning was
   originally part of the mismatched DUS check and was removed along
   with it.  However, this one still makes sense, so let's do it
   in the eclass.

Michał Górny (6):
  distutils-r1.eclass: Disable stale egg-info cleaning in PEP517 mode
  distutils-r1.eclass: make esetup.py require setup.{py,cfg}
  distutils-r1.eclass: Fix subphase return value passthrough
  distutils-r1.eclass: Move DISTUTILS_OPTIONAL check into set_globals
  distutils-r1.eclass: Add DISTUTILS_DEPS output var for PEP 517 mode
  distutils-r1.eclass: Restore QA warning for DUS + DISTUTILS_OPTIONAL

 eclass/distutils-r1.eclass | 114 +++++++++++++++++++++++++++++--------
 1 file changed, 90 insertions(+), 24 deletions(-)

-- 
2.35.1

[gentoo-dev] [PATCH 1/6] distutils-r1.eclass: Disable stale egg-info cleaning in PEP517 mode

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 8d03d2d1771d..4ec252958408 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1428,6 +1428,10 @@ distutils-r1_src_compile() {
 # Those files ended up being unversioned, and caused issues:
 # https://bugs.gentoo.org/534058
 _distutils-r1_clean_egg_info() {
+	if [[ ${DISTUTILS_USE_PEP517} ]]; then
+		die "${FUNCNAME} is not implemented in PEP517 mode"
+	fi
+
 	rm -rf "${BUILD_DIR}"/lib/*.egg-info || die
 }
 
@@ -1436,7 +1440,9 @@ distutils-r1_src_test() {
 
 	if declare -f python_test >/dev/null; then
 		_distutils-r1_run_foreach_impl python_test
-		_distutils-r1_run_foreach_impl _distutils-r1_clean_egg_info
+		if [[ ! ${DISTUTILS_USE_PEP517} ]]; then
+			_distutils-r1_run_foreach_impl _distutils-r1_clean_egg_info
+		fi
 	fi
 
 	if declare -f python_test_all >/dev/null; then
-- 
2.35.1

[gentoo-dev] [PATCH 2/6] distutils-r1.eclass: make esetup.py require setup.{py,cfg}

[Michał Górny]

Make esetup.py fail if neither setup.py nor setup.cfg is available.
To support PEP 517 builds, the function has been modified to work
without setup.py.  However, this also caused esetup.py to implicitly
succeed in installing zero files on non-distutils/setuptools packages.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 4ec252958408..a81d95f4eb6e 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -544,6 +544,9 @@ esetup.py() {
 	if [[ ${DISTUTILS_USE_SETUPTOOLS} == pyproject.toml ]]; then
 		setup_py=( -m pyproject2setuppy )
 	elif [[ ! -f setup.py ]]; then
+		if [[ ! -f setup.cfg ]]; then
+			die "${FUNCNAME}: setup.py nor setup.cfg not found"
+		fi
 		setup_py=( -c "from setuptools import setup; setup()" )
 	fi
 
-- 
2.35.1

[gentoo-dev] [PATCH 3/6] distutils-r1.eclass: Fix subphase return value passthrough

[Michał Górny]

Fix distutils-r1 phase functions to correctly pass through the return
value from the subphases.  This fixes e.g. the mistake of virtx
not failing in the following case:

    src_test() {
        virtx distutils-r1_src_test
    }

    python_test() {
        epytest
    }

This is because virtx implicitly uses nonfatal and epytest uses
`die -n`.  However, since the return value was not passed through, virtx
never knew that anything has failed.

While this covers only trivial cases and this is better solved via dying
explicitly in the redefined python_test(), there's no harm in adding
this passthrough.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 48 ++++++++++++++++++++++++++------------
 1 file changed, 33 insertions(+), 15 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index a81d95f4eb6e..8942a6149c93 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -1319,8 +1319,10 @@ distutils-r1_run_phase() {
 	local -x LDSHARED="${CC} ${ldopts}" LDCXXSHARED="${CXX} ${ldopts}"
 
 	"${@}"
+	local ret=${?}
 
 	cd "${_DISTUTILS_INITIAL_CWD}" || die
+	return "${ret}"
 }
 
 # @FUNCTION: _distutils-r1_run_common_phase
@@ -1378,14 +1380,14 @@ _distutils-r1_run_foreach_impl() {
 
 distutils-r1_src_prepare() {
 	debug-print-function ${FUNCNAME} "${@}"
-
+	local ret=0
 	local _DISTUTILS_DEFAULT_CALLED
 
 	# common preparations
 	if declare -f python_prepare_all >/dev/null; then
-		python_prepare_all
+		python_prepare_all || ret=${?}
 	else
-		distutils-r1_python_prepare_all
+		distutils-r1_python_prepare_all || ret=${?}
 	fi
 
 	if [[ ! ${_DISTUTILS_DEFAULT_CALLED} ]]; then
@@ -1393,35 +1395,45 @@ distutils-r1_src_prepare() {
 	fi
 
 	if declare -f python_prepare >/dev/null; then
-		_distutils-r1_run_foreach_impl python_prepare
+		_distutils-r1_run_foreach_impl python_prepare || ret=${?}
 	fi
+
+	return ${ret}
 }
 
 distutils-r1_src_configure() {
+	debug-print-function ${FUNCNAME} "${@}"
+	local ret=0
+
 	python_export_utf8_locale
 	[[ ${EAPI} == 6 ]] && xdg_environment_reset # Bug 577704
 
 	if declare -f python_configure >/dev/null; then
-		_distutils-r1_run_foreach_impl python_configure
+		_distutils-r1_run_foreach_impl python_configure || ret=${?}
 	fi
 
 	if declare -f python_configure_all >/dev/null; then
-		_distutils-r1_run_common_phase python_configure_all
+		_distutils-r1_run_common_phase python_configure_all || ret=${?}
 	fi
+
+	return ${ret}
 }
 
 distutils-r1_src_compile() {
 	debug-print-function ${FUNCNAME} "${@}"
+	local ret=0
 
 	if declare -f python_compile >/dev/null; then
-		_distutils-r1_run_foreach_impl python_compile
+		_distutils-r1_run_foreach_impl python_compile || ret=${?}
 	else
-		_distutils-r1_run_foreach_impl distutils-r1_python_compile
+		_distutils-r1_run_foreach_impl distutils-r1_python_compile || ret=${?}
 	fi
 
 	if declare -f python_compile_all >/dev/null; then
-		_distutils-r1_run_common_phase python_compile_all
+		_distutils-r1_run_common_phase python_compile_all || ret=${?}
 	fi
+
+	return ${ret}
 }
 
 # @FUNCTION: _distutils-r1_clean_egg_info
@@ -1440,17 +1452,20 @@ _distutils-r1_clean_egg_info() {
 
 distutils-r1_src_test() {
 	debug-print-function ${FUNCNAME} "${@}"
+	local ret=0
 
 	if declare -f python_test >/dev/null; then
-		_distutils-r1_run_foreach_impl python_test
+		_distutils-r1_run_foreach_impl python_test || ret=${?}
 		if [[ ! ${DISTUTILS_USE_PEP517} ]]; then
 			_distutils-r1_run_foreach_impl _distutils-r1_clean_egg_info
 		fi
 	fi
 
 	if declare -f python_test_all >/dev/null; then
-		_distutils-r1_run_common_phase python_test_all
+		_distutils-r1_run_common_phase python_test_all || ret=${?}
 	fi
+
+	return ${ret}
 }
 
 # @FUNCTION: _distutils-r1_check_namespace_pth
@@ -1482,20 +1497,23 @@ _distutils-r1_check_namespace_pth() {
 
 distutils-r1_src_install() {
 	debug-print-function ${FUNCNAME} "${@}"
+	local ret=0
 
 	if declare -f python_install >/dev/null; then
-		_distutils-r1_run_foreach_impl python_install
+		_distutils-r1_run_foreach_impl python_install || ret=${?}
 	else
-		_distutils-r1_run_foreach_impl distutils-r1_python_install
+		_distutils-r1_run_foreach_impl distutils-r1_python_install || ret=${?}
 	fi
 
 	if declare -f python_install_all >/dev/null; then
-		_distutils-r1_run_common_phase python_install_all
+		_distutils-r1_run_common_phase python_install_all || ret=${?}
 	else
-		_distutils-r1_run_common_phase distutils-r1_python_install_all
+		_distutils-r1_run_common_phase distutils-r1_python_install_all || ret=${?}
 	fi
 
 	_distutils-r1_check_namespace_pth
+
+	return ${ret}
 }
 
 _DISTUTILS_R1=1
-- 
2.35.1

[gentoo-dev] [PATCH 4/6] distutils-r1.eclass: Move DISTUTILS_OPTIONAL check into set_globals

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 8942a6149c93..4a9fdb4018b4 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -213,15 +213,17 @@ _distutils_set_globals() {
 		[[ -n ${rdep} ]] && rdep="$(python_gen_cond_dep "${rdep}")"
 	fi
 
-	RDEPEND="${PYTHON_DEPS} ${rdep}"
-	if [[ ${EAPI} != 6 ]]; then
-		BDEPEND="${PYTHON_DEPS} ${bdep}"
-	else
-		DEPEND="${PYTHON_DEPS} ${bdep}"
+	if [[ ! ${DISTUTILS_OPTIONAL} ]]; then
+		RDEPEND="${PYTHON_DEPS} ${rdep}"
+		if [[ ${EAPI} != 6 ]]; then
+			BDEPEND="${PYTHON_DEPS} ${bdep}"
+		else
+			DEPEND="${PYTHON_DEPS} ${bdep}"
+		fi
+		REQUIRED_USE=${PYTHON_REQUIRED_USE}
 	fi
-	REQUIRED_USE=${PYTHON_REQUIRED_USE}
 }
-[[ ! ${DISTUTILS_OPTIONAL} ]] && _distutils_set_globals
+_distutils_set_globals
 unset -f _distutils_set_globals
 
 # @ECLASS-VARIABLE: PATCHES
-- 
2.35.1

[gentoo-dev] [PATCH 5/6] distutils-r1.eclass: Add DISTUTILS_DEPS output var for PEP 517 mode

[Michał Górny]

The PEP 517 build-time deps have gotten more complex, and largely depend
on the internal eclass logic used to build and install wheels.
Introduce a DISTUTILS_DEPS output variable that contains the correct
BDEPEND string for use in DISTUTILS_OPTIONAL=1 ebuilds.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 4a9fdb4018b4..a0ad598eb58f 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -128,6 +128,24 @@ esac
 # It is available only in non-PEP517 mode.  It needs to be set before
 # the inherit line.
 
+# @ECLASS-VARIABLE: DISTUTILS_DEPS
+# @OUTPUT_VARIABLE
+# @DESCRIPTION:
+# This is an eclass-generated build-time dependency string for the build
+# system packages.  This string is automatically appended to BDEPEND
+# unless DISTUTILS_OPTIONAL is used.  This variable is available only
+# in PEP 517 mode.
+#
+# Example use:
+# @CODE
+# DISTUTILS_OPTIONAL=1
+# # ...
+# RDEPEND="${PYTHON_DEPS}"
+# BDEPEND="
+#     ${PYTHON_DEPS}
+#     ${DISTUTILS_DEPS}"
+# @CODE
+
 if [[ ! ${_DISTUTILS_R1} ]]; then
 
 [[ ${EAPI} == 6 ]] && inherit eutils xdg-utils
@@ -156,7 +174,7 @@ _distutils_set_globals() {
 
 		# installer is used to install the wheel
 		# tomli is used to read build-backend from pyproject.toml
-		bdep+='
+		bdep='
 			>=dev-python/installer-0.4.0_p20220124[${PYTHON_USEDEP}]
 			dev-python/tomli[${PYTHON_USEDEP}]'
 		case ${DISTUTILS_USE_PEP517} in
@@ -213,6 +231,20 @@ _distutils_set_globals() {
 		[[ -n ${rdep} ]] && rdep="$(python_gen_cond_dep "${rdep}")"
 	fi
 
+	if [[ ${DISTUTILS_USE_PEP517} ]]; then
+		if [[ ${DISTUTILS_DEPS+1} ]]; then
+			if [[ ${DISTUTILS_DEPS} != "${bdep}" ]]; then
+				eerror "DISTUTILS_DEPS have changed between inherits!"
+				eerror "Before: ${DISTUTILS_DEPS}"
+				eerror "Now   : ${bdep}"
+				die "DISTUTILS_DEPS integrity check failed"
+			fi
+		else
+			DISTUTILS_DEPS=${bdep}
+			readonly DISTUTILS_DEPS
+		fi
+	fi
+
 	if [[ ! ${DISTUTILS_OPTIONAL} ]]; then
 		RDEPEND="${PYTHON_DEPS} ${rdep}"
 		if [[ ${EAPI} != 6 ]]; then
-- 
2.35.1

[gentoo-dev] [PATCH 6/6] distutils-r1.eclass: Restore QA warning for DUS + DISTUTILS_OPTIONAL

[Michał Górny]

Restore the QA warning (proviously issued as part of install-qa-check.d)
for combining DISTUTILS_USE_SETUPTOOLS and DISTUTILS_OPTIONAL.

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/distutils-r1.eclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index a0ad598eb58f..2b5acb09d926 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -201,6 +201,11 @@ _distutils_set_globals() {
 				die "Unknown DISTUTILS_USE_PEP517=${DISTUTILS_USE_PEP517}"
 				;;
 		esac
+	elif [[ ${DISTUTILS_OPTIONAL} ]]; then
+		if [[ ${DISTUTILS_USE_SETUPTOOLS} ]]; then
+			eqawarn "QA Notice: DISTUTILS_USE_SETUPTOOLS is not used when DISTUTILS_OPTIONAL"
+			eqawarn "is enabled."
+		fi
 	else
 		local setuptools_dep='>=dev-python/setuptools-42.0.2[${PYTHON_USEDEP}]'
 
-- 
2.35.1

Re: [gentoo-dev] [PATCH 5/6] distutils-r1.eclass: Add DISTUTILS_DEPS output var for PEP 517 mode

[Sam James]

[-- Attachment #1.1: Type: text/plain, Size: 562 bytes --]



> On 31 Jan 2022, at 22:59, Michał Górny <mgorny@gentoo.org> wrote:
> 
> The PEP 517 build-time deps have gotten more complex, and largely depend
> on the internal eclass logic used to build and install wheels.
> Introduce a DISTUTILS_DEPS output variable that contains the correct
> BDEPEND string for use in DISTUTILS_OPTIONAL=1 ebuilds.
> 
> Signed-off-by: Michał Górny <mgorny@gentoo.org>
> ---

All LGTM (checked when on GH), but add bug ref to this commit:
https://bugs.gentoo.org/832337 <https://bugs.gentoo.org/832337>.

Best,
sam

[-- Attachment #1.2: Type: text/html, Size: 1182 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 5/6] distutils-r1.eclass: Add DISTUTILS_DEPS output var for PEP 517 mode

[Michał Górny]

On Mon, 2022-01-31 at 23:06 +0000, Sam James wrote:
> 
> > On 31 Jan 2022, at 22:59, Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > The PEP 517 build-time deps have gotten more complex, and largely depend
> > on the internal eclass logic used to build and install wheels.
> > Introduce a DISTUTILS_DEPS output variable that contains the correct
> > BDEPEND string for use in DISTUTILS_OPTIONAL=1 ebuilds.
> > 
> > Signed-off-by: Michał Górny <mgorny@gentoo.org>
> > ---
> 
> All LGTM (checked when on GH), but add bug ref to this commit:
> https://bugs.gentoo.org/832337 <https://bugs.gentoo.org/832337>.
> 

Done.

-- 
Best regards,
Michał Górny