[gentoo-dev] [PATCH] metadata/install-qa-check.d: fix python checks for non-distutils software

[gentoo-dev] [PATCH] metadata/install-qa-check.d: fix python checks for non-distutils software

[Sam James]

From: Eli Schwartz <eschwartz@gentoo.org>

The existing check makes an intimidating value proposition: that all
software being checked was installed using distutils-r1.eclass, hence
moving the check from there as-is to a new home is sufficient. This
includes the use of functions specific to the distutils-r1 eclass
inheritance chain. In particular, get_modname is part of
multilib.eclass, which distutils-r1 inherits, but python-single-r1 does
not inherit.

This results in the following QA warning:

```
/var/db/repos/gentoo/metadata/install-qa-check.d/60python-site: line 53: get_modname: command not found
/var/db/repos/gentoo/metadata/install-qa-check.d/60python-site: line 53: get_modname: command not found
/var/db/repos/gentoo/metadata/install-qa-check.d/60python-site: line 53: get_modname: command not found
 * Verifying compiled files for python3.12
 *
 * QA Notice: Extensions found compiled for the wrong Python version
 * (likely broken build isolation):
 *
 *   /usr/lib/python3.12/site-packages/__pycache__/init_calibre.cpython-312.pyc
 *   /usr/lib/python3.12/site-packages/__pycache__/init_calibre.cpython-312.opt-1.pyc
 *   /usr/lib/python3.12/site-packages/__pycache__/init_calibre.cpython-312.opt-2.pyc
```

because we are matching all files matching "*.cpython*" that are also
named "*" instead of all files named "*$(get_modname)".

Instead of using multilib.eclass, go directly to the preferred canonical
source, and query cpython what *it* thinks a module extension should be.
This is also more flexible since cpython itself doesn't really guarantee
that extension modules are named anything like get_modname, but
generally equals -- for backwards compatibility -- the final value from
`_PyImport_DynLoadFiletab` / `_imp.extension_suffixes()` (and on
Windows, that is .pyd instead of .dll, though admittedly,
sysconfig.get_config_vars is pretty empty there; on the other hand, PyPy
doesn't recognize unadorned .so because it doesn't need the
compatibility, so we see that it's not really a guarantee, and might as
well go for the sysconfig variable which is unambiguous where present).

Signed-off-by: Eli Schwartz <eschwartz@gentoo.org>
Signed-off-by: Sam James <sam@gentoo.org>
---
This fixes the case where we install some .so without distutils-r1
because of multilib.eclass not being inherited. It's also a bit
faster hopefully (cost of Python startup once vs repeated subshells).

I'll commit this later today if no objections.

 metadata/install-qa-check.d/60python-site | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/metadata/install-qa-check.d/60python-site b/metadata/install-qa-check.d/60python-site
index 3791fa80ecddb..49d4b3a1a8303 100644
--- a/metadata/install-qa-check.d/60python-site
+++ b/metadata/install-qa-check.d/60python-site
@@ -75,6 +75,13 @@ python_site_check() {
 		local sitedir=( "${pydir}"/site-packages )
 		[[ -d ${sitedir} ]] || continue
 
+		local modname=$(
+			"${impl}" - <<-EOF
+				import sysconfig
+				print(sysconfig.get_config_var("SHLIB_SUFFIX"))
+			EOF
+		)
+
 		# check for bad package versions
 		while IFS= read -d $'\0' -r f; do
 			bad_versions+=( "${f#${ED}}" )
@@ -107,7 +114,7 @@ python_site_check() {
 					-name '*.pth' -o \
 					-name '*.py' -o \
 					-name '*.pyi' -o \
-					-name "*$(get_modname)" -o \
+					-name "*${modname}" -o \
 					-name 'README.txt' \
 				')' -print0
 		)
@@ -130,8 +137,8 @@ python_site_check() {
 			wrong_ext+=( "${f#${ED}}" )
 		done < <(
 			find "${sitedir}" '(' \
-				-name "*.pypy*$(get_modname)" -o \
-				-name "*.cpython*$(get_modname)" \
+				-name "*.pypy*${modname}" -o \
+				-name "*.cpython*${modname}" \
 				')' -a '!' -name "*${ext_suffix}" -print0
 		)
 

base-commit: 8a7efac6a20aafc403fcf36c88156ac6b388fb9b
-- 
2.49.0

Re: [gentoo-dev] [PATCH] metadata/install-qa-check.d: fix python checks for non-distutils software

[Sam James]

Sam James <sam@gentoo.org> writes:

> From: Eli Schwartz <eschwartz@gentoo.org>
>
> The existing check makes an intimidating value proposition: that all
> software being checked was installed using distutils-r1.eclass, hence
> moving the check from there as-is to a new home is sufficient. This
> includes the use of functions specific to the distutils-r1 eclass
> inheritance chain. In particular, get_modname is part of
> multilib.eclass, which distutils-r1 inherits, but python-single-r1 does
> not inherit.
>
> This results in the following QA warning:
>
> ```
> /var/db/repos/gentoo/metadata/install-qa-check.d/60python-site: line 53: get_modname: command not found
> /var/db/repos/gentoo/metadata/install-qa-check.d/60python-site: line 53: get_modname: command not found
> /var/db/repos/gentoo/metadata/install-qa-check.d/60python-site: line 53: get_modname: command not found
>  * Verifying compiled files for python3.12
>  *
>  * QA Notice: Extensions found compiled for the wrong Python version
>  * (likely broken build isolation):
>  *
>  *   /usr/lib/python3.12/site-packages/__pycache__/init_calibre.cpython-312.pyc
>  *   /usr/lib/python3.12/site-packages/__pycache__/init_calibre.cpython-312.opt-1.pyc
>  *   /usr/lib/python3.12/site-packages/__pycache__/init_calibre.cpython-312.opt-2.pyc
> ```
>
> because we are matching all files matching "*.cpython*" that are also
> named "*" instead of all files named "*$(get_modname)".
>
> Instead of using multilib.eclass, go directly to the preferred canonical
> source, and query cpython what *it* thinks a module extension should be.
> This is also more flexible since cpython itself doesn't really guarantee
> that extension modules are named anything like get_modname, but
> generally equals -- for backwards compatibility -- the final value from
> `_PyImport_DynLoadFiletab` / `_imp.extension_suffixes()` (and on
> Windows, that is .pyd instead of .dll, though admittedly,
> sysconfig.get_config_vars is pretty empty there; on the other hand, PyPy
> doesn't recognize unadorned .so because it doesn't need the
> compatibility, so we see that it's not really a guarantee, and might as
> well go for the sysconfig variable which is unambiguous where present).
>
> Signed-off-by: Eli Schwartz <eschwartz@gentoo.org>
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
> This fixes the case where we install some .so without distutils-r1
> because of multilib.eclass not being inherited. It's also a bit
> faster hopefully (cost of Python startup once vs repeated subshells).
>
> I'll commit this later today if no objections.

I didn't mean to push this yet but forgot I had it staged, so it's in
now. I don't see much point in reverting it though.

>
>  metadata/install-qa-check.d/60python-site | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/metadata/install-qa-check.d/60python-site b/metadata/install-qa-check.d/60python-site
> index 3791fa80ecddb..49d4b3a1a8303 100644
> --- a/metadata/install-qa-check.d/60python-site
> +++ b/metadata/install-qa-check.d/60python-site
> @@ -75,6 +75,13 @@ python_site_check() {
>  		local sitedir=( "${pydir}"/site-packages )
>  		[[ -d ${sitedir} ]] || continue
>  
> +		local modname=$(
> +			"${impl}" - <<-EOF
> +				import sysconfig
> +				print(sysconfig.get_config_var("SHLIB_SUFFIX"))
> +			EOF
> +		)
> +
>  		# check for bad package versions
>  		while IFS= read -d $'\0' -r f; do
>  			bad_versions+=( "${f#${ED}}" )
> @@ -107,7 +114,7 @@ python_site_check() {
>  					-name '*.pth' -o \
>  					-name '*.py' -o \
>  					-name '*.pyi' -o \
> -					-name "*$(get_modname)" -o \
> +					-name "*${modname}" -o \
>  					-name 'README.txt' \
>  				')' -print0
>  		)
> @@ -130,8 +137,8 @@ python_site_check() {
>  			wrong_ext+=( "${f#${ED}}" )
>  		done < <(
>  			find "${sitedir}" '(' \
> -				-name "*.pypy*$(get_modname)" -o \
> -				-name "*.cpython*$(get_modname)" \
> +				-name "*.pypy*${modname}" -o \
> +				-name "*.cpython*${modname}" \
>  				')' -a '!' -name "*${ext_suffix}" -print0
>  		)
>  
>
> base-commit: 8a7efac6a20aafc403fcf36c88156ac6b388fb9b