[gentoo-dev] [PATCH 0/7] pypi.eclass: Filename and version normalization

[gentoo-dev] [PATCH 0/7] pypi.eclass: Filename and version normalization

[Michał Górny]

Hi,

Here's a major patchset to the recently committed pypi.eclass (i.e.
the "change while it's still warm and not used much" kind).

It focuses on two significant changes:

1. Normalizing filenames according to the current sdist/wheel standards.

2. Translating Gentoo-specific components in ${PV} to the corresponding
   standard Python versions.

The former change is focused on improving correctness (i.e. sometimes
the current API didn't allow for getting the correct URI), the latter
on convenience (i.e. not having to override ${PV} sometimes).

For example, a package called "Test-Package-1.2.3_beta4" that followed
PEP 625 (e.g. by using Hatchling) would require an ebuild calling
the equivalent of:

  $(pypi_sdist_url "test_package" "${PV/_beta/b}")

while now it will work out of the box.

Unfortunately, PEP 625 is not yet universally followed, so many packages
will instead need to:

  $(pypi_sdist_url --no-normalize)

but I think following the standard immediately and requiring a number
of `--no-normalize` calls is better than doing it the other way around
and then switching the defaults, even if the majority of affected
packages use "legacy" naming right now.

That said, this only affects packages with uppercase letters, "-" or "."
in their names.

More details in eclassdoc.

The patchset also adds tests for the eclass.

Relevant PR (including fixes to existing consumers):
https://github.com/gentoo/gentoo/pull/29526

Fortunately, breakage is easy to detect by remanifesting ebuilds.

-- 
Best regards,
Michał Górny


Michał Górny (7):
  pypi.eclass: Use pypi_sdist_url to generate the default SRC_URI
  pypi.eclass: Add a name normalization function
  pypi.eclass: Normalize wheel filenames
  pypi.eclass: Add `--unpack` to usage error in pypi_wheel_url
  pypi.eclass: Normalize sdist filenames by default
  pypi.eclass: Add a version translation function
  pypi.eclass: Translate ${PV} by default

 eclass/pypi.eclass   | 94 ++++++++++++++++++++++++++++++++++++--------
 eclass/tests/pypi.sh | 93 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 171 insertions(+), 16 deletions(-)
 create mode 100755 eclass/tests/pypi.sh

-- 
2.39.1

[gentoo-dev] [PATCH 1/7] pypi.eclass: Use pypi_sdist_url to generate the default SRC_URI

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index e9d3eec1268b..d00b1171fd16 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -34,10 +34,6 @@ esac
 if [[ ! ${_PYPI_ECLASS} ]]; then
 _PYPI_ECLASS=1
 
-SRC_URI="
-	https://files.pythonhosted.org/packages/source/${PN::1}/${PN}/${P}.tar.gz
-"
-
 # @FUNCTION: pypi_sdist_url
 # @USAGE: [<project> [<version> [<suffix>]]]
 # @DESCRIPTION:
@@ -132,4 +128,6 @@ pypi_wheel_url() {
 	fi
 }
 
+SRC_URI="$(pypi_sdist_url)"
+
 fi
-- 
2.39.1

[gentoo-dev] [PATCH 2/7] pypi.eclass: Add a name normalization function

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass   | 20 ++++++++++++++++++++
 eclass/tests/pypi.sh | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+)
 create mode 100755 eclass/tests/pypi.sh

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index d00b1171fd16..3a37214f8977 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -34,6 +34,26 @@ esac
 if [[ ! ${_PYPI_ECLASS} ]]; then
 _PYPI_ECLASS=1
 
+# @FUNCTION: pypi_normalize_name
+# @USAGE: <name>
+# @DESCRIPTION:
+# Normalize the project name according to sdist/wheel normalization
+# rules.  That is, convert to lowercase and replace runs of [._-]
+# with a single underscore.
+#
+# Based on the spec, as of 2023-02-10:
+# https://packaging.python.org/en/latest/specifications/#package-distribution-file-formats
+pypi_normalize_name() {
+	[[ ${#} -ne 1 ]] && die "Usage: ${FUNCNAME} <name>"
+
+	local name=${1}
+	local shopt_save=$(shopt -p extglob)
+	shopt -s extglob
+	name=${name//+([._-])/_}
+	${shopt_save}
+	echo "${name,,}"
+}
+
 # @FUNCTION: pypi_sdist_url
 # @USAGE: [<project> [<version> [<suffix>]]]
 # @DESCRIPTION:
diff --git a/eclass/tests/pypi.sh b/eclass/tests/pypi.sh
new file mode 100755
index 000000000000..67b2c3c481fb
--- /dev/null
+++ b/eclass/tests/pypi.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+# Copyright 2023 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+EAPI=8
+source tests-common.sh || exit
+
+inherit pypi
+
+test-eq() {
+	local call=${1}
+	local exp=${2}
+
+	tbegin "${call} -> ${exp}"
+	local ret=0
+	local have=$(${call})
+	if [[ ${have} != ${exp} ]]; then
+		eindent
+		eerror "incorrect result: ${have}"
+		eoutdent
+		ret=1
+	fi
+	tend "${ret}"
+}
+
+test-eq "pypi_normalize_name foo" foo
+test-eq "pypi_normalize_name foo_bar" foo_bar
+test-eq "pypi_normalize_name foo___bar" foo_bar
+test-eq "pypi_normalize_name Flask-BabelEx" flask_babelex
+test-eq "pypi_normalize_name jaraco.context" jaraco_context
+
+texit
-- 
2.39.1

[gentoo-dev] [PATCH 3/7] pypi.eclass: Normalize wheel filenames

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass   |  5 +++--
 eclass/tests/pypi.sh | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index 3a37214f8977..c7b9604807eb 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -82,7 +82,8 @@ pypi_sdist_url() {
 # @DESCRIPTION:
 # Output the wheel filename for the specified project/version tuple.
 #
-# If <package> is unspecified, it defaults to ${PN}.
+# If <package> is unspecified, it defaults to ${PN}.  The package name
+# is normalized according to the wheel specification.
 #
 # If <version> is unspecified, it defaults to ${PV}.
 #
@@ -97,7 +98,7 @@ pypi_wheel_name() {
 		die "Usage: ${FUNCNAME} <project> [<version> [<python-tag> [<abi-platform-tag>]]]"
 	fi
 
-	local project=${1-"${PN}"}
+	local project=$(pypi_normalize_name "${1-"${PN}"}")
 	local version=${2-"${PV}"}
 	local pytag=${3-py3}
 	local abitag=${4-none-any}
diff --git a/eclass/tests/pypi.sh b/eclass/tests/pypi.sh
index 67b2c3c481fb..111b61380fe4 100755
--- a/eclass/tests/pypi.sh
+++ b/eclass/tests/pypi.sh
@@ -29,4 +29,37 @@ test-eq "pypi_normalize_name foo___bar" foo_bar
 test-eq "pypi_normalize_name Flask-BabelEx" flask_babelex
 test-eq "pypi_normalize_name jaraco.context" jaraco_context
 
+PN=Foo.Bar
+PV=1.2.3
+
+test-eq "pypi_wheel_name" foo_bar-1.2.3-py3-none-any.whl
+test-eq "pypi_wheel_name Flask-BabelEx" flask_babelex-1.2.3-py3-none-any.whl
+test-eq "pypi_wheel_name Flask-BabelEx 4" flask_babelex-4-py3-none-any.whl
+test-eq "pypi_wheel_name Flask-BabelEx 4 py2.py3" \
+	flask_babelex-4-py2.py3-none-any.whl
+test-eq "pypi_wheel_name cryptography 39.0.1 cp36 abi3-manylinux_2_28_x86_64" \
+	cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl
+
+test-eq "pypi_wheel_url" \
+	https://files.pythonhosted.org/packages/py3/F/Foo.Bar/foo_bar-1.2.3-py3-none-any.whl
+test-eq "pypi_wheel_url Flask-BabelEx" \
+	https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-1.2.3-py3-none-any.whl
+test-eq "pypi_wheel_url Flask-BabelEx 4" \
+	https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-4-py3-none-any.whl
+test-eq "pypi_wheel_url Flask-BabelEx 4 py2.py3" \
+	https://files.pythonhosted.org/packages/py2.py3/F/Flask-BabelEx/flask_babelex-4-py2.py3-none-any.whl
+test-eq "pypi_wheel_url cryptography 39.0.1 cp36 abi3-manylinux_2_28_x86_64" \
+	https://files.pythonhosted.org/packages/cp36/c/cryptography/cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl
+
+test-eq "pypi_wheel_url --unpack" \
+	"https://files.pythonhosted.org/packages/py3/F/Foo.Bar/foo_bar-1.2.3-py3-none-any.whl -> foo_bar-1.2.3-py3-none-any.whl.zip"
+test-eq "pypi_wheel_url --unpack Flask-BabelEx" \
+	"https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-1.2.3-py3-none-any.whl -> flask_babelex-1.2.3-py3-none-any.whl.zip"
+test-eq "pypi_wheel_url --unpack Flask-BabelEx 4" \
+	"https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-4-py3-none-any.whl -> flask_babelex-4-py3-none-any.whl.zip"
+test-eq "pypi_wheel_url --unpack Flask-BabelEx 4 py2.py3" \
+	"https://files.pythonhosted.org/packages/py2.py3/F/Flask-BabelEx/flask_babelex-4-py2.py3-none-any.whl -> flask_babelex-4-py2.py3-none-any.whl.zip"
+test-eq "pypi_wheel_url --unpack cryptography 39.0.1 cp36 abi3-manylinux_2_28_x86_64" \
+	"https://files.pythonhosted.org/packages/cp36/c/cryptography/cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl -> cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl.zip"
+
 texit
-- 
2.39.1

[gentoo-dev] [PATCH 4/7] pypi.eclass: Add `--unpack` to usage error in pypi_wheel_url

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index c7b9604807eb..a3c38aa5f3ec 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -134,7 +134,7 @@ pypi_wheel_url() {
 	fi
 
 	if [[ ${#} -gt 4 ]]; then
-		die "Usage: ${FUNCNAME} <project> [<version> [<python-tag> [<abi-platform-tag>]]]"
+		die "Usage: ${FUNCNAME} [--unpack] <project> [<version> [<python-tag> [<abi-platform-tag>]]]"
 	fi
 
 	local filename=$(pypi_wheel_name "${@}")
-- 
2.39.1

[gentoo-dev] [PATCH 5/7] pypi.eclass: Normalize sdist filenames by default

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass   | 23 +++++++++++++++++++----
 eclass/tests/pypi.sh | 27 ++++++++++++++++++++++++---
 2 files changed, 43 insertions(+), 7 deletions(-)

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index a3c38aa5f3ec..dd24b8337e62 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -55,26 +55,41 @@ pypi_normalize_name() {
 }
 
 # @FUNCTION: pypi_sdist_url
-# @USAGE: [<project> [<version> [<suffix>]]]
+# @USAGE: [--no-normalize] [<project> [<version> [<suffix>]]]
 # @DESCRIPTION:
 # Output the URL to PyPI sdist for specified project/version tuple.
 #
-# If <package> is unspecified, it defaults to ${PN}.
+# The `--no-normalize` option disables project name normalization
+# for sdist filename.  This may be necessary when dealing with distfiles
+# generated using build systems that did not follow PEP 625
+# (i.e. the sdist name contains uppercase letters, hyphens or dots).
+#
+# If <package> is unspecified, it defaults to ${PN}.  The package name
+# is normalized according to the specification unless `--no-normalize`
+# is passed.
 #
 # If <version> is unspecified, it defaults to ${PV}.
 #
 # If <format> is unspecified, it defaults to ".tar.gz".  Another valid
 # value is ".zip" (please remember to add a BDEPEND on app-arch/unzip).
 pypi_sdist_url() {
+	local normalize=1
+	if [[ ${1} == --no-normalize ]]; then
+		normalize=
+		shift
+	fi
+
 	if [[ ${#} -gt 3 ]]; then
-		die "Usage: ${FUNCNAME} <project> [<version> [<suffix>]]"
+		die "Usage: ${FUNCNAME} [--no-normalize] <project> [<version> [<suffix>]]"
 	fi
 
 	local project=${1-"${PN}"}
 	local version=${2-"${PV}"}
 	local suffix=${3-.tar.gz}
+	local fn_project=${project}
+	[[ ${normalize} ]] && fn_project=$(pypi_normalize_name "${project}")
 	printf "https://files.pythonhosted.org/packages/source/%s" \
-		"${project::1}/${project}/${project}-${version}${suffix}"
+		"${project::1}/${project}/${fn_project}-${version}${suffix}"
 }
 
 # @FUNCTION: pypi_wheel_name
diff --git a/eclass/tests/pypi.sh b/eclass/tests/pypi.sh
index 111b61380fe4..385b1c028bce 100755
--- a/eclass/tests/pypi.sh
+++ b/eclass/tests/pypi.sh
@@ -5,6 +5,9 @@
 EAPI=8
 source tests-common.sh || exit
 
+PN=Foo.Bar
+PV=1.2.3
+
 inherit pypi
 
 test-eq() {
@@ -29,9 +32,6 @@ test-eq "pypi_normalize_name foo___bar" foo_bar
 test-eq "pypi_normalize_name Flask-BabelEx" flask_babelex
 test-eq "pypi_normalize_name jaraco.context" jaraco_context
 
-PN=Foo.Bar
-PV=1.2.3
-
 test-eq "pypi_wheel_name" foo_bar-1.2.3-py3-none-any.whl
 test-eq "pypi_wheel_name Flask-BabelEx" flask_babelex-1.2.3-py3-none-any.whl
 test-eq "pypi_wheel_name Flask-BabelEx 4" flask_babelex-4-py3-none-any.whl
@@ -62,4 +62,25 @@ test-eq "pypi_wheel_url --unpack Flask-BabelEx 4 py2.py3" \
 test-eq "pypi_wheel_url --unpack cryptography 39.0.1 cp36 abi3-manylinux_2_28_x86_64" \
 	"https://files.pythonhosted.org/packages/cp36/c/cryptography/cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl -> cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl.zip"
 
+test-eq "pypi_sdist_url" \
+	https://files.pythonhosted.org/packages/source/F/Foo.Bar/foo_bar-1.2.3.tar.gz
+test-eq "pypi_sdist_url Flask-BabelEx" \
+	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/flask_babelex-1.2.3.tar.gz
+test-eq "pypi_sdist_url Flask-BabelEx 4" \
+	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/flask_babelex-4.tar.gz
+test-eq "pypi_sdist_url Flask-BabelEx 4 .zip" \
+	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/flask_babelex-4.zip
+
+test-eq "pypi_sdist_url --no-normalize" \
+	https://files.pythonhosted.org/packages/source/F/Foo.Bar/Foo.Bar-1.2.3.tar.gz
+test-eq "pypi_sdist_url --no-normalize Flask-BabelEx" \
+	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/Flask-BabelEx-1.2.3.tar.gz
+test-eq "pypi_sdist_url --no-normalize Flask-BabelEx 4" \
+	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/Flask-BabelEx-4.tar.gz
+test-eq "pypi_sdist_url --no-normalize Flask-BabelEx 4 .zip" \
+	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/Flask-BabelEx-4.zip
+
+test-eq 'declare -p SRC_URI' \
+	'declare -- SRC_URI="https://files.pythonhosted.org/packages/source/F/Foo.Bar/foo_bar-1.2.3.tar.gz"'
+
 texit
-- 
2.39.1

[gentoo-dev] [PATCH 6/7] pypi.eclass: Add a version translation function

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass   | 19 +++++++++++++++++++
 eclass/tests/pypi.sh |  7 +++++++
 2 files changed, 26 insertions(+)

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index dd24b8337e62..27bd17403100 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -54,6 +54,25 @@ pypi_normalize_name() {
 	echo "${name,,}"
 }
 
+# @FUNCTION: pypi_translate_version
+# @USAGE: <version>
+# @DESCRIPTION:
+# Translate the specified Gentoo version into the usual Python
+# counterpart.  Assumes PEP 440 versions.
+#
+# Note that we do not have clear counterparts for the epoch segment,
+# nor for development release segment.
+pypi_translate_version() {
+	[[ ${#} -ne 1 ]] && die "Usage: ${FUNCNAME} <version>"
+
+	local version=${1}
+	version=${version/_alpha/a}
+	version=${version/_beta/b}
+	version=${version/_rc/rc}
+	version=${version/_p/.post}
+	echo "${version}"
+}
+
 # @FUNCTION: pypi_sdist_url
 # @USAGE: [--no-normalize] [<project> [<version> [<suffix>]]]
 # @DESCRIPTION:
diff --git a/eclass/tests/pypi.sh b/eclass/tests/pypi.sh
index 385b1c028bce..af5d714af748 100755
--- a/eclass/tests/pypi.sh
+++ b/eclass/tests/pypi.sh
@@ -32,6 +32,13 @@ test-eq "pypi_normalize_name foo___bar" foo_bar
 test-eq "pypi_normalize_name Flask-BabelEx" flask_babelex
 test-eq "pypi_normalize_name jaraco.context" jaraco_context
 
+test-eq "pypi_translate_version 1.2.3" 1.2.3
+test-eq "pypi_translate_version 1.2.3_p101" 1.2.3.post101
+test-eq "pypi_translate_version 1.2.3_alpha4" 1.2.3a4
+test-eq "pypi_translate_version 1.2.3_beta1" 1.2.3b1
+test-eq "pypi_translate_version 1.2.3_rc2" 1.2.3rc2
+test-eq "pypi_translate_version 1.2.3_rc2_p1" 1.2.3rc2.post1
+
 test-eq "pypi_wheel_name" foo_bar-1.2.3-py3-none-any.whl
 test-eq "pypi_wheel_name Flask-BabelEx" flask_babelex-1.2.3-py3-none-any.whl
 test-eq "pypi_wheel_name Flask-BabelEx 4" flask_babelex-4-py3-none-any.whl
-- 
2.39.1

[gentoo-dev] [PATCH 7/7] pypi.eclass: Translate ${PV} by default

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass   | 21 +++++++++++++++------
 eclass/tests/pypi.sh | 24 ++++++++++++------------
 2 files changed, 27 insertions(+), 18 deletions(-)

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index 27bd17403100..be0e498fcbf3 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -87,7 +87,10 @@ pypi_translate_version() {
 # is normalized according to the specification unless `--no-normalize`
 # is passed.
 #
-# If <version> is unspecified, it defaults to ${PV}.
+# If <version> is unspecified, it defaults to ${PV} translated
+# via pypi_translate_version.  If it is specified, then it is used
+# verbatim (the function can be called explicitly to translate custom
+# version number).
 #
 # If <format> is unspecified, it defaults to ".tar.gz".  Another valid
 # value is ".zip" (please remember to add a BDEPEND on app-arch/unzip).
@@ -103,7 +106,7 @@ pypi_sdist_url() {
 	fi
 
 	local project=${1-"${PN}"}
-	local version=${2-"${PV}"}
+	local version=${2-"$(pypi_translate_version "${PV}")"}
 	local suffix=${3-.tar.gz}
 	local fn_project=${project}
 	[[ ${normalize} ]] && fn_project=$(pypi_normalize_name "${project}")
@@ -119,7 +122,10 @@ pypi_sdist_url() {
 # If <package> is unspecified, it defaults to ${PN}.  The package name
 # is normalized according to the wheel specification.
 #
-# If <version> is unspecified, it defaults to ${PV}.
+# If <version> is unspecified, it defaults to ${PV} translated
+# via pypi_translate_version.  If it is specified, then it is used
+# verbatim (the function can be called explicitly to translate custom
+# version number).
 #
 # If <python-tag> is unspecified, it defaults to "py3".  It can also be
 # "py2.py3", or a specific version in case of non-pure wheels.
@@ -133,7 +139,7 @@ pypi_wheel_name() {
 	fi
 
 	local project=$(pypi_normalize_name "${1-"${PN}"}")
-	local version=${2-"${PV}"}
+	local version=${2-"$(pypi_translate_version "${PV}")"}
 	local pytag=${3-py3}
 	local abitag=${4-none-any}
 	echo "${project}-${version}-${pytag}-${abitag}.whl"
@@ -152,7 +158,10 @@ pypi_wheel_name() {
 #
 # If <package> is unspecified, it defaults to ${PN}.
 #
-# If <version> is unspecified, it defaults to ${PV}.
+# If <version> is unspecified, it defaults to ${PV} translated
+# via pypi_translate_version.  If it is specified, then it is used
+# verbatim (the function can be called explicitly to translate custom
+# version number).
 #
 # If <python-tag> is unspecified, it defaults to "py3".  It can also be
 # "py2.py3", or a specific version in case of non-pure wheels.
@@ -173,7 +182,7 @@ pypi_wheel_url() {
 
 	local filename=$(pypi_wheel_name "${@}")
 	local project=${1-"${PN}"}
-	local version=${2-"${PV}"}
+	local version=${2-"$(pypi_translate_version "${PV}")"}
 	local pytag=${3-py3}
 	printf "https://files.pythonhosted.org/packages/%s" \
 		"${pytag}/${project::1}/${project}/${filename}"
diff --git a/eclass/tests/pypi.sh b/eclass/tests/pypi.sh
index af5d714af748..e114549633a0 100755
--- a/eclass/tests/pypi.sh
+++ b/eclass/tests/pypi.sh
@@ -6,7 +6,7 @@ EAPI=8
 source tests-common.sh || exit
 
 PN=Foo.Bar
-PV=1.2.3
+PV=1.2.3_beta2
 
 inherit pypi
 
@@ -39,8 +39,8 @@ test-eq "pypi_translate_version 1.2.3_beta1" 1.2.3b1
 test-eq "pypi_translate_version 1.2.3_rc2" 1.2.3rc2
 test-eq "pypi_translate_version 1.2.3_rc2_p1" 1.2.3rc2.post1
 
-test-eq "pypi_wheel_name" foo_bar-1.2.3-py3-none-any.whl
-test-eq "pypi_wheel_name Flask-BabelEx" flask_babelex-1.2.3-py3-none-any.whl
+test-eq "pypi_wheel_name" foo_bar-1.2.3b2-py3-none-any.whl
+test-eq "pypi_wheel_name Flask-BabelEx" flask_babelex-1.2.3b2-py3-none-any.whl
 test-eq "pypi_wheel_name Flask-BabelEx 4" flask_babelex-4-py3-none-any.whl
 test-eq "pypi_wheel_name Flask-BabelEx 4 py2.py3" \
 	flask_babelex-4-py2.py3-none-any.whl
@@ -48,9 +48,9 @@ test-eq "pypi_wheel_name cryptography 39.0.1 cp36 abi3-manylinux_2_28_x86_64" \
 	cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl
 
 test-eq "pypi_wheel_url" \
-	https://files.pythonhosted.org/packages/py3/F/Foo.Bar/foo_bar-1.2.3-py3-none-any.whl
+	https://files.pythonhosted.org/packages/py3/F/Foo.Bar/foo_bar-1.2.3b2-py3-none-any.whl
 test-eq "pypi_wheel_url Flask-BabelEx" \
-	https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-1.2.3-py3-none-any.whl
+	https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-1.2.3b2-py3-none-any.whl
 test-eq "pypi_wheel_url Flask-BabelEx 4" \
 	https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-4-py3-none-any.whl
 test-eq "pypi_wheel_url Flask-BabelEx 4 py2.py3" \
@@ -59,9 +59,9 @@ test-eq "pypi_wheel_url cryptography 39.0.1 cp36 abi3-manylinux_2_28_x86_64" \
 	https://files.pythonhosted.org/packages/cp36/c/cryptography/cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl
 
 test-eq "pypi_wheel_url --unpack" \
-	"https://files.pythonhosted.org/packages/py3/F/Foo.Bar/foo_bar-1.2.3-py3-none-any.whl -> foo_bar-1.2.3-py3-none-any.whl.zip"
+	"https://files.pythonhosted.org/packages/py3/F/Foo.Bar/foo_bar-1.2.3b2-py3-none-any.whl -> foo_bar-1.2.3b2-py3-none-any.whl.zip"
 test-eq "pypi_wheel_url --unpack Flask-BabelEx" \
-	"https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-1.2.3-py3-none-any.whl -> flask_babelex-1.2.3-py3-none-any.whl.zip"
+	"https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-1.2.3b2-py3-none-any.whl -> flask_babelex-1.2.3b2-py3-none-any.whl.zip"
 test-eq "pypi_wheel_url --unpack Flask-BabelEx 4" \
 	"https://files.pythonhosted.org/packages/py3/F/Flask-BabelEx/flask_babelex-4-py3-none-any.whl -> flask_babelex-4-py3-none-any.whl.zip"
 test-eq "pypi_wheel_url --unpack Flask-BabelEx 4 py2.py3" \
@@ -70,24 +70,24 @@ test-eq "pypi_wheel_url --unpack cryptography 39.0.1 cp36 abi3-manylinux_2_28_x8
 	"https://files.pythonhosted.org/packages/cp36/c/cryptography/cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl -> cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl.zip"
 
 test-eq "pypi_sdist_url" \
-	https://files.pythonhosted.org/packages/source/F/Foo.Bar/foo_bar-1.2.3.tar.gz
+	https://files.pythonhosted.org/packages/source/F/Foo.Bar/foo_bar-1.2.3b2.tar.gz
 test-eq "pypi_sdist_url Flask-BabelEx" \
-	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/flask_babelex-1.2.3.tar.gz
+	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/flask_babelex-1.2.3b2.tar.gz
 test-eq "pypi_sdist_url Flask-BabelEx 4" \
 	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/flask_babelex-4.tar.gz
 test-eq "pypi_sdist_url Flask-BabelEx 4 .zip" \
 	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/flask_babelex-4.zip
 
 test-eq "pypi_sdist_url --no-normalize" \
-	https://files.pythonhosted.org/packages/source/F/Foo.Bar/Foo.Bar-1.2.3.tar.gz
+	https://files.pythonhosted.org/packages/source/F/Foo.Bar/Foo.Bar-1.2.3b2.tar.gz
 test-eq "pypi_sdist_url --no-normalize Flask-BabelEx" \
-	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/Flask-BabelEx-1.2.3.tar.gz
+	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/Flask-BabelEx-1.2.3b2.tar.gz
 test-eq "pypi_sdist_url --no-normalize Flask-BabelEx 4" \
 	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/Flask-BabelEx-4.tar.gz
 test-eq "pypi_sdist_url --no-normalize Flask-BabelEx 4 .zip" \
 	https://files.pythonhosted.org/packages/source/F/Flask-BabelEx/Flask-BabelEx-4.zip
 
 test-eq 'declare -p SRC_URI' \
-	'declare -- SRC_URI="https://files.pythonhosted.org/packages/source/F/Foo.Bar/foo_bar-1.2.3.tar.gz"'
+	'declare -- SRC_URI="https://files.pythonhosted.org/packages/source/F/Foo.Bar/foo_bar-1.2.3b2.tar.gz"'
 
 texit
-- 
2.39.1

Re: [gentoo-dev] [PATCH 0/7] pypi.eclass: Filename and version normalization

[Michał Górny]

On Sat, 2023-02-11 at 10:16 +0100, Michał Górny wrote:
> Hi,
> 
> Here's a major patchset to the recently committed pypi.eclass (i.e.
> the "change while it's still warm and not used much" kind).
> 
> It focuses on two significant changes:
> 
> 1. Normalizing filenames according to the current sdist/wheel standards.
> 
> 2. Translating Gentoo-specific components in ${PV} to the corresponding
>    standard Python versions.
> 
> The former change is focused on improving correctness (i.e. sometimes
> the current API didn't allow for getting the correct URI), the latter
> on convenience (i.e. not having to override ${PV} sometimes).
> 
> For example, a package called "Test-Package-1.2.3_beta4" that followed
> PEP 625 (e.g. by using Hatchling) would require an ebuild calling
> the equivalent of:
> 
>   $(pypi_sdist_url "test_package" "${PV/_beta/b}")
> 
> while now it will work out of the box.
> 
> Unfortunately, PEP 625 is not yet universally followed, so many packages
> will instead need to:
> 
>   $(pypi_sdist_url --no-normalize)
> 
> but I think following the standard immediately and requiring a number
> of `--no-normalize` calls is better than doing it the other way around
> and then switching the defaults, even if the majority of affected
> packages use "legacy" naming right now.
> 
> That said, this only affects packages with uppercase letters, "-" or "."
> in their names.
> 

A minute after sending this I've realized that we need to apply the same
operations to S as well.  In reply, I'm adding two more patches that
respectively:

1) perform normalization and translation in S by default (i.e. make it
match the default SRC_URI),

2) add a more convenient PYPI_NO_NORMALIZE variable that can be used to
control whether SRC_URI and S use normalized or legacy names.

-- 
Best regards,
Michał Górny

[gentoo-dev] [PATCH 1/2] pypi.eclass: Normalize PN and translate PV for S as well

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass   | 19 ++++++++++++++-----
 eclass/tests/pypi.sh |  3 +++
 2 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index be0e498fcbf3..f4367b3fbfec 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -11,12 +11,20 @@
 # @DESCRIPTION:
 # The pypi.eclass can be used to easily obtain URLs for artifacts
 # uploaded to PyPI.org.  When inherited, the eclass defaults SRC_URI
-# to fetch ${P}.tar.gz sdist.
+# and S to fetch .tar.gz sdist.  The project filename is normalized
+# by default, and the version is translated using
+# pypi_translate_version.
 #
-# If necessary, SRC_URI can be overriden by the ebuild.  Two helper
-# functions, pypi_sdist_url and pypi_wheel_url are provided to generate
-# URLs to artifacts of specified type, with customizable project name.
-# Additionally, pypi_wheel_name can be used to generate wheel filename.
+# If necessary, SRC_URI and S can be overriden by the ebuild.  Two
+# helper functions, pypi_sdist_url and pypi_wheel_url are provided
+# to generate URLs to artifacts of specified type, with customizable
+# URL components.  Additionally, pypi_wheel_name can be used to generate
+# wheel filename.
+#
+# pypi_normalize_name can be used to normalize an arbitrary project name
+# according to sdist/wheel normalization rules.  pypi_translate_version
+# can be used to translate a Gentoo version string into its PEP 440
+# equivalent.
 #
 # @EXAMPLE:
 # @CODE@
@@ -193,5 +201,6 @@ pypi_wheel_url() {
 }
 
 SRC_URI="$(pypi_sdist_url)"
+S="${WORKDIR}/$(pypi_normalize_name "${PN}")-$(pypi_translate_version "${PV}")"
 
 fi
diff --git a/eclass/tests/pypi.sh b/eclass/tests/pypi.sh
index e114549633a0..ebfcdb630856 100755
--- a/eclass/tests/pypi.sh
+++ b/eclass/tests/pypi.sh
@@ -7,6 +7,7 @@ source tests-common.sh || exit
 
 PN=Foo.Bar
 PV=1.2.3_beta2
+WORKDIR='<WORKDIR>'
 
 inherit pypi
 
@@ -89,5 +90,7 @@ test-eq "pypi_sdist_url --no-normalize Flask-BabelEx 4 .zip" \
 
 test-eq 'declare -p SRC_URI' \
 	'declare -- SRC_URI="https://files.pythonhosted.org/packages/source/F/Foo.Bar/foo_bar-1.2.3b2.tar.gz"'
+test-eq 'declare -p S' \
+	'declare -- S="<WORKDIR>/foo_bar-1.2.3b2"'
 
 texit
-- 
2.39.1

[gentoo-dev] [PATCH 2/2] pypi.eclass: Add PYPI_NO_NORMALIZE control var

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 eclass/pypi.eclass | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/eclass/pypi.eclass b/eclass/pypi.eclass
index f4367b3fbfec..182b0b6b8c3e 100644
--- a/eclass/pypi.eclass
+++ b/eclass/pypi.eclass
@@ -12,7 +12,8 @@
 # The pypi.eclass can be used to easily obtain URLs for artifacts
 # uploaded to PyPI.org.  When inherited, the eclass defaults SRC_URI
 # and S to fetch .tar.gz sdist.  The project filename is normalized
-# by default, and the version is translated using
+# by default (unless PYPI_NO_NORMALIZE is set prior to inheriting
+# the eclass), and the version is translated using
 # pypi_translate_version.
 #
 # If necessary, SRC_URI and S can be overriden by the ebuild.  Two
@@ -42,6 +43,13 @@ esac
 if [[ ! ${_PYPI_ECLASS} ]]; then
 _PYPI_ECLASS=1
 
+# @ECLASS_VARIABLE: PYPI_NO_NORMALIZE
+# @PRE_INHERIT
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# When set to a non-empty value, disables project name normalization
+# for the default SRC_URI and S values.
+
 # @FUNCTION: pypi_normalize_name
 # @USAGE: <name>
 # @DESCRIPTION:
@@ -200,7 +208,12 @@ pypi_wheel_url() {
 	fi
 }
 
-SRC_URI="$(pypi_sdist_url)"
-S="${WORKDIR}/$(pypi_normalize_name "${PN}")-$(pypi_translate_version "${PV}")"
+if [[ ${PYPI_NO_NORMALIZE} ]]; then
+	SRC_URI="$(pypi_sdist_url --no-normalize)"
+	S="${WORKDIR}/${PN}-$(pypi_translate_version "${PV}")"
+else
+	SRC_URI="$(pypi_sdist_url)"
+	S="${WORKDIR}/$(pypi_normalize_name "${PN}")-$(pypi_translate_version "${PV}")"
+fi
 
 fi
-- 
2.39.1