[gentoo-dev] [RFC] dev-python/namespace-* retirement plan

[gentoo-dev] [RFC] dev-python/namespace-* retirement plan

[Michał Górny]

Hi, everyone.

TL;DR I think I came up with a feasible plan towards cleanly removing
dev-python/namespace-* packages that aren't technically needed anymore
with modern versions of Python.


What are namespace packages?  Let's take "zope" as an example. 
Normally, various subpackages like "zope.interface" and
"zope.configuration" would have all to be present in a single directory,
that is inside the "zope" package.  However, if "zope" is made a
namespace package, its subpackages can be loaded from different
locations.  Notably, in order to test zope.configuration prior to
installing it, we load it from build tree but its dependency
zope.interface from system site-packages.

Historically, for this to work, the "zope/__init__.py" had to specify
some magic incantations.  On Gentoo, we've added dev-python/namespace-*
that installed these magical "__init__.py" files and made all
subpackages (e.g. dev-python/zope-*) depend on it.

However, Python 3.3 introduced PEP 420 implicit namespaces that made
magical incantations unnecessary -- if "zope" didn't include
"__init__.py", it was implicitly treated as a namespace.  Unfortunately,
there were some interoperability issues between the two kinds of magical
incantations and implicit namespaces.  So while we were able to retire
pkgutil-style namespaces, setuptools-style namespaces remained.

I think we can change that now.


I've done some experiments, particularly with dev-python/zope-interface
and dev-python/zope-configuration.  If nobody finds a problem with this
solution, I think we can aim to bump all packages relying on namespaces
and retire them in 1-2 months.

The rough idea is to remove RDEP on dev-python/namespace-* from
the package, and updating python_test to use a temporary pkgutil-style
incantation (yes, for setuptools-style packages), e.g.:

```
python_compile() {
	distutils-r1_python_compile
	find "${BUILD_DIR}" -name '*.pth' -delete || die
}

python_test() {
	cd "${BUILD_DIR}/install$(python_get_sitedir)" || die
	# this is needed to keep the tests working while
	# dev-python/namespace-zope is still installed
	cat > zope/__init__.py <<-EOF || die
		__path__ = __import__('pkgutil').extend_path(__path__,
__name__)
	EOF
	eunittest
	rm zope/__init__.py || die
}
```

With this hack, I've been able to get the tests to work with all
the possible scenarios, i.e.:

a. with and without the package in question installed,

b. with and without dev-python/namespace-* installed.

The second point means that we can transition gradually.  Once all
packages with namespace-* dep are gone, we can lastrite that package
and add blockers to trigger cleanup from user systems.  We'll have to
keep the test hack for some more time but I don't think that's a deal
breaker.


WDYT?

-- 
Best regards,
Michał Górny

[gentoo-dev] Re: [RFC] dev-python/namespace-* retirement plan

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 3431 bytes --]

On 09/04/2022 18.20, Michał Górny wrote:
> Hi, everyone.
> 
> TL;DR I think I came up with a feasible plan towards cleanly removing
> dev-python/namespace-* packages that aren't technically needed anymore
> with modern versions of Python.
> 
> 
> What are namespace packages?  Let's take "zope" as an example. 
> Normally, various subpackages like "zope.interface" and
> "zope.configuration" would have all to be present in a single directory,
> that is inside the "zope" package.  However, if "zope" is made a
> namespace package, its subpackages can be loaded from different
> locations.  Notably, in order to test zope.configuration prior to
> installing it, we load it from build tree but its dependency
> zope.interface from system site-packages.
> 
> Historically, for this to work, the "zope/__init__.py" had to specify
> some magic incantations.  On Gentoo, we've added dev-python/namespace-*
> that installed these magical "__init__.py" files and made all
> subpackages (e.g. dev-python/zope-*) depend on it.
> 
> However, Python 3.3 introduced PEP 420 implicit namespaces that made
> magical incantations unnecessary -- if "zope" didn't include
> "__init__.py", it was implicitly treated as a namespace.  Unfortunately,
> there were some interoperability issues between the two kinds of magical
> incantations and implicit namespaces.  So while we were able to retire
> pkgutil-style namespaces, setuptools-style namespaces remained.
> 
> I think we can change that now.
> 
> 
> I've done some experiments, particularly with dev-python/zope-interface
> and dev-python/zope-configuration.  If nobody finds a problem with this
> solution, I think we can aim to bump all packages relying on namespaces
> and retire them in 1-2 months.
> 
> The rough idea is to remove RDEP on dev-python/namespace-* from
> the package, and updating python_test to use a temporary pkgutil-style
> incantation (yes, for setuptools-style packages), e.g.:
> 
> ```
> python_compile() {
> 	distutils-r1_python_compile
> 	find "${BUILD_DIR}" -name '*.pth' -delete || die
> }
> 
> python_test() {
> 	cd "${BUILD_DIR}/install$(python_get_sitedir)" || die
> 	# this is needed to keep the tests working while
> 	# dev-python/namespace-zope is still installed
> 	cat > zope/__init__.py <<-EOF || die
> 		__path__ = __import__('pkgutil').extend_path(__path__,
> __name__)
> 	EOF
> 	eunittest
> 	rm zope/__init__.py || die
> }
> ```

I quite like that we make the "hack" more local, just in test phase. But
I think this code is quite error prone, and an easy mistake can be made
if someone copies it incorrectly.

Is it possible to add an eclass function to auto create and cleanup this
file, and all I need to do is to pass it the correct namespace ("zope"
in above example)?

Also, by using a common function name, it would be easy to fix it if we
did a mistake and find all consumers when we feel safe to cleanup this code.

One point of failure for this function is how to do the auto cleanup? I
was thinking using environment variables to hold our added magic names,
which is created before `python_test` call and cleans after it is done.
I really prefer if we could call it during prepare phase (so we can
still use auto generated d_e_t calls).

> ...
> 
> WDYT?
> 


-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, GURU, Arch Teams)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Re: [RFC] dev-python/namespace-* retirement plan

[Michał Górny]

On Sat, 2022-04-09 at 19:08 +0300, Arthur Zamarin wrote:
> On 09/04/2022 18.20, Michał Górny wrote:
> > Hi, everyone.
> > 
> > TL;DR I think I came up with a feasible plan towards cleanly removing
> > dev-python/namespace-* packages that aren't technically needed anymore
> > with modern versions of Python.
> > 
> > 
> > What are namespace packages?  Let's take "zope" as an example. 
> > Normally, various subpackages like "zope.interface" and
> > "zope.configuration" would have all to be present in a single directory,
> > that is inside the "zope" package.  However, if "zope" is made a
> > namespace package, its subpackages can be loaded from different
> > locations.  Notably, in order to test zope.configuration prior to
> > installing it, we load it from build tree but its dependency
> > zope.interface from system site-packages.
> > 
> > Historically, for this to work, the "zope/__init__.py" had to specify
> > some magic incantations.  On Gentoo, we've added dev-python/namespace-*
> > that installed these magical "__init__.py" files and made all
> > subpackages (e.g. dev-python/zope-*) depend on it.
> > 
> > However, Python 3.3 introduced PEP 420 implicit namespaces that made
> > magical incantations unnecessary -- if "zope" didn't include
> > "__init__.py", it was implicitly treated as a namespace.  Unfortunately,
> > there were some interoperability issues between the two kinds of magical
> > incantations and implicit namespaces.  So while we were able to retire
> > pkgutil-style namespaces, setuptools-style namespaces remained.
> > 
> > I think we can change that now.
> > 
> > 
> > I've done some experiments, particularly with dev-python/zope-interface
> > and dev-python/zope-configuration.  If nobody finds a problem with this
> > solution, I think we can aim to bump all packages relying on namespaces
> > and retire them in 1-2 months.
> > 
> > The rough idea is to remove RDEP on dev-python/namespace-* from
> > the package, and updating python_test to use a temporary pkgutil-style
> > incantation (yes, for setuptools-style packages), e.g.:
> > 
> > ```
> > python_compile() {
> > 	distutils-r1_python_compile
> > 	find "${BUILD_DIR}" -name '*.pth' -delete || die
> > }
> > 
> > python_test() {
> > 	cd "${BUILD_DIR}/install$(python_get_sitedir)" || die
> > 	# this is needed to keep the tests working while
> > 	# dev-python/namespace-zope is still installed
> > 	cat > zope/__init__.py <<-EOF || die
> > 		__path__ = __import__('pkgutil').extend_path(__path__,
> > __name__)
> > 	EOF
> > 	eunittest
> > 	rm zope/__init__.py || die
> > }
> > ```
> 
> I quite like that we make the "hack" more local, just in test phase. But
> I think this code is quite error prone, and an easy mistake can be made
> if someone copies it incorrectly.
> 
> Is it possible to add an eclass function to auto create and cleanup this
> file, and all I need to do is to pass it the correct namespace ("zope"
> in above example)?
> 
> Also, by using a common function name, it would be easy to fix it if we
> did a mistake and find all consumers when we feel safe to cleanup this code.
> 
> One point of failure for this function is how to do the auto cleanup? I
> was thinking using environment variables to hold our added magic names,
> which is created before `python_test` call and cleans after it is done.
> I really prefer if we could call it during prepare phase (so we can
> still use auto generated d_e_t calls).
> 

That's pretty much the big problem -- I don't really know.  I feel like
adding two functions is pretty much error prone itself, especially given
that this 'cd' logic is not necessarily correct for other packages.

In the end, we're talking about a few ebuilds for period of a few
months, so I'm not sure if there's really a point in putting more effort
into it.

-- 
Best regards,
Michał Górny