Re: [gentoo-python] RFC: Providing explicit support for namespaces

[Mike Gilbert <floppym@gentoo.org>]

On Fri, May 5, 2017 at 7:55 AM, Michał Górny <mgorny@gentoo.org> wrote:
> Hi, everyone.
>
> Python namespaces are an ugly hack that Python upstreams do mostly
> to look more enterprise'y and support installing subpackages of a common
> parent package in different locations (i.e. zope.*, ...). For more
> information on them, read PEP 420 [1], Python Packaging Guide [2]
> and our wiki page [3].
>
> So far we haven't been really providing any special support for
> namespaces. We were rather relying on things magically working out
> of the box and applying distutils_install_for_testing whenever they
> weren't. However, this was really a 'big hammer' solution -- not very
> precise, not very reliable.
>
> I've been working on adding missing test dependencies of yet another
> zope-related package lately [4] and hit a few more issues with namespace
> of zope.*. It turned out that due to some random changes in Python 3.5+,
> 'setup.py install --home=...' (used for d_i_f_t) no longer correctly
> established namespaces for our use and I had to use 'setup.py develop'
> instead for zope.testrunner [5]. However, even this didn't work for
> zope.exceptions where I would probably have to also deploy 'setup.py
> easy_install' or add requirements to setup.py.
>
> So I've went ahead and explored other possibilities. I've figured out
> that at the moment namespace packages work on old Python versions
> because of *.pth files installed by setuptools along with the packages.
> However, this hack might be deprecated [6]. Also, it seemed to be
> the source of problems with Python 3.5+.
>
>
> You can see what I did on the PR [4]. It boils down to:
>
> 1. Adding dev-python/namespace-zope package (name selected to avoid
> collisions with potential packages) that installs setuptools-compliant
> __init__.py.
>
> 2. Removing *.pth files from dev-python/zope* and making those packages
> depend on dev-python/namespace-zope instead.
>
> This done, I was able to get the tests on the 5 zope packages to pass
> on all Python implementations without any additional logic
> (in particular, without a call to d_i_f_t).
>
>
> If you consider this an acceptable solution to the namespace problem,
> I'd like to set up the following guidelines for dealing with namespaces:
>
> 1. A single package should be selected to 'hold' the namespace. It can
> be a common package for the ns (e.g. dev-python/logilab-common)
> or a dedicated dev-python/namespace-* package.
>
> 2. This package should install appropriate __init__.py for the
> namespace. The contents should be selected to match the namespace logic
> used by the packages (pkgutil vs setuptools).
>
> 3. All subpackages in the namespace should RDEP (+ DEP) on this package,
> and not install *-nspkg.pth files.
>
>
> Your thoughts?

Thank you for the well-organized walkthrough. Your approach makes sense to me.