Re: [gentoo-user] Properly isolating system Python to nuke PEP 668 from orbit forever?

[Eli Schwartz <eschwartz@gentoo.org>]

[-- Attachment #1.1: Type: text/plain, Size: 5564 bytes --]

On 1/27/25 11:36 AM, Matthew Brooks wrote:
> Hello!
> 
> 
> Not sure where the most appropriate place to ask this is, so if some
> other list or something would be more appropriate, please let me
> know.
> 
> I'm interested in trying to solve the potential system python
> breakage that PEP 668 was meant to address, but solving it in a way
> that *doesn't* irrevocably break users' python package install
> workflows for all eternity like PEP 668 does. From what I've read,
> it looks like the issue is at least potentially addressable from the
> distro side, and since Gentoo is the only distro I care about, I
> figured I'd check here first.
> 
> So, does anyone know if there was any particular discussions of what
> alternatives might have been available, or what things specifically
> could/would break in Gentoo as a result of user-installed python
> packages interacting with system ones? I know I can't be the only
> one who was rather bothered by the manner it was addressed, so I
> figure there was probably at least some exploratory testing or
> theorizing done before PEP 668 was addopted into Gentoo, which would
> give me a starting point to work from.


The PyPA community motivating factors here were:

- if you pip install --user, it can upgrade copies of packages such that
  running /usr/bin/emerged-program as non-root sees the copy in $HOME
  which it is incompatible with. Fedora discussed at the time that it is
  possible to simply patch the shebang to use `python -s`, then decided
  to not use that approach in the end because their package manager
  supports pip install --user plugins and using `-s` would prevent
  detecting plugins, therefore "better to forbid pip install --user".
  Yes, that conclusion confused me too.

- if you `pip install --user some-app` it will use system copies of e.g.
  scipy/numpy. If you then emerge -uDU @world and upgrade to a new
  version of numpy, your app might not be compatible with it. This is
  not actually a problem with the distro, and the famous byword of the
  Python community is "the consenting adults principle" which states
  that users should be permitted to choose whether the risk of breakage
  is acceptable to them.


This "consenting adults" principle does not apply with regard to PEP
668. The PyPA community insists that the adult in the room is the
distro, and that PyPA has not forbidden anything -- distros are free to
use PEP 668 if they wish to, or not use it if they don't wish to.


The distro motivating factor here varies according to distro.

For Debian, users are definitely not adults. Debian got bug reports for
users whose --user installed software was causing issues, and did not
wish to support this use case. They wrote a PEP and got it approved, to
allow the distro to indicate "go jump in a lake, we don't support you",
which eventually coalesced into "either pass the --break-system-packages
option or jump in a lake, either way you agree that we don't support you".

The existence of that option highlights the motivating purpose of the
PEP. It's not about whether you can or cannot do stuff, it's about
"whether you are allowed to ask for help after you do it". Pass the
option and you "become an adult", to use the PyPA expression.


I asked about this with regard to Gentoo. The answer I received was that
previously, Gentoo patched pip to error out if you run `pip install` as
root, to prevent pip deleting or overwriting files managed by portage,
and that was undesirable because distro patches need to be "maintained".
Furthermore, my offer to maintain the patch is not an acceptable solution.

It is acknowledged by Gentoo Python (apparently) that PEP 668 is a
terrible idea and absolutely not what anyone wants (especially since it
does not prevent pip from deleting or overwriting files managed by
portage), but that it ***will*** be used anyway because it allows
avoiding a pip patch.

Here is my 2023 updated patch for pip, it still applies today without
additional maintenance:
https://github.com/eli-schwartz/pip/commit/f5c63c0deff8e90862fb3586c381c8664a98a2ef

Here is my 2023 PR to gpep517 to allow updating shebangs to add `-s`
when installing a python program with emerge:
https://github.com/projg2/gpep517/pull/13

Assuming this PR could someday get merged, though it's not clear how
much interest there is (I've tried asking for reviews several times, as
have others), it would be trivial to pass the newly added flag in the
::gentoo eclass (distutils-r1) that currently invokes gpep517 to install
python software. It would also need to be separately reimplemented in
python-single-r1 for packages that ship python scripts directly.


A plan of action for how I can proceed would be amazing. I'm out of ideas!


> For clarity, I'm *not* looking for a workaround for my particular
> system. My end goal (and not an easy one, granted) is to hopefully
> eventually make a proper fix that makes it so no user of any distro
> ever needs to touch a venv again if they don't want to.
> 
> I'm only a hobbyist programmer, but since nearly every distro under
> the sun went with the "externally managed environment" thing, it's
> clearly more an issue of motivation/time rather than raw skill. So I
> figure maybe my extreme distaste for the current solution might
> allow me to bash my head against the problem long enough to make
> some headway.


I feel your frustration, especially the motivation part. :(


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]