[gentoo-user] Properly isolating system Python to nuke PEP 668 from orbit forever?

[gentoo-user] Properly isolating system Python to nuke PEP 668 from orbit forever?

[Matthew Brooks]

Hello!


Not sure where the most appropriate place to ask this is, so if some other list or something would be more appropriate, please let me know.

I'm interested in trying to solve the potential system python breakage that PEP 668 was meant to address, but solving it in a way that *doesn't* irrevocably break users' python package install workflows for all eternity like PEP 668 does. From what I've read, it looks like the issue is at least potentially addressable from the distro side, and since Gentoo is the only distro I care about, I figured I'd check here first.

So, does anyone know if there was any particular discussions of what alternatives might have been available, or what things specifically could/would break in Gentoo as a result of user-installed python packages interacting with system ones? I know I can't be the only one who was rather bothered by the manner it was addressed, so I figure there was probably at least some exploratory testing or theorizing done before PEP 668 was addopted into Gentoo, which would give me a starting point to work from.


For clarity, I'm *not* looking for a workaround for my particular system. My end goal (and not an easy one, granted) is to hopefully eventually make a proper fix that makes it so no user of any distro ever needs to touch a venv again if they don't want to.

I'm only a hobbyist programmer, but since nearly every distro under the sun went with the "externally managed environment" thing, it's clearly more an issue of motivation/time rather than raw skill. So I figure maybe my extreme distaste for the current solution might allow me to bash my head against the problem long enough to make some headway.

Re: [gentoo-user] Properly isolating system Python to nuke PEP 668 from orbit forever?

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 1739 bytes --]

On Mon, Jan 27, 2025 at 9:37 AM Matthew Brooks <matthewfbrooks@posteo.net>
wrote:
>
> Hello!
>

Hi.

First, I cannot offer any insight into the PEP668 stuff. For me, it is what
it is.

> For clarity, I'm *not* looking for a workaround for my particular system.
My end goal (and not an easy one, granted) is to hopefully eventually make
a proper fix that makes it so no user of any distro ever needs to touch a
venv again if they don't want to.
>
> I'm only a hobbyist programmer, but since nearly every distro under the
sun went with the "externally managed environment" thing, it's clearly more
an issue of motivation/time rather than raw skill. So I figure maybe my
extreme distaste for the current solution might allow me to bash my head
against the problem long enough to make some headway.

Like you I am a hobbyist programmer and, in a _very_ general way agree with
your setup of "so no user of any distro ever needs to touch a venv again if
they don't want to". However my experience programming Python always leads
me towards the use of off-brand and sometimes poorly maintained Python
libraries that don't work in all python releases so I'm forced to use venv,
which for me works great. I also use a lot of CUDA stuff within Python and
tend, possibly due to my own lack of skills as a hobbyist, or just my
general paranoia about having to debug things I don't want to mess with, to
do _everything_ personal in a venv. Grant you, I never intend to release a
program to the public so to me it's most important to keep it working, and
clearly nothing about what you might end up suggesting stops me from doing
what you are interested in.

I wish you the best in your adventure.

Cheers,
Mark

[-- Attachment #2: Type: text/html, Size: 2032 bytes --]

Re: [gentoo-user] Properly isolating system Python to nuke PEP 668 from orbit forever?

[William Kenworthy]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]


On 28/1/25 01:16, Mark Knecht wrote:
>
>
> On Mon, Jan 27, 2025 at 9:37 AM Matthew Brooks 
> <matthewfbrooks@posteonet <mailto:matthewfbrooks@posteo.net>> wrote:
> >
> > Hello!
> >
>
> Hi.
>
> First, I cannot offer any insight into the PEP668 stuff. For me, it is 
> what it is.
>
> > For clarity, I'm *not* looking for a workaround for my particular 
> system. My end goal (and not an easy one, granted) is to hopefully 
> eventually make a proper fix that makes it so no user of any distro 
> ever needs to touch a venv again if they don't want to.
> >
> > I'm only a hobbyist programmer, but since nearly every distro under 
> the sun went with the "externally managed environment" thing, it's 
> clearly more an issue of motivation/time rather than raw skill. So I 
> figure maybe my extreme distaste for the current solution might allow 
> me to bash my head against the problem long enough to make some headway.
>
> Like you I am a hobbyist programmer and, in a _very_ general way agree 
> with your setup of "so no user of any distro ever needs to touch a 
> venv again if they don't want to". However my experience programming 
> Python always leads me towards the use of off-brand and sometimes 
> poorly maintained Python libraries that don't work in all python 
> releases so I'm forced to use venv, which for me works great. I also 
> use a lot of CUDA stuff within Python and tend, possibly due to my own 
> lack of skills as a hobbyist, or just my general paranoia about having 
> to debug things I don't want to mess with, to do _everything_ personal 
> in a venv. Grant you, I never intend to release a program to the 
> public so to me it's most important to keep it working, and clearly 
> nothing about what you might end up suggesting stops me from doing 
> what you are interested in.
>
> I wish you the best in your adventure.
>
> Cheers,
> Mark
>
Same here, I regard current python and python based programs a mostly 
broken despite the considerable effort of the gentoo devs ... if you 
want privacy, do not want to use someone elses docker containers or just 
use portage installed programs.  Example Frigate, example Homeassistant. 
example python itself ... venv's and all these walled gardens that are 
not talking to each other ...

I am developing quite an aversion to python.

BillK


[-- Attachment #2: Type: text/html, Size: 3601 bytes --]

Re: [gentoo-user] Properly isolating system Python to nuke PEP 668 from orbit forever?

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 5564 bytes --]

On 1/27/25 11:36 AM, Matthew Brooks wrote:
> Hello!
> 
> 
> Not sure where the most appropriate place to ask this is, so if some
> other list or something would be more appropriate, please let me
> know.
> 
> I'm interested in trying to solve the potential system python
> breakage that PEP 668 was meant to address, but solving it in a way
> that *doesn't* irrevocably break users' python package install
> workflows for all eternity like PEP 668 does. From what I've read,
> it looks like the issue is at least potentially addressable from the
> distro side, and since Gentoo is the only distro I care about, I
> figured I'd check here first.
> 
> So, does anyone know if there was any particular discussions of what
> alternatives might have been available, or what things specifically
> could/would break in Gentoo as a result of user-installed python
> packages interacting with system ones? I know I can't be the only
> one who was rather bothered by the manner it was addressed, so I
> figure there was probably at least some exploratory testing or
> theorizing done before PEP 668 was addopted into Gentoo, which would
> give me a starting point to work from.


The PyPA community motivating factors here were:

- if you pip install --user, it can upgrade copies of packages such that
  running /usr/bin/emerged-program as non-root sees the copy in $HOME
  which it is incompatible with. Fedora discussed at the time that it is
  possible to simply patch the shebang to use `python -s`, then decided
  to not use that approach in the end because their package manager
  supports pip install --user plugins and using `-s` would prevent
  detecting plugins, therefore "better to forbid pip install --user".
  Yes, that conclusion confused me too.

- if you `pip install --user some-app` it will use system copies of e.g.
  scipy/numpy. If you then emerge -uDU @world and upgrade to a new
  version of numpy, your app might not be compatible with it. This is
  not actually a problem with the distro, and the famous byword of the
  Python community is "the consenting adults principle" which states
  that users should be permitted to choose whether the risk of breakage
  is acceptable to them.


This "consenting adults" principle does not apply with regard to PEP
668. The PyPA community insists that the adult in the room is the
distro, and that PyPA has not forbidden anything -- distros are free to
use PEP 668 if they wish to, or not use it if they don't wish to.


The distro motivating factor here varies according to distro.

For Debian, users are definitely not adults. Debian got bug reports for
users whose --user installed software was causing issues, and did not
wish to support this use case. They wrote a PEP and got it approved, to
allow the distro to indicate "go jump in a lake, we don't support you",
which eventually coalesced into "either pass the --break-system-packages
option or jump in a lake, either way you agree that we don't support you".

The existence of that option highlights the motivating purpose of the
PEP. It's not about whether you can or cannot do stuff, it's about
"whether you are allowed to ask for help after you do it". Pass the
option and you "become an adult", to use the PyPA expression.


I asked about this with regard to Gentoo. The answer I received was that
previously, Gentoo patched pip to error out if you run `pip install` as
root, to prevent pip deleting or overwriting files managed by portage,
and that was undesirable because distro patches need to be "maintained".
Furthermore, my offer to maintain the patch is not an acceptable solution.

It is acknowledged by Gentoo Python (apparently) that PEP 668 is a
terrible idea and absolutely not what anyone wants (especially since it
does not prevent pip from deleting or overwriting files managed by
portage), but that it ***will*** be used anyway because it allows
avoiding a pip patch.

Here is my 2023 updated patch for pip, it still applies today without
additional maintenance:
https://github.com/eli-schwartz/pip/commit/f5c63c0deff8e90862fb3586c381c8664a98a2ef

Here is my 2023 PR to gpep517 to allow updating shebangs to add `-s`
when installing a python program with emerge:
https://github.com/projg2/gpep517/pull/13

Assuming this PR could someday get merged, though it's not clear how
much interest there is (I've tried asking for reviews several times, as
have others), it would be trivial to pass the newly added flag in the
::gentoo eclass (distutils-r1) that currently invokes gpep517 to install
python software. It would also need to be separately reimplemented in
python-single-r1 for packages that ship python scripts directly.


A plan of action for how I can proceed would be amazing. I'm out of ideas!


> For clarity, I'm *not* looking for a workaround for my particular
> system. My end goal (and not an easy one, granted) is to hopefully
> eventually make a proper fix that makes it so no user of any distro
> ever needs to touch a venv again if they don't want to.
> 
> I'm only a hobbyist programmer, but since nearly every distro under
> the sun went with the "externally managed environment" thing, it's
> clearly more an issue of motivation/time rather than raw skill. So I
> figure maybe my extreme distaste for the current solution might
> allow me to bash my head against the problem long enough to make
> some headway.


I feel your frustration, especially the motivation part. :(


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]