From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id D42F01580E0 for ; Mon, 27 Jan 2025 17:37:25 +0000 (UTC) Received: from lists.gentoo.org (bobolink.gentoo.org [140.211.166.189]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) (Authenticated sender: relay-lists.gentoo.org@gentoo.org) by smtp.gentoo.org (Postfix) with ESMTPSA id BD2D53433AB for ; Mon, 27 Jan 2025 17:37:25 +0000 (UTC) Received: from bobolink.gentoo.org (localhost [127.0.0.1]) by bobolink.gentoo.org (Postfix) with ESMTP id 7C6C411047D; Mon, 27 Jan 2025 17:36:19 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by bobolink.gentoo.org (Postfix) with ESMTPS id 6DE9911046F for ; Mon, 27 Jan 2025 17:36:18 +0000 (UTC) Received: from [IPV6:2603:6011:3f0:6f00::12ac] (unknown [IPv6:2603:6011:3f0:6f00::12ac]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: eschwartz) by smtp.gentoo.org (Postfix) with ESMTPSA id E5AD834338E for ; Mon, 27 Jan 2025 17:36:17 +0000 (UTC) Message-ID: Date: Mon, 27 Jan 2025 12:36:14 -0500 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [gentoo-user] Properly isolating system Python to nuke PEP 668 from orbit forever? To: gentoo-user@lists.gentoo.org References: <20250127103640.27896b0d@framework> Content-Language: en-US From: Eli Schwartz Autocrypt: addr=eschwartz@gentoo.org; keydata= xjMEZmeRNBYJKwYBBAHaRw8BAQdAYNZ7pUDWhx1i2f3p6L2ZLu4FcY18UoeGC04Gq/khqwfN I0VsaSBTY2h3YXJ0eiA8ZXNjaHdhcnR6QGdlbnRvby5vcmc+wpYEExYKAD4WIQTvUdMIsc4j CIi+DYTqQj6ToWND8QUCZoRL+gIbAwUJBKKGAAULCQgHAwUVCgkICwUWAgMBAAIeBQIXgAAK CRDqQj6ToWND8aB5AP9r4kB691nNtNwKkdRiOdl7/k6WYzokvHvDamXxRJ0I+gEAjZqR5V8y mfR3fy2Z+r2Joeqdt3CIv5IwPs64spBvigLOOARmZ5E0EgorBgEEAZdVAQUBAQdATT46Z06b 1X9xjXFCYFxmq/Tj3tSEKZInDWTpoHQp4l8DAQgHwn4EGBYKACYWIQTvUdMIsc4jCIi+DYTq Qj6ToWND8QUCZmeRNAIbDAUJBKKGAAAKCRDqQj6ToWND8a2RAP40KPfbfoiZAJW5boFmFJ3G TUBDJRh9CWHyaPqq2PN+0wD/R07oLzfnJUN209mzi9TuTuHjeZybysyqXSw4MAxkMAY= In-Reply-To: <20250127103640.27896b0d@framework> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------02rWtaxVIFbIymqgBQSI2mu2" X-Archives-Salt: f42800de-a9b4-47f6-b985-026b3a79fe76 X-Archives-Hash: 4051d7592c2f7a0373ef4b2ab6e92ead This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------02rWtaxVIFbIymqgBQSI2mu2 Content-Type: multipart/mixed; boundary="------------h9uso10o8VlfoRxIEbgSOMrB"; protected-headers="v1" From: Eli Schwartz To: gentoo-user@lists.gentoo.org Message-ID: Subject: Re: [gentoo-user] Properly isolating system Python to nuke PEP 668 from orbit forever? References: <20250127103640.27896b0d@framework> In-Reply-To: <20250127103640.27896b0d@framework> --------------h9uso10o8VlfoRxIEbgSOMrB Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 1/27/25 11:36 AM, Matthew Brooks wrote: > Hello! >=20 >=20 > Not sure where the most appropriate place to ask this is, so if some > other list or something would be more appropriate, please let me > know. >=20 > I'm interested in trying to solve the potential system python > breakage that PEP 668 was meant to address, but solving it in a way > that *doesn't* irrevocably break users' python package install > workflows for all eternity like PEP 668 does. From what I've read, > it looks like the issue is at least potentially addressable from the > distro side, and since Gentoo is the only distro I care about, I > figured I'd check here first. >=20 > So, does anyone know if there was any particular discussions of what > alternatives might have been available, or what things specifically > could/would break in Gentoo as a result of user-installed python > packages interacting with system ones? I know I can't be the only > one who was rather bothered by the manner it was addressed, so I > figure there was probably at least some exploratory testing or > theorizing done before PEP 668 was addopted into Gentoo, which would > give me a starting point to work from. The PyPA community motivating factors here were: - if you pip install --user, it can upgrade copies of packages such that running /usr/bin/emerged-program as non-root sees the copy in $HOME which it is incompatible with. Fedora discussed at the time that it is possible to simply patch the shebang to use `python -s`, then decided to not use that approach in the end because their package manager supports pip install --user plugins and using `-s` would prevent detecting plugins, therefore "better to forbid pip install --user". Yes, that conclusion confused me too. - if you `pip install --user some-app` it will use system copies of e.g. scipy/numpy. If you then emerge -uDU @world and upgrade to a new version of numpy, your app might not be compatible with it. This is not actually a problem with the distro, and the famous byword of the Python community is "the consenting adults principle" which states that users should be permitted to choose whether the risk of breakage is acceptable to them. This "consenting adults" principle does not apply with regard to PEP 668. The PyPA community insists that the adult in the room is the distro, and that PyPA has not forbidden anything -- distros are free to use PEP 668 if they wish to, or not use it if they don't wish to. The distro motivating factor here varies according to distro. For Debian, users are definitely not adults. Debian got bug reports for users whose --user installed software was causing issues, and did not wish to support this use case. They wrote a PEP and got it approved, to allow the distro to indicate "go jump in a lake, we don't support you", which eventually coalesced into "either pass the --break-system-packages option or jump in a lake, either way you agree that we don't support you"= =2E The existence of that option highlights the motivating purpose of the PEP. It's not about whether you can or cannot do stuff, it's about "whether you are allowed to ask for help after you do it". Pass the option and you "become an adult", to use the PyPA expression. I asked about this with regard to Gentoo. The answer I received was that previously, Gentoo patched pip to error out if you run `pip install` as root, to prevent pip deleting or overwriting files managed by portage, and that was undesirable because distro patches need to be "maintained". Furthermore, my offer to maintain the patch is not an acceptable solution= =2E It is acknowledged by Gentoo Python (apparently) that PEP 668 is a terrible idea and absolutely not what anyone wants (especially since it does not prevent pip from deleting or overwriting files managed by portage), but that it ***will*** be used anyway because it allows avoiding a pip patch. Here is my 2023 updated patch for pip, it still applies today without additional maintenance: https://github.com/eli-schwartz/pip/commit/f5c63c0deff8e90862fb3586c381c8= 664a98a2ef Here is my 2023 PR to gpep517 to allow updating shebangs to add `-s` when installing a python program with emerge: https://github.com/projg2/gpep517/pull/13 Assuming this PR could someday get merged, though it's not clear how much interest there is (I've tried asking for reviews several times, as have others), it would be trivial to pass the newly added flag in the ::gentoo eclass (distutils-r1) that currently invokes gpep517 to install python software. It would also need to be separately reimplemented in python-single-r1 for packages that ship python scripts directly. A plan of action for how I can proceed would be amazing. I'm out of ideas= ! > For clarity, I'm *not* looking for a workaround for my particular > system. My end goal (and not an easy one, granted) is to hopefully > eventually make a proper fix that makes it so no user of any distro > ever needs to touch a venv again if they don't want to. >=20 > I'm only a hobbyist programmer, but since nearly every distro under > the sun went with the "externally managed environment" thing, it's > clearly more an issue of motivation/time rather than raw skill. So I > figure maybe my extreme distaste for the current solution might > allow me to bash my head against the problem long enough to make > some headway. I feel your frustration, especially the motivation part. :( --=20 Eli Schwartz --------------h9uso10o8VlfoRxIEbgSOMrB-- --------------02rWtaxVIFbIymqgBQSI2mu2 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature.asc" -----BEGIN PGP SIGNATURE----- wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZ5fEDwUDAAAAAAAKCRCEp9ErcA0vV5HE AQDG9uCrmJlsh+I8IblZejW8G9ZDMERR3DFDU345M5gQxgEA69tKB0D4rnnbT/w9v+0/HLiVvAqU 77sKntqnTgaFBQI= =Bxyp -----END PGP SIGNATURE----- --------------02rWtaxVIFbIymqgBQSI2mu2--