From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id A4FA9138334 for ; Mon, 25 Nov 2019 19:29:47 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 86649E09D4; Mon, 25 Nov 2019 19:29:43 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 19A2BE09C3 for ; Mon, 25 Nov 2019 19:29:43 +0000 (UTC) Received: from pomiot (c134-66.icpnet.pl [85.221.134.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: mgorny) by smtp.gentoo.org (Postfix) with ESMTPSA id 297E734D39B; Mon, 25 Nov 2019 19:29:40 +0000 (UTC) Message-ID: <4aa5755bc00cb14f04f284bbe2c0416638a9debb.camel@gentoo.org> Subject: Re: [gentoo-dev] [RFC] dev-python/setuptools deps in distutils-r1 packages From: =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?= To: gentoo-dev@lists.gentoo.org Date: Mon, 25 Nov 2019 20:29:36 +0100 In-Reply-To: <5d7c0ecb-605e-b813-abad-20ea05ca4848@gentoo.org> References: <5d7c0ecb-605e-b813-abad-20ea05ca4848@gentoo.org> Organization: Gentoo Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-xg/3q41e74kajp+QOuG+" User-Agent: Evolution 3.32.4 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 X-Archives-Salt: b3339b84-f337-4d95-9f9b-ccb62b7b3379 X-Archives-Hash: 3631a7bd4cf56ec3ae88feaf2115f4ed --=-xg/3q41e74kajp+QOuG+ Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2019-11-25 at 20:39 +0200, Joonas Niilola wrote: > Hey, >=20 >=20 > On 11/25/19 7:38 PM, Micha=C5=82 G=C3=B3rny wrote: > > Hi, > >=20 > > TL;DR: should we depend on setuptools by default? Alternatively, shoul= d > > we add distutils_enable_setuptools API to provide at least partial > > validity checks. > >=20 > >=20 > > The problem > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > The vast majority of Python packages nowadays uses setuptools as their > > build system. According to a cheap grep, 1633 out of 2415 packages > > using distutils-r1 depends on it -- and I suspect the vast majority of > > those that do not are simply missing the dependency. >=20 > Truthfully I thought distutils already (B)DEPENDed on setuptools, so at > least I've left it out "on purpose". I was recently made aware it > doesn't. No bug reports are made about it, so I think it's safe to > assume setuptools is already present in every system? Yes, we do get bug reports about it occassionally. >=20 >=20 > > Are there valid cases for not using setuptools? Notably, packages > > needed to bootstrap setuptools aren't using it. Plus some simple > > packages that really don't need its features. > >=20 > > There are also packages that use setuptools but have distutils fallback= . > > We don't like them because switching between build systems involves fil= e > > <-> directory replacement that isn't handled cleanly by our package > > managers. Therefore, we want to always force one build system in them. > >=20 > > The setuptools dependency is so common it's easy to miss. Notably, it'= s > > a prerequisite of specifying package dependencies, so it's normally not > > listed in dependencies. > >=20 > > Finally, while most of the time setuptools is just BDEPEND, there are > > cases when it's RDEPEND as well. The most common is the use > > of entry_points -- and again, upstreams don't mention it explicitly. > >=20 > > All that considered, I think we should work on providing a better API > > for depending on setuptools, to reduce the number of missing > > dependencies and make it possible to automatically test for them > > (reminder: PMS doesn't permit inspecting *DEPEND). > >=20 > >=20 > > Variant 1: automatic dependency on setuptools > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Basically, we add a new trinary pre-inherit variable: > >=20 > > DISTUTILS_USE_SETUPTOOLS=3Dno > > -> no deps > > DISTUTILS_USE_SETUPTOOLS=3Dbdepend > > -> add to BDEPEND (the default) > > DISTUTILS_USE_SETUPTOOLS=3Drdepend > > -> add to BDEPEND+RDEPEND > >=20 > > This is roughly 'erring on the safe side'. The default will work for > > the majority of packages. We will have to disable it for setuptools > > bootstrap deps, and devs will be able to adjust it to correct values > > as they update ebuilds. For the time being, existing *DEPEND > > on setuptools will avoid breaking stuff. > >=20 > > This will also enable me to add extra QA checks to esetup.py. It shoul= d > > be able to reasonably detect incorrect value and report it. This will > > imply some 'false positives' for packages that use the old method of > > specifying setuptools in RDEPEND but that's a minor hassle. > >=20 > > Pros: > > - works out of the box for the majority of packages > > - enables full-range QA checking > >=20 > > Cons: > > - pre-inherit variable > > - some (harmless) false positives on existing packages >=20 > I'm a fan of this solution, unless it causes a lot of new CI noise. I > think people are getting fed up with recent changes already, starting to > ignore whatever CI bot says, unless it's fatal. >=20 > No harm in double-defining setuptools in BDEPEND right...? (via eclass > and ebuild) Yes, double defining is fine. >=20 >=20 > >=20 > > Variant 2: distutils_enable_setuptools > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > The alternative method is to add another function > > to the distutils_enable* series: > >=20 > > distutils_enable_setuptools [-r] > >=20 > > The basic form adds it to BDEPEND, -r B+RDEPEND. Of course, no dep is > > present by default. The main difference from setting deps explicitly i= s > > that it permits us to do a minimal QA check between pure BDEPEND > > and B+RDEPEND for now. > >=20 > > When all ebuilds are migrated from explicit dependencies to this method= , > > we can also start detecting missing deps completely. However, that > > presumes we require using this function rather than explicit deps. > >=20 > > Pros: > > - no pre-inherit variables > >=20 > > Cons: > > - only partial QA check possible for the time being > > - requires migrating all ebuilds long-term >=20 > I don't like this at this point. This would just bring unwanted noise > and lots of editing in ebuilds. V1 and V3 are better IMHO. This could be > implemented in distutils-r2.eclass, though. >=20 >=20 > >=20 > > Variant 3: leave as-is, add minimal install-qa-check.d > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D > > We can just continue adding deps manually, and add a minimal install-qa= - > > check.d that greps installed scripts for entry_points usage. This will > > let us detect missing RDEPEND on setuptools but not BDEPEND. > >=20 > > Pros: > > - no changes to ebuilds > >=20 > > Cons: > > - only partial QA check possible >=20 > The old way, is also good. >=20 >=20 > >=20 > > WDYT? > > =3D=3D=3D=3D=3D > > Both options have their pros and cons. I think V1 is the best since it > > avoids a common mistake, and gives full range QA check. It also doesn'= t > > interfere with existing deps. V2 neatly fits into the recent series bu= t > > still requires users to remember to call it, and we can't report missin= g > > calls until we clean up all ebuilds. V3 provides only minimal QA > > improvement without changing the eclass. > >=20 > > WDYT? Do you have any other ideas? >=20 > As I said, I don't even know how one ends up with a system without > setuptools, so I don't view this "superimportant". Still I like your > suggestions 1 & 3 in handling it in future. However if you're going to > implement CI checks with this, I don't believe the timing is right > currently. >=20 1. If you have no packages with RDEPEND on setuptools and some --with- bdeps setting, --depclean can kill it. 2. Early after install. 3. On tinderbox, if no dep pulled it in. --=20 Best regards, Micha=C5=82 G=C3=B3rny --=-xg/3q41e74kajp+QOuG+ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQGTBAABCgB9FiEEx2qEUJQJjSjMiybFY5ra4jKeJA4FAl3cK6FfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEM3 NkE4NDUwOTQwOThEMjhDQzhCMjZDNTYzOUFEQUUyMzI5RTI0MEUACgkQY5ra4jKe JA4zBAf9H+pksv9b7PNLC2rp6lGum7mwvk1JYZxrvCs/AGVOK/8UTcssEbYOCzVQ 7zf3oMiqTeeUY/OS9PR+60Cpt3PH5COENcORh0CrkJ80tqnCgn/VlnBe7Na+/8iB armtev5u0WnWPRJAtGsRBTcTHb38PrEIeTHNOWsL/1sY2UIPCbUJPyYAgK0/tuU4 QxjukqyLw2weQAuo6NQ5heYrAu8ABsHAqT+X0SJsHBDi+qxQH8DcpkB4FxPMLqoe 40JbHn9NycJ/DPsb/us/zRdCFR9Q6vLyr+T+MuZFGpkZh+FNJJ1Q3s8mCbP7IfSs 5CQMoKk99CcWm94BO6F79CZ9oTPf0A== =ZxwJ -----END PGP SIGNATURE----- --=-xg/3q41e74kajp+QOuG+--