From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 529B9158094 for ; Fri, 15 Jul 2022 21:35:03 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 91511E0F61; Fri, 15 Jul 2022 21:34:58 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id B656FE0C9A for ; Fri, 15 Jul 2022 21:34:57 +0000 (UTC) Received: (nullmailer pid 15863 invoked by uid 1000); Fri, 15 Jul 2022 21:34:54 -0000 Date: Fri, 15 Jul 2022 16:34:54 -0500 From: William Hubbs To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] Proposal to undeprecate EGO_SUM Message-ID: Mail-Followup-To: gentoo-dev@lists.gentoo.org References: <20220613074411.341909-1-flow@gentoo.org> <1a712a66f55e241ce6b6084eb19e1f34@sinustrom.info> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="5tGxYi5j2Qr8a3Eq" Content-Disposition: inline In-Reply-To: <1a712a66f55e241ce6b6084eb19e1f34@sinustrom.info> X-Archives-Salt: 609f77d1-6b4b-4793-add8-b62919efe32c X-Archives-Hash: 7ff79a5ec239c90827c8795c30e1b65a --5tGxYi5j2Qr8a3Eq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jun 27, 2022 at 01:43:19AM +0200, Zoltan Puskas wrote: *snip* > First of all one of the advantages of Gentoo is that it gets it's source= =20 > code from upstream (yes, I'm aware of mirrors acting as a cache layer),= =20 > which means that poisoning source code needs to be done at upstream=20 > level (effectively means hacking GitHub, PyPi, or some standalone=20 > project's Gitea/cgit/gitlab/etc. instance or similar), sources which=20 > either have more scrutiny or have a limited blast radius. I don't quite follow what you mean. Upstream for go modules is actually proxy.golang.org, or some other similar proxy, which the go tooling knows how to access [1]. > Additionally if an upstream dependency has a security issue it's easier= =20 > to scan all EGO_SUM content and find packages that potentially depend on= =20 > a broken dependency and force a re-pinning and rebuild. The tarball=20 > magic hides this completely and makes searching very expensive. I'm not comfortable at all with us changing the dependencies like this downstream for the same reason the Debian folks ultimately were against it for kubernetes. If you make these kinds of changes you are affectively creating a fork, and that would mean we would be building packages with unt= ested libraries [2]. *snip* > Considering that BTRFS (and possibly other filesystems) support on the=20 > fly compression the physical cost of a few inflated ebuilds and=20 The problem here is the size of SRC_URI when you add the EGO_SUM_SRC_URI to it. SRC_URI gets exported to the environment, so it can crash portage if it is too big. > Manifests is actually way smaller than the logical size would indicate.= =20 > Compare that to the huge incompressible tarballs that now we need to=20 > store. >=20 > As a proxied maintainer or overlay owner hosting these huge tarballs=20 > also becomes problem (i.e. we need some public space with potentially=20 > gigabytes of free space and enough bandwidth to push that to users).=20 > Pushing toward vendor tarballs creates an extra expense on every level=20 > (Gentoo infra, mirrors, proxy maintainers, overlay owners, users). I agree that creating the dependency tarballs is not ideal. We asked for another option [3], but as you can see from the bug this was refused by the PMS team. That refusal is the only reason we have to worry about dependency tarballs. > It also breaks reproducibility. With EGO_SUM I can check out an older=20 > version of portage tree (well to some extent) and rebuild packages since= =20 > dependency upstream is very likely to host old versions of their source.= =20 > With the tarballs this breaks since as soon as an ebuild is dropped from= =20 > mainline portage the vendor tarballs follow them too. There is no way=20 > for the user to roll back a package a few weeks back (e.g. if new=20 > version has bugs), unlike with EGO_SUM. The contents of a dependency tarball is created using "go mod download", which is controlled by the go.mod/go.sum files in the package. So, it is possible to recreate the dependency tarball any time. I do not see any advantage EGO_SUM offers over the dependency tarballs in this space. > Finally with EGO_SUM we had a nice tool get-ego-vendor which produced=20 > the EGO_SUM for maintainers which has made maintenance easier. However I= =20 > haven't found any new guidance yet on how to maintain go packages with=20 > the new tarball method (e.g. what needs to go into the vendor tarball,=20 > what changes are needed in ebuilds). Overall this complifates further=20 > ebuild development and verification of PRs. The documentation for how to build dependency tarballs is in the eclass. The GOMODCACHE environment variable is used in the eclass to point to the location where the dependency tarball is unpacked, and that location is read by the normal go tooling. > In summary, IMHO the EGO_SUM way of handling of go packages has more=20 > benefits than drawbacks compared to the vendor tarballs. EGO_SUM can cause portage to break; that is the primary reason support is going away. We attempted another solution that was refused, so the only option we have currently is to build the dependency tarballs. >=20 > Cheers, > Zoltan >=20 > [1]=20 > https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-= nightmare/ >=20 [1] https://go.dev/ref/mod [2] https://lwn.net/Articles/835599/ [3] https://bugs.gentoo.org/833567 --5tGxYi5j2Qr8a3Eq Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iF0EABECAB0WIQTVeuxEZo4uUHOkQAluVBb0MMRlOAUCYtHdeAAKCRBuVBb0MMRl OIpNAKCdYN7FSaRw774BEAJDlIUdxKEgPQCfbomzx3rnmV8poqiQIZ9axtGjh3Q= =p+QH -----END PGP SIGNATURE----- --5tGxYi5j2Qr8a3Eq--