From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 13A05158091 for ; Wed, 8 Jun 2022 20:42:55 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 04F60E07C9; Wed, 8 Jun 2022 20:42:51 +0000 (UTC) Received: from smtp.gentoo.org (dev.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id B2297E0794 for ; Wed, 8 Jun 2022 20:42:50 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 866BC3410AE for ; Wed, 8 Jun 2022 20:42:49 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) by grubbs.orbis-terrarum.net (Postfix) with ESMTP id EBEEC2601C6 for ; Wed, 8 Jun 2022 20:42:48 +0000 (UTC) Received: (qmail 12479 invoked by uid 10000); 8 Jun 2022 20:42:48 -0000 Date: Wed, 8 Jun 2022 20:42:48 +0000 From: "Robin H. Johnson" To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] About EGO_SUM Message-ID: References: <9d4adb56-34be-7058-3979-2c99178251dd@gentoo.org> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="/aH5F4uIR/cvc9Ha" Content-Disposition: inline In-Reply-To: <9d4adb56-34be-7058-3979-2c99178251dd@gentoo.org> X-Archives-Salt: 857e1a95-8711-4dd1-a9f9-b59bb4654fb9 X-Archives-Hash: 8e2a4002bfc6258d65dcf725db347cb9 --/aH5F4uIR/cvc9Ha Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 03, 2022 at 01:18:08PM +0200, Florian Schmaus wrote: > EGO_SUM is marked as 'deprecated' in go-module.eclass [1, 2]. I=20 > acknowledge that there are packages where the usage of EGO_SUM is very=20 > problematic. However, I wonder if there are packages where using=20 > dependency tarballs is problematic while using EGO_SUM would be not. =2E.. [snip all the great points] > Even more problematic are that dependency tarballs require additional=20 > steps that would not be required when EGO_SUM is used. While those steps= =20 > appear simple, behavioral theory shows that even the tiniest additional= =20 > steps have a huge impact (e.g., online shops loose a relative large=20 > share of customers if for each an additional checkout step). If we force= =20 > dependency tarballs for Go software, then packaging Go software just=20 > become a little bit harder. Your above is entirely correct, and I was against the plan to introduce dependency tarballs. > This leads me to the question why are we actually deprecating EGO_SUM?=20 > It seems like a nice alternative for Go packaging that we may want to=20 > keep. But maybe I am missing something? EGO_SUM vs dependency tarballs: - bloats ebuilds - bloats Manifests - bloats metadata/md5-cache/ (SRC_URI etc) - doesn't bloat mirrors with gentoo-unique distfiles - EGO_SUM is verifiable/reproducible from Upstream Go systems - less downloads on upgrades (only changed Go deps, not entire dep tarballs) EGO_SUM data right now adds, to every user's system: - 2.6MB of text to ebuilds (340k after de-dupe) - 7MB of text to Manifests (2M after de-dupe) - 6.4MB+ of text to metadata/md5-cache (I don't have a easy way to calc ded= uped amount here) On the server side: - The sum total of Go distfiles mirrored on Gentoo mirrors right now is onl= y 3.4GB. - less downloads Dependency tarballs: - Right now ~15GiB on each mirror, plus storage of the primary copy somewhere (dev.g.o right now, but not great) - Conservatively if the remaining EGO_SUM packages converted to Dep tarballs, it would need another 8GB each of primary location and mirrors. - larger downloads for users who DO want to upgrade a Go package (all new deps tarball even if only one or two deps changed) - must be preserved much longer, unless we can introduce a guaranteed way to regenerate them for any prior ebuild. I was trying to introduce a third option, but I haven't had the time to write an entire GLEP. The TL;DR is introducing a 2nd-level Manifest+metadata file, that tries to move just the metadata out of the tree, in a way that can be regenerated (specifically, a 1:1 reproducible creation from a given go.sum). It DOES need to contain slightly more data than the present Manifest, specifically a full SRC_URI entry for each file (upstream URI plus what to rename it to on Gentoo side) The 2nd-level Manifest would be listed as SRC_URI, and be handled in src_fetch/src_unpack. Download & verify the extra distfiles, against the Manifest checksum data (and for Golang against go.sum checksums). The Portage mirrordist code needs the most work in this case, as it would need to fetch the 2nd-level Manifests so it can populate Gentoo mastermirror with the distfiles mirrored from upstream. The storage costs for the proposed idea: - same 1:1 base distfile storage as EGO_SUM (e.g. upstream distfiles are mirrored 1:1 content, just different naming) - Probably 1 Metadata-Manifest file per ebuild $PVR (conceptually it could be split more or shared between some ebuilds/packages) - Main tree Manifests: 1 DIST entry per Metadata-Manifest in a given package - Main tree ebuilds: 1 line for the Metadata-Manifest in the ebuild. - metadata/md5-cache: 1 src_uri line! - mirrors: add the Metadata-Manifest --=20 Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 --/aH5F4uIR/cvc9Ha Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it. iQKTBAABCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAmKhCcZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsSmnxAAiJ/HitJcTsqa7IIN9cp22VtBtMu5H3WtJ2XzDDZmxO822BLphY0ecMVT YhriXtVD0uaUhKXVAD4LkKHQDung81+n/xKjv39/Vlq3+S38TeUdjiwKotKCrXNX CkcSZKsTN7SaA1KkTrnicdQT+tSIfiZBIHK+O4LDhqt9xX6+Cky0GW+oRSJ1IFG4 eKi1UzHg52vviPrYNFjufYjvxyBMvYmXDY0xB1Xv+1wfcc4xjSL4NVU6SdBBtYBT zCqcUFJhuETaCS0SiXNR+011UXucFO7ZnPLKrkX0OEelXsSvVLgWtYkse0cBtdAj a55NKZYwGMWu95jRC9MN1U5mZ4uRd2DYY/ySE5OCySZONMPlHbZgaWxWrlLULsZ8 zrOwhWc90eEa+wlF1TSr7r8cr/e40OdNRqZsif1RtSNOxEgjIuR9L9DG7Jr7qN+E /dwkc+WzqkZsMHp6p02DZwa2tgj162jH330P4GkOY0oEh5W6dS8AM0FsCR9CQNZg ya21nUIB+P0OpCQRKd6q0a69d1iAiJ+G+lB3kTLvukFHNrUtXJHPKZDNj33IHcG3 Fk++3KscI/h3TEI4qvf9s1nYihAyjgap9ewVeCIa3NTlNenr+Uf3DXFeONzo0RaD UN003xX4Un6gwff7FGIPK/XsWddeBbmBNkbInEhVVbTP4PYPDg8= =Xa0H -----END PGP SIGNATURE----- --/aH5F4uIR/cvc9Ha--