From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 6AE04158094 for ; Mon, 27 Jun 2022 06:09:27 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 36DA4E0882; Mon, 27 Jun 2022 06:09:21 +0000 (UTC) Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id E7575E0874 for ; Mon, 27 Jun 2022 06:09:20 +0000 (UTC) Received: by mail-wr1-x435.google.com with SMTP id o4so7488245wrh.3 for ; Sun, 26 Jun 2022 23:09:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:subject:message-id:mail-followup-to:references :mime-version:content-disposition:in-reply-to; bh=uZ9JiMKCpvvYV1SIt6AAhPaaQnORXejuEyPaZMSg2Ow=; b=dfr1yjdgc05bHvnV2QXnTdmKW8ecOFZAH59GBbpiAPUxwn6FfJKy4c+WQz+wR0DeKa LQyH7YOfme/qCIvjAxQRLh/pmRTOcvu8z9ova8GDdM8Mgxp4i0AywOPuNnSZLOugxNaW KtXaxLB+wpQAxYhICknU8ObRFlmwhN5qjYrqUFqt1h1ECEIeNaLFE6eLDpTDdwdhzaGw 6gusnyRTyR96563QN1HqGlZTQMcFAP+FeByQW3iqwO/gTREAiDaSPvRYd5pnfFKKoWKr sXIjjZd2IKIlXgs/rAuDd/c1ly1IbzqdexDrrsxrI6OZ9XI9OOnCW6dwrr4Yh5mA0K3G 2zNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:subject:message-id:mail-followup-to :references:mime-version:content-disposition:in-reply-to; bh=uZ9JiMKCpvvYV1SIt6AAhPaaQnORXejuEyPaZMSg2Ow=; b=KXVX8CvpbPNnA2wa1eTX8v7MatGBpMxQn/VsL+O4kvD4ItzGeH6TquQsFr1TsGuVUr iPVBf52f7ywgLHisxdAXKyMlWED8/1UwxOzE/gUbMtpqphCNCJZrsrzXHx70GX4Ir1Iy KkhPZ39o7guMe2fOd1bgl0imiu0aE/Ffm2fYTo1vyNzZmp7OldJ+rDccM1insfVC+uIk Wlgg/rP9sp80e/E2TDhbEU9hBh3rKI1Cg8RO8GUamKJOtAT8IrbF3Q3LOxWV+nubO7ax u6BV9mkfsQHTe3am4qs9TueHdnHqIqbXz44fxtdUDZvo9X5rPsYodMXa4eLFSFb1qeXm hUoA== X-Gm-Message-State: AJIora/cb3fqoCtIxYjESBdG4TMNEle/5ipRTUie73VE929vAE/qpjKA x3EBlfSsTzuJbBxC/OD6j9fSqeuL/lg= X-Google-Smtp-Source: AGRyM1s+V8Vi++4xRkk2sl5bYGovttegAQ0TOCe9mtolNCiV5GQrY1KQ5clJGkuGDJrCuPxAuvX+NQ== X-Received: by 2002:a5d:5a19:0:b0:21b:8fe9:8fe with SMTP id bq25-20020a5d5a19000000b0021b8fe908femr10408439wrb.85.1656310159294; Sun, 26 Jun 2022 23:09:19 -0700 (PDT) Received: from dj3ntoo (54.sub-72-105-159.myvzw.com. [72.105.159.54]) by smtp.gmail.com with ESMTPSA id h6-20020adffd46000000b0021b96cdf68fsm9307398wrs.97.2022.06.26.23.09.17 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Jun 2022 23:09:18 -0700 (PDT) Date: Mon, 27 Jun 2022 01:09:14 -0500 From: Oskari Pirhonen To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] Proposal to undeprecate EGO_SUM Message-ID: Mail-Followup-To: gentoo-dev@lists.gentoo.org References: <20220613074411.341909-1-flow@gentoo.org> <1a712a66f55e241ce6b6084eb19e1f34@sinustrom.info> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="CDbWLZStpNKdiCVt" Content-Disposition: inline In-Reply-To: <1a712a66f55e241ce6b6084eb19e1f34@sinustrom.info> X-Archives-Salt: 192e4b9b-71ab-4879-a986-7bd58698d42c X-Archives-Hash: a6b1222513ac01a1926b91888c96c3e4 --CDbWLZStpNKdiCVt Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jun 27, 2022 at 01:43:19 +0200, Zoltan Puskas wrote: > Hi, >=20 > I've been working on adding a go based ebuild to Gentoo yesterday and I= =20 > got this warning form portage saying that EGO_SUM is deprecated and=20 > should be avoided. Since I remember there was an intense discussion=20 > about this on the ML I went back and have re-read the threads before=20 > writing this piece. I'd like to provide my perspective as user, a=20 > proxied maintainer, and overlay owner. I also run a private mirror on my= =20 > LAN to serve my hosts in order to reduce load on external mirrors. >=20 > Before diving in I think it's worth reading mgorny's blog post "The=20 > modern packager=E2=80=99s security nightmare"[1] as it's relevant to the= =20 > discussion, and something I deeply agree with. >=20 > With all that being said, I feel that the tarball idea is a bad due to=20 > many reasons. >=20 > From security point of view, I understand that we still have to trust=20 > maintainers not to do funky stuff, but I think this issue goes beyond=20 > that. >=20 > First of all one of the advantages of Gentoo is that it gets it's source= =20 > code from upstream (yes, I'm aware of mirrors acting as a cache layer),= =20 > which means that poisoning source code needs to be done at upstream=20 > level (effectively means hacking GitHub, PyPi, or some standalone=20 > project's Gitea/cgit/gitlab/etc. instance or similar), sources which=20 > either have more scrutiny or have a limited blast radius. >=20 > Additionally if an upstream dependency has a security issue it's easier= =20 > to scan all EGO_SUM content and find packages that potentially depend on= =20 > a broken dependency and force a re-pinning and rebuild. The tarball=20 > magic hides this completely and makes searching very expensive. >=20 > In fact using these vendor tarballs is the equivalent of "static=20 > linking" in the packaging space. Why are we introducing the same issue=20 > in the repository space? This kills the reusability of already=20 > downloaded dependencies and bloats storage requirements. This is=20 > especially bad on laptops, where SSD free space might be limited, in=20 > case the user does not nuke their distfiles after each upgrade. >=20 > Considering that BTRFS (and possibly other filesystems) support on the=20 > fly compression the physical cost of a few inflated ebuilds and=20 > Manifests is actually way smaller than the logical size would indicate.= =20 > Compare that to the huge incompressible tarballs that now we need to=20 > store. >=20 > As a proxied maintainer or overlay owner hosting these huge tarballs=20 > also becomes problem (i.e. we need some public space with potentially=20 > gigabytes of free space and enough bandwidth to push that to users).=20 > Pushing toward vendor tarballs creates an extra expense on every level=20 > (Gentoo infra, mirrors, proxy maintainers, overlay owners, users). >=20 > If bloating portage is a big issue and we frown upon go stuff anyway (or= =20 > only a few users need these packages), why not consider moving all go=20 > packages into an officially supported go packages only overlay? I=20 > understand that this would not solve the kernel buffer issue where we=20 > run out of environment variable space, but it would debloat the main=20 > portage tree. >=20 Rephrasing this just to ensure I'm understanding it correctly: you're suggesting to move _everything_ that uses Go into its own overlay. Let's call it gentoo-go for the sake of the example. If the above is accurate, then I hard disagree. The biggest package that I have that uses Go is docker (and accompanying tools). Personal distaste of docker aside, it's a very popular piece of software, and I don't think it's fair to require all the people who want to use it to first enable and sync gentoo-go before they can install it. And what about transitive dependencies? Suppose app-misc/cool-package is written in some language that isn't Go, but it has a dependency on sys-apps/cool-util which has a dependency on something written in Go. Should a user wanting to install cool-package have to enable the gentoo-go overlay now too? Even though app-misc/cool-package would look like it doesn't need the overlay unless you dig into the deps. Not a dev, just a user who really likes Gentoo :) - Oskari > It also breaks reproducibility. With EGO_SUM I can check out an older=20 > version of portage tree (well to some extent) and rebuild packages since= =20 > dependency upstream is very likely to host old versions of their source.= =20 > With the tarballs this breaks since as soon as an ebuild is dropped from= =20 > mainline portage the vendor tarballs follow them too. There is no way=20 > for the user to roll back a package a few weeks back (e.g. if new=20 > version has bugs), unlike with EGO_SUM. >=20 > In fact I feel this goes against the spirit of portage too, since now=20 > instead of "just describing" how to obtain sources and build them, now=20 > it now depends on essentially ephemeral blobs, which happens to be=20 > externalized from the portage tree itself. I'm aware that we have=20 > ebuilds that pull in patches and other stuff from dev space already, but= =20 > we shouldn't make this even worse. >=20 > Finally with EGO_SUM we had a nice tool get-ego-vendor which produced=20 > the EGO_SUM for maintainers which has made maintenance easier. However I= =20 > haven't found any new guidance yet on how to maintain go packages with=20 > the new tarball method (e.g. what needs to go into the vendor tarball,=20 > what changes are needed in ebuilds). Overall this complifates further=20 > ebuild development and verification of PRs. >=20 > In summary, IMHO the EGO_SUM way of handling of go packages has more=20 > benefits than drawbacks compared to the vendor tarballs. >=20 > Cheers, > Zoltan >=20 > [1]=20 > https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-= nightmare/ >=20 --CDbWLZStpNKdiCVt Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEABYIAB0WIQQfOU+JeXjo4uxN6vCp8he9GGIfEQUCYrlJhAAKCRCp8he9GGIf EXnuAQDLqkwFjxZdETddSQBZKl8k+bHk4K8cU4DaRmkHGDiAWQD/QoSrPhP4ssvj dZkGOwJ3zO+qai1esKchayWrB8xjNAM= =53Ms -----END PGP SIGNATURE----- --CDbWLZStpNKdiCVt--