Re: [gentoo-dev] Proposal to undeprecate EGO_SUM

[Oskari Pirhonen <xxc3ncoredxx@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5879 bytes --]

On Mon, Jun 27, 2022 at 01:43:19 +0200, Zoltan Puskas wrote:
> Hi,
> 
> I've been working on adding a go based ebuild to Gentoo yesterday and I 
> got this warning form portage saying that EGO_SUM is deprecated and 
> should be avoided. Since I remember there was an intense discussion 
> about this on the ML I went back and have re-read the threads before 
> writing this piece. I'd like to provide my perspective as user, a 
> proxied maintainer, and overlay owner. I also run a private mirror on my 
> LAN to serve my hosts in order to reduce load on external mirrors.
> 
> Before diving in I think it's worth reading mgorny's blog post "The 
> modern packager’s security nightmare"[1] as it's relevant to the 
> discussion, and something I deeply agree with.
> 
> With all that being said, I feel that the tarball idea is a bad due to 
> many reasons.
> 
>  From security point of view, I understand that we still have to trust 
> maintainers not to do funky stuff, but I think this issue goes beyond 
> that.
> 
> First of all one of the advantages of Gentoo is that it gets it's source 
> code from upstream (yes, I'm aware of mirrors acting as a cache layer), 
> which means that poisoning source code needs to be done at upstream 
> level (effectively means hacking GitHub, PyPi, or some standalone 
> project's Gitea/cgit/gitlab/etc. instance or similar), sources which 
> either have more scrutiny or have a limited blast radius.
> 
> Additionally if an upstream dependency has a security issue it's easier 
> to scan all EGO_SUM content and find packages that potentially depend on 
> a broken dependency and force a re-pinning and rebuild. The tarball 
> magic hides this completely and makes searching very expensive.
> 
> In fact using these vendor tarballs is the equivalent of "static 
> linking" in the packaging space. Why are we introducing the same issue 
> in the repository space? This kills the reusability of already 
> downloaded dependencies and bloats storage requirements. This is 
> especially bad on laptops, where SSD free space might be limited, in 
> case the user does not nuke their distfiles after each upgrade.
> 
> Considering that BTRFS (and possibly other filesystems) support on the 
> fly compression the physical cost of a few inflated ebuilds and 
> Manifests is actually way smaller than the logical size would indicate. 
> Compare that to the huge incompressible tarballs that now we need to 
> store.
> 
> As a proxied maintainer or overlay owner hosting these huge tarballs 
> also becomes problem (i.e. we need some public space with potentially 
> gigabytes of free space and enough bandwidth to push that to users). 
> Pushing toward vendor tarballs creates an extra expense on every level 
> (Gentoo infra, mirrors, proxy maintainers, overlay owners, users).
> 
> If bloating portage is a big issue and we frown upon go stuff anyway (or 
> only a few users need these packages), why not consider moving all go 
> packages into an officially supported go packages only overlay? I 
> understand that this would not solve the kernel buffer issue where we 
> run out of environment variable space, but it would debloat the main 
> portage tree.
> 

Rephrasing this just to ensure I'm understanding it correctly: you're
suggesting to move _everything_ that uses Go into its own overlay. Let's
call it gentoo-go for the sake of the example.

If the above is accurate, then I hard disagree.

The biggest package that I have that uses Go is docker (and accompanying
tools). Personal distaste of docker aside, it's a very popular piece of
software, and I don't think it's fair to require all the people who want
to use it to first enable and sync gentoo-go before they can install it.

And what about transitive dependencies? Suppose app-misc/cool-package is
written in some language that isn't Go, but it has a dependency on
sys-apps/cool-util which has a dependency on something written in Go.
Should a user wanting to install cool-package have to enable the
gentoo-go overlay now too? Even though app-misc/cool-package would look
like it doesn't need the overlay unless you dig into the deps.

Not a dev, just a user who really likes Gentoo :)

- Oskari

> It also breaks reproducibility. With EGO_SUM I can check out an older 
> version of portage tree (well to some extent) and rebuild packages since 
> dependency upstream is very likely to host old versions of their source. 
> With the tarballs this breaks since as soon as an ebuild is dropped from 
> mainline portage the vendor tarballs follow them too. There is no way 
> for the user to roll back a package a few weeks back (e.g. if new 
> version has bugs), unlike with EGO_SUM.
> 
> In fact I feel this goes against the spirit of portage too, since now 
> instead of "just describing" how to obtain sources and build them, now 
> it now depends on essentially ephemeral blobs, which happens to be 
> externalized from the portage tree itself. I'm aware that we have 
> ebuilds that pull in patches and other stuff from dev space already, but 
> we shouldn't make this even worse.
> 
> Finally with EGO_SUM we had a nice tool get-ego-vendor which produced 
> the EGO_SUM for maintainers which has made maintenance easier. However I 
> haven't found any new guidance yet on how to maintain go packages with 
> the new tarball method (e.g. what needs to go into the vendor tarball, 
> what changes are needed in ebuilds). Overall this complifates further 
> ebuild development and verification of PRs.
> 
> In summary, IMHO the EGO_SUM way of handling of go packages has more 
> benefits than drawbacks compared to the vendor tarballs.
> 
> Cheers,
> Zoltan
> 
> [1] 
> https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nightmare/
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]