On Tue, Jul 04, 2023 at 12:44:39PM +0200, Gerion Entrup wrote:
> just to be curious about the whole discussion. I did not follow in the
> deepest detail but what I got is:
> - EGO_SUM blows up the Manifest file, since every little Go module needs
>   to be respected. A lot of these Manifest files lead to a extremely
>   increased Portage tree size. EGO_SUM is just one example (though the
>   biggest one). Statically linked languages like Rust etc. have the same
>   problem.
> - The current solution is to prepackage all modules, put it somewhere on
>   a webserver and just manifest that file. This make the Portage tree
>   small in size again, but requires a webserver/mirror and is thus
>   unfriendly for overlay devs.
> 
> I'm not sure if it was mentioned before but has anyone considered hash
> trees / Merkle trees for the manifest file? The idea would be to hash
> the standard manifest file a second time if it gets too big and write
> down that hash as new manifest file and leave EGO_SUM as is.
This is out-of-tree/indirect Manifests, that I proposed here, more than
a year ago:
https://marc.info/?l=gentoo-dev&m=168280762310716&w=2
https://marc.info/?l=gentoo-dev&m=165472088822215&w=2

Developing it requires PMS work in addition to package manager
development, because it introduces phases.

- primary fetch of $SRC_URI per ebuild, including indirect Manifest
- primary validation of distfiles
- secondary fetch of $SRC_URI per indirect Manifest
- secondary validation of additional distfiles

A significantly impacted use case is "emerge -f", it now needs to run
downloads twice.

The rest of the posts also go into the matter of duplication within
EGO_SUM & the indirect Manifests: limiting the growth requires some form
of content-addressed layout.

It's absolutely something we should get developed, but it's a lot of
work.

The indirect Manifests still provide a hosting challenge for overlays.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136