From: "Robin H. Johnson" <robbat2@gentoo.org>
To: gentoo-dev@lists.gentoo.org
Subject: Re: [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)
Date: Tue, 4 Jul 2023 21:56:26 +0000 [thread overview]
Message-ID: <robbat2-20230704T214756-550209641Z@orbis-terrarum.net> (raw)
In-Reply-To: <1940209.tdWV9SEqCh@falbala>
[-- Attachment #1: Type: text/plain, Size: 2185 bytes --]
On Tue, Jul 04, 2023 at 12:44:39PM +0200, Gerion Entrup wrote:
> just to be curious about the whole discussion. I did not follow in the
> deepest detail but what I got is:
> - EGO_SUM blows up the Manifest file, since every little Go module needs
> to be respected. A lot of these Manifest files lead to a extremely
> increased Portage tree size. EGO_SUM is just one example (though the
> biggest one). Statically linked languages like Rust etc. have the same
> problem.
> - The current solution is to prepackage all modules, put it somewhere on
> a webserver and just manifest that file. This make the Portage tree
> small in size again, but requires a webserver/mirror and is thus
> unfriendly for overlay devs.
>
> I'm not sure if it was mentioned before but has anyone considered hash
> trees / Merkle trees for the manifest file? The idea would be to hash
> the standard manifest file a second time if it gets too big and write
> down that hash as new manifest file and leave EGO_SUM as is.
This is out-of-tree/indirect Manifests, that I proposed here, more than
a year ago:
https://marc.info/?l=gentoo-dev&m=168280762310716&w=2
https://marc.info/?l=gentoo-dev&m=165472088822215&w=2
Developing it requires PMS work in addition to package manager
development, because it introduces phases.
- primary fetch of $SRC_URI per ebuild, including indirect Manifest
- primary validation of distfiles
- secondary fetch of $SRC_URI per indirect Manifest
- secondary validation of additional distfiles
A significantly impacted use case is "emerge -f", it now needs to run
downloads twice.
The rest of the posts also go into the matter of duplication within
EGO_SUM & the indirect Manifests: limiting the growth requires some form
of content-addressed layout.
It's absolutely something we should get developed, but it's a lot of
work.
The indirect Manifests still provide a hosting challenge for overlays.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]
next prev parent reply other threads:[~2023-07-04 21:56 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <2ZKWN4KF.MKEFFMWE.LGPKYP47@RTL7EJXF.RN4PF6UF.MDFBGF3C>
[not found] ` <be450641-94ff-a0d9-51da-3a7a3abcc6c7@gentoo.org>
[not found] ` <b7309a3f-2980-b390-a16a-0518cce1da75@gentoo.org>
[not found] ` <87y1k33aoy.fsf@gentoo.org>
2023-06-30 8:15 ` [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.) Florian Schmaus
2023-06-30 8:22 ` Sam James
2023-06-30 9:38 ` Tim Harder
2023-06-30 11:33 ` Eray Aslan
2023-07-03 10:17 ` Florian Schmaus
2023-07-04 7:13 ` Tim Harder
2023-07-04 10:44 ` Gerion Entrup
2023-07-04 21:56 ` Robin H. Johnson [this message]
2023-07-04 23:09 ` Oskari Pirhonen
2023-07-05 18:40 ` Gerion Entrup
2023-07-05 19:32 ` Rich Freeman
2023-07-06 2:48 ` Oskari Pirhonen
2023-07-06 6:09 ` Zoltan Puskas
2023-07-06 19:46 ` [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open Hank Leininger
2023-07-08 20:49 ` [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.) Sam James
2023-07-03 10:17 ` Florian Schmaus
2023-07-03 11:12 ` [gentoo-dev] EGO_SUM Ulrich Mueller
2023-07-08 21:21 ` [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.) Sam James
[not found] ` <cdf5ddb7-8f65-74cf-5594-3e3eec86c915@gentoo.org>
[not found] ` <1913d3c2-5f54-acea-0ed3-930371ea1884@gentoo.org>
[not found] ` <CAAr7Pr9+zq2NV=7zhj5e+4LWOmNavCrfMstNTqkthk5uxQVNtg@mail.gmail.com>
2023-07-14 7:14 ` [gentoo-dev] Re: Flow's Manifesto and questions for nominees (was: " Florian Schmaus
2023-07-14 7:33 ` Sam James
2023-07-14 8:19 ` Sam James
2023-07-14 9:07 ` Florian Schmaus
2023-07-14 8:39 ` [gentoo-dev] Re: Flow's Manifesto and questions for nominees Ulrich Mueller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=robbat2-20230704T214756-550209641Z@orbis-terrarum.net \
--to=robbat2@gentoo.org \
--cc=gentoo-dev@lists.gentoo.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox