From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 4EDAE15800D for ; Tue, 4 Jul 2023 21:56:32 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id DAE88E081A; Tue, 4 Jul 2023 21:56:27 +0000 (UTC) Received: from smtp.gentoo.org (mail.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 9B6AAE07E6 for ; Tue, 4 Jul 2023 21:56:27 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id F1B34340F58 for ; Tue, 4 Jul 2023 21:56:26 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) by grubbs.orbis-terrarum.net (Postfix) with ESMTP id 752E1264AEF for ; Tue, 4 Jul 2023 21:56:26 +0000 (UTC) Received: (qmail 27061 invoked by uid 10000); 4 Jul 2023 21:56:26 -0000 Date: Tue, 4 Jul 2023 21:56:26 +0000 From: "Robin H. Johnson" To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.) Message-ID: References: <2ZKWN4KF.MKEFFMWE.LGPKYP47@RTL7EJXF.RN4PF6UF.MDFBGF3C> <52703145-a284-30f3-aac8-69ed086a5f4a@gentoo.org> <1940209.tdWV9SEqCh@falbala> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="yzSqXaAbrpdTY9Bv" Content-Disposition: inline In-Reply-To: <1940209.tdWV9SEqCh@falbala> X-Archives-Salt: 0ff9e76d-c396-4357-9ea1-3fcbda3a513d X-Archives-Hash: 5f1c9d8ef206d8b80685153a690c99d3 --yzSqXaAbrpdTY9Bv Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 04, 2023 at 12:44:39PM +0200, Gerion Entrup wrote: > just to be curious about the whole discussion. I did not follow in the > deepest detail but what I got is: > - EGO_SUM blows up the Manifest file, since every little Go module needs > to be respected. A lot of these Manifest files lead to a extremely > increased Portage tree size. EGO_SUM is just one example (though the > biggest one). Statically linked languages like Rust etc. have the same > problem. > - The current solution is to prepackage all modules, put it somewhere on > a webserver and just manifest that file. This make the Portage tree > small in size again, but requires a webserver/mirror and is thus > unfriendly for overlay devs. >=20 > I'm not sure if it was mentioned before but has anyone considered hash > trees / Merkle trees for the manifest file? The idea would be to hash > the standard manifest file a second time if it gets too big and write > down that hash as new manifest file and leave EGO_SUM as is. This is out-of-tree/indirect Manifests, that I proposed here, more than a year ago: https://marc.info/?l=3Dgentoo-dev&m=3D168280762310716&w=3D2 https://marc.info/?l=3Dgentoo-dev&m=3D165472088822215&w=3D2 Developing it requires PMS work in addition to package manager development, because it introduces phases. - primary fetch of $SRC_URI per ebuild, including indirect Manifest - primary validation of distfiles - secondary fetch of $SRC_URI per indirect Manifest - secondary validation of additional distfiles A significantly impacted use case is "emerge -f", it now needs to run downloads twice. The rest of the posts also go into the matter of duplication within EGO_SUM & the indirect Manifests: limiting the growth requires some form of content-addressed layout. It's absolutely something we should get developed, but it's a lot of work. The indirect Manifests still provide a hosting challenge for overlays. --=20 Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 --yzSqXaAbrpdTY9Bv Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it. iQKTBAABCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAmSklYhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsTy6A/8DuwD+3qt9aLCme2UZnHRJUuvSY8PDks5HWHFofiYEVjK/vpeKfkjwL8c lUrXibyDv0mqo4qNFMTK2q85D+l6Cllmy2JXUsPAi0BfXhNYca73vkFayTBRl+zi dyPA4ropRXEVp1HSdLqt6qVk1rZWM/6hKLHx3tSuD82buShFdrruGMQBttfPL32/ Y6yee5IjVmxj+87Pa43dayAMrjdMOyfzEtM1R882bgsJeWZb/dtS8xs8jrJZm7Vr KDmidam3HTn+4NJZyrPxa5SPBxJYlDFVWZ1O8sNTLcrNXW0T/ewLV8IgHo/t+P02 slc0bgXzUuKWxAUTiU0gnJVCWIff4gus8u9Vt+jrDQ8PSAwsulEadhA0e1VPHNva +lN96nz6PE8S4l1XRjQ7u2SI8aub1+BlL3dXe9rxDXDScz6CGFGFh4W87xwQZLyB RUWkYVKSLojRKbEfB0AIlDSlqEQvRAWBK4tNA03x9cAlwmUoEU3Z4MCf7WHh9tRL xYu8YuNTBjaZ5pS28UavRDIdeDJW8DQIBpUvY0EII2mFH6DlnpxutVjWBrlihbxs 98yRq5a56kJBoyT97yFlatvjdkewFXbycZbgJ2S6o0r0U8TZdQinjODjBIx633wv MHlsZg7WoOLNoKYYFSTa2z0E5fkf8vtwnadz+l8lli9S/1B9oLU= =PRm9 -----END PGP SIGNATURE----- --yzSqXaAbrpdTY9Bv--