From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id C3E6715800D for ; Wed, 5 Jul 2023 19:32:28 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 8881DE0894; Wed, 5 Jul 2023 19:32:25 +0000 (UTC) Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 60162E085E for ; Wed, 5 Jul 2023 19:32:25 +0000 (UTC) Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-bb3a77abd7bso7927976276.0 for ; Wed, 05 Jul 2023 12:32:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688585544; x=1691177544; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S7ylYMA2baOoT7WLTs7X7locRWMxIfPukIzzSsrPVFM=; b=UEKXFwFG3j0UVpZuVBkLXwZ7xIgbrSMuuJ9Zl0ZRD+Z0t3KSoqLw2vL168AEQWByp8 AmZdAHEm8eN9cZmPUeZvCwVSYH6KSklhJHBQrYBzFQx2oGIUhYLVra07HKk9DuO4UOvM ZSmZXzV7uKgQsR4BPf8GLBqUb+YxpDlRGABJGtlZzxP4YlN8M1RASalIIopog9OfBxSm fkjBSuIZjosAZi0lcmYqjLzsvjLXaJ5e8Y4Vo1BR/NV1g6Sko19OuSDViYvPANgPRQSQ VJGycEVSf5jNppu2zuq4rTGRbjspQGm7BDhmTGpjbjhUaOo9WCpjF3tr+3Ctp+N1zjgu y3vg== X-Gm-Message-State: ABy/qLZjAnVrxf7T2ZOgw3JQxrzJIdZQ1ChgGt/AndaMzrl9Skm4sshg 73nd1XinMbJcSJWed378YwV0yyOpUMy/fw4TLblE0MNt X-Google-Smtp-Source: APBJJlH+KdMRN31tasexVt/ZH/kc45SHZc7bT59HUIpL5YtZldIE0Bnm6nLZ13meQw0RG/Rs5fSZ/gthnVCx2cXCH4k= X-Received: by 2002:a25:a8a:0:b0:bcb:9b43:5a89 with SMTP id 132-20020a250a8a000000b00bcb9b435a89mr15051472ybk.61.1688585544507; Wed, 05 Jul 2023 12:32:24 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <2ZKWN4KF.MKEFFMWE.LGPKYP47@RTL7EJXF.RN4PF6UF.MDFBGF3C> <2243341.Icojqenx9y@falbala> In-Reply-To: <2243341.Icojqenx9y@falbala> From: Rich Freeman Date: Wed, 5 Jul 2023 15:32:15 -0400 Message-ID: Subject: Re: [gentoo-dev] EGO_SUM (was: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.) To: gentoo-dev@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 03b9480e-1993-4f5a-826b-2168ccfe052a X-Archives-Hash: 7864c683683d7a42663d28299e25777c On Wed, Jul 5, 2023 at 2:40=E2=80=AFPM Gerion Entrup wrote: > > Am Mittwoch, 5. Juli 2023, 01:09:30 CEST schrieb Oskari Pirhonen: > > On Tue, Jul 04, 2023 at 21:56:26 +0000, Robin H. Johnson wrote: > > > > > > Developing it requires PMS work in addition to package manager > > > development, because it introduces phases. > > > > > > - primary fetch of $SRC_URI per ebuild, including indirect Manifest > > > - primary validation of distfiles > > > - secondary fetch of $SRC_URI per indirect Manifest > > > - secondary validation of additional distfiles > > > > > > A significantly impacted use case is "emerge -f", it now needs to run > > > downloads twice. > > > > I'm not sure double downloading is required. Consider a flow similar to > > this: > > > > 1. distfiles are fetched as per the ebuild > > 2. distfiles are hashed into a temporary Manifest > > 3. temporary Manifest is hashed and compared with the hashes stored in > > the in-tree Manifest for the direct Manifest > > This is exactly, what I meant. A webstorage is not needed. A second > download process is also not needed. Just an additional Manifest format > is needed for ebuilds with more than n distfiles. > I suspect that Robin was proposing indirect manfests AND src uris, and not just indirect manifests. In any case, if he wasn't, then I'd suggest it would make sense to have that so that we don't need giant lists of src_uris or go sums or whatever in ebuilds. Sure, the manifests are even larger than the original file references, but those will still be long. Plus if a file is used by 5 versions of an ebuild it will be present in the manifests once per hash function, but in the ebuilds 5 times. I agree though that if only the manifests are moved to a fetched file then you could fetch that on the first pass, though you'd still need the extra logic to parse it. I'm not sure it really is much of a difference to the effort involved. Aren't go sums already content hashes? It might make even more sense to create some kind of modular manifest verification logic in portage so that the same eclass that handles EGO_SUM could tell the package manager how to check the integrity of the files that are fetched. Well, assuming we trust whatever hash function they're using (I'm afraid to check - maybe this isn't such a great idea...). --=20 Rich