From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 6C7F3138334 for ; Sat, 19 Oct 2019 23:57:19 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 517B0E0922; Sat, 19 Oct 2019 23:57:15 +0000 (UTC) Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 0D9BEE08DA for ; Sat, 19 Oct 2019 23:57:14 +0000 (UTC) Received: by mail-il1-x12d.google.com with SMTP id u1so8792196ilq.12 for ; Sat, 19 Oct 2019 16:57:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gentoo-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=TxhvctNhyOjgzVYYp/igYrwNuUHoPO1fUNcbUC7A490=; b=HuEoKDqXd3FGQEpwdkjHKKO4+SpoffzQ6iGf0VKwb5+wT0k+VkTIoVNLlyGhtXugP/ tnERW0d6gZj4EntQI8gt3KLhDBuPk8BdeGEXOj5ONXKhoZr0XDqA3ipPniEz3AD9tplf OiuBR97CrFkeLqTKh+P4Wf4GaSw5C8WlXnovRaex6g3xVHHzir05l8HcwB0fMa8ph+k9 2jTGvp41uq8YW8fUPiplbCHL9m3mxG05K06VT2Ku8uGw7x0QjFwcNeT81cGyCkmBqCIy mR7K3gsq1aaeuNhRHPwARaJoTc69eAyeKMnlO9rIlou2ufsgk0P2KqfKPvMh4GIWcIW8 Bvjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=TxhvctNhyOjgzVYYp/igYrwNuUHoPO1fUNcbUC7A490=; b=oNg7ai/H7eT1gD2I11XsCAriOdUNrtxtMnr6SZtYRRFs5e/dhqhrTOM5ZrVmm0fQss hwgneFTshPF66wUU6cFYwceLoxkyh8OvRR0ve5DcllSsekUzsC0oOATipuUKpiKKWLnN /zdyXl9OVDCpfOio8pMoX90zsZLvzPBH5m8/OPmpBgcD7Sw9dl8X1M2VSmQcoayHIv5/ tZF7CibGxDsBxIeyGgULz7jBli7kHwemb4Bm3zy/eFVN3RXPiHxLneZD1WplLoOGrWdj mr269Ksm8ZxTbg5JEREPtLua1fKD1w1g7HnC2jTxIPO9EwOKEfsJS1wYnVTq15HsXEfa MoTA== X-Gm-Message-State: APjAAAWc5b9HvW01hiAF8frEHY+MIF8JCOEPpNTLvxyYy74iD5zR1LvI DlYJej8eXW7ur8oOf6ycIYJ7txqEryqVyiWfL28OxdaP X-Google-Smtp-Source: APXvYqzDaK9YSwKCVbfhYJcohJHOS+tvt45l4ceqoLlLBiQPVfgX8MqWxRv/fTte/amk3Do5GJQ1RWyyL6LTMisFols= X-Received: by 2002:a92:8189:: with SMTP id q9mr19488834ilk.38.1571529433778; Sat, 19 Oct 2019 16:57:13 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <4c7465824f1fb69924c826f6bbe3ee73afa08ec8.camel@gentoo.org> <2d15507e-98ad-9466-75b7-7e8268ef2eb9@gentoo.org> In-Reply-To: <2d15507e-98ad-9466-75b7-7e8268ef2eb9@gentoo.org> From: Alec Warner Date: Sat, 19 Oct 2019 16:57:02 -0700 Message-ID: Subject: Re: [gentoo-dev] New distfile mirror layout To: Gentoo Dev Content-Type: multipart/alternative; boundary="000000000000a79e9905954c35de" X-Archives-Salt: 741f24d4-2819-4a77-96fe-9a3680114700 X-Archives-Hash: 2778356a448e3a2868b97f659bba7b5d --000000000000a79e9905954c35de Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Oct 19, 2019 at 4:24 PM Joshua Kinard wrote: > On 10/18/2019 09:41, Micha=C5=82 G=C3=B3rny wrote: > > Hi, everybody. > > > > It is my pleasure to announce that yesterday (EU) evening we've switche= d > > to a new distfile mirror layout. Users will be switching to the new > > layout either as they upgrade Portage to 2.3.77 or -- if they upgraded > > already -- as their caches expire (24hrs). > > > > The new layout is mostly a bow towards mirror admins, for some of whom > > having a 60000+ files in a single directory have been a problem. > > However, I suppose some of you also found e.g. the directory index > > hardly usable due to its size. > > > > Throughout a transitional period (whose exact length hasn't been decide= d > > yet), both layouts will be available. Afterwards, the old layout will > > be removed from mirrors. This has a few implications: > > > > 1. Users who don't upgrade their package managers in time will lose > > the ability of fetching from Gentoo mirrors. This shouldn't be that > > much of a problem given that the core software needed to upgrade Portag= e > > should all have reliable upstream SRC_URIs. > > > > 2. mirror://gentoo/file URIs will stop working. While technically you > > could use mirror://gentoo/XX/file, I'd rather recommend finally > > discarding its usage and moving distfiles to devspace. > > > > 3. Directly fetching files from distfiles.gentoo.org will become > > a little harder. To fetch a distfile named 'foo-1.tar.gz', you'd have > > to use something like: > > > > $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2 > > 1b > > $ wget http://distfiles.gentoo.org/distfiles/1b/foo-1.tar.gz > > ... > > > > > > Alternatively, you can: > > > > $ wget http://distfiles.gentoo.org/distfiles/INDEX > > > > and grep for the right path there. This INDEX is also a more > > lightweight alternative to HTML indexes generated by the servers. > > > > > > If you're interested in more background details and some plots, see [1]= . > > > > [1] > https://dev.gentoo.org/~mgorny/articles/improving-distfile-mirror-structu= re.html > > > > So the answer I didn't really see directly stated here is, where do new > distfiles need to go //now//? E.g., if on woodpecker, I currently cp a > distfile to /space/distfiles-local. What is the new directory I need to > use? And if mirror://gentoo/${FOO} is going away, for the new distfiles > target, what would be the applicable prefix to use? > > > Directly using devspace seems like a bad idea, IMHO. Once long ago, we a= ll > got chastised for doing exactly that. Too much possibility of > fragmentation > as devs retire or package maintainership changes hands. > > I looked at the whitepaper'ish-like writeup, and I kinda don't like using= a > hash-based naming scheme on the new distfiles layout. I really kind pref= er > breaking the directories up based on the first letter of the distfiles in > question, factoring case-sensitivity in (so you'd have 52 top-level > directories for A-Z and a-z, plus 10 more for 0-9). Under each of those > directories, additional subdirectories for the next few letters (say, > letters 2-3). Yes, this leads to some orphan cases where a distfile migh= t > live on its own, but from a direct navigation standpoint, it's easy to fi= nd > for someone browsing the distfiles server and easy to predict where a > distfile is at. > > No math, statistical analysis, or deep-rooted knowledge of filesystems > behind that paragraph. Just a plain old unfiltered opinion. Sometimes, = I > need to go get a distfile off the Gentoo mirrors, and being able to quick= ly > find it in the mirror root is great. Having to do hash calculations to > work > out the file path will be *really* annoying. > So if you want a tool that "downloads a distfile off of the mirrors" we should be able to build such a utility. I'm not really sure why that tool needs to be: *copy DISTFILENAME* wget distilfes.gentoo.org/$PASTE It could just `ebuild portageq download $DISTFILENAME or similar.` -A > > -- > Joshua Kinard > Gentoo/MIPS > kumba@gentoo.org > rsa6144/5C63F4E3F5C6C943 2015-04-27 > 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 > > "The past tempts us, the present confuses us, the future frightens us. A= nd > our lives slip away, moment by moment, lost in that vast, terrible > in-between." > > --Emperor Turhan, Centauri Republic > > --000000000000a79e9905954c35de Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sat, Oct 19, 2019 at 4:24 PM Joshu= a Kinard <kumba@gentoo.org> w= rote:
On 10/18/2= 019 09:41, Micha=C5=82 G=C3=B3rny wrote:
> Hi, everybody.
>
> It is my pleasure to announce that yesterday (EU) evening we've sw= itched
> to a new distfile mirror layout.=C2=A0 Users will be switching to the = new
> layout either as they upgrade Portage to 2.3.77 or -- if they upgraded=
> already -- as their caches expire (24hrs).
>
> The new layout is mostly a bow towards mirror admins, for some of whom=
> having a 60000+ files in a single directory have been a problem.
> However, I suppose some of you also found e.g. the directory index
> hardly usable due to its size.
>
> Throughout a transitional period (whose exact length hasn't been d= ecided
> yet), both layouts will be available.=C2=A0 Afterwards, the old layout= will
> be removed from mirrors.=C2=A0 This has a few implications:
>
> 1. Users who don't upgrade their package managers in time will los= e
> the ability of fetching from Gentoo mirrors.=C2=A0 This shouldn't = be that
> much of a problem given that the core software needed to upgrade Porta= ge
> should all have reliable upstream SRC_URIs.
>
> 2. mirror://gentoo/file URIs will stop working.=C2=A0 While technicall= y you
> could use mirror://gentoo/XX/file, I'd rather recommend finally > discarding its usage and moving distfiles to devspace.
>
> 3. Directly fetching files from distfiles.gentoo.org will become=
> a little harder.=C2=A0 To fetch a distfile named 'foo-1.tar.gz'= ;, you'd have
> to use something like:
>
> $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2
> 1b
> $ wget http://distfiles.gentoo.org/distfil= es/1b/foo-1.tar.gz
> ...
>
>
> Alternatively, you can:
>
> $ wget http://distfiles.gentoo.org/distfiles/INDEX
>
> and grep for the right path there.=C2=A0 This INDEX is also a more
> lightweight alternative to HTML indexes generated by the servers.
>
>
> If you're interested in more background details and some plots, se= e [1].
>
> [1] https://dev= .gentoo.org/~mgorny/articles/improving-distfile-mirror-structure.html >

So the answer I didn't really see directly stated here is, where do new=
distfiles need to go //now//?=C2=A0 E.g., if on woodpecker, I currently cp = a
distfile to /space/distfiles-local.=C2=A0 What is the new directory I need = to
use?=C2=A0 And if mirror://gentoo/${FOO} is going away, for the new distfil= es
target, what would be the applicable prefix to use?

=C2=A0

Directly using devspace seems like a bad idea, IMHO.=C2=A0 Once long ago, w= e all
got chastised for doing exactly that.=C2=A0 Too much possibility of fragmen= tation
as devs retire or package maintainership changes hands.

I looked at the whitepaper'ish-like writeup, and I kinda don't like= using a
hash-based naming scheme on the new distfiles layout.=C2=A0 I really kind p= refer
breaking the directories up based on the first letter of the distfiles in question, factoring case-sensitivity in (so you'd have 52 top-level
directories for A-Z and a-z, plus 10 more for 0-9).=C2=A0 Under each of tho= se
directories, additional subdirectories for the next few letters (say,
letters 2-3).=C2=A0 Yes, this leads to some orphan cases where a distfile m= ight
live on its own, but from a direct navigation standpoint, it's easy to = find
for someone browsing the distfiles server and easy to predict where a
distfile is at.

No math, statistical analysis, or deep-rooted knowledge of filesystems
behind that paragraph.=C2=A0 Just a plain old unfiltered opinion.=C2=A0 Som= etimes, I
need to go get a distfile off the Gentoo mirrors, and being able to quickly=
find it in the mirror root is great.=C2=A0 Having to do hash calculations t= o work
out the file path will be *really* annoying.

So if you want a tool that "downloads a distfile off of the mirr= ors" we should be able to build such a utility.

I'm not really sure why that tool needs to be:
*copy DISTF= ILENAME*

It could just `ebu= ild portageq download $DISTFILENAME or similar.`

-= A




=C2= =A0

--
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org<= br> rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.= =C2=A0 And
our lives slip away, moment by moment, lost in that vast, terrible in-betwe= en."

--Emperor Turhan, Centauri Republic

--000000000000a79e9905954c35de--