From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 01941138334 for ; Fri, 18 Oct 2019 19:53:50 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 80EB1E0922; Fri, 18 Oct 2019 19:53:46 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 3477CE08AA; Fri, 18 Oct 2019 19:53:45 +0000 (UTC) Received: from [192.168.5.101] (pool-96-232-115-28.nycmny.fios.verizon.net [96.232.115.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: ryao) by smtp.gentoo.org (Postfix) with ESMTPSA id 4256034BD6F; Fri, 18 Oct 2019 19:53:44 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Richard Yao Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply Mime-Version: 1.0 (1.0) Subject: Re: [gentoo-dev] New distfile mirror layout Date: Fri, 18 Oct 2019 15:53:41 -0400 Message-Id: References: <4c7465824f1fb69924c826f6bbe3ee73afa08ec8.camel@gentoo.org> Cc: gentoo-dev-announce In-Reply-To: <4c7465824f1fb69924c826f6bbe3ee73afa08ec8.camel@gentoo.org> To: gentoo-dev@lists.gentoo.org X-Mailer: iPhone Mail (17A860) X-Archives-Salt: 9a882b0c-8efb-4a74-ac11-572985736b3b X-Archives-Hash: 617cb06997486386336eb27425ed4065 > On Oct 18, 2019, at 9:42 AM, Micha=C5=82 G=C3=B3rny wr= ote: >=20 > =EF=BB=BFHi, everybody. >=20 > It is my pleasure to announce that yesterday (EU) evening we've switched > to a new distfile mirror layout. Users will be switching to the new > layout either as they upgrade Portage to 2.3.77 or -- if they upgraded > already -- as their caches expire (24hrs). >=20 > The new layout is mostly a bow towards mirror admins, for some of whom > having a 60000+ files in a single directory have been a problem.=20 > However, I suppose some of you also found e.g. the directory index > hardly usable due to its size. This sounds like a filesystem issue. Do we know which filesystems are suffer= ing? ZFS should be fine. I believe ext2/ext3 have problems with this many files. e= xt4 is probably okay, but don=E2=80=99t quote me on that. >=20 > Throughout a transitional period (whose exact length hasn't been decided > yet), both layouts will be available. Afterwards, the old layout will > be removed from mirrors. This has a few implications: >=20 > 1. Users who don't upgrade their package managers in time will lose > the ability of fetching from Gentoo mirrors. This shouldn't be that > much of a problem given that the core software needed to upgrade Portage > should all have reliable upstream SRC_URIs. >=20 > 2. mirror://gentoo/file URIs will stop working. While technically you > could use mirror://gentoo/XX/file, I'd rather recommend finally > discarding its usage and moving distfiles to devspace. >=20 > 3. Directly fetching files from distfiles.gentoo.org will become > a little harder. To fetch a distfile named 'foo-1.tar.gz', you'd have > to use something like: >=20 > $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2 > 1b > $ wget http://distfiles.gentoo.org/distfiles/1b/foo-1.tar.gz > ... >=20 >=20 > Alternatively, you can: >=20 > $ wget http://distfiles.gentoo.org/distfiles/INDEX >=20 > and grep for the right path there. This INDEX is also a more > lightweight alternative to HTML indexes generated by the servers. >=20 >=20 > If you're interested in more background details and some plots, see [1]. >=20 > [1] https://dev.gentoo.org/~mgorny/articles/improving-distfile-mirror-stru= cture.html >=20 > --=20 > Best regards, > Micha=C5=82 G=C3=B3rny >=20