From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 645FD1382C5 for ; Sun, 28 Jan 2018 21:18:00 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id A9E3FE0CFF; Sun, 28 Jan 2018 21:17:54 +0000 (UTC) Received: from mail-lf0-x233.google.com (mail-lf0-x233.google.com [IPv6:2a00:1450:4010:c07::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 35239E0CAF for ; Sun, 28 Jan 2018 21:17:54 +0000 (UTC) Received: by mail-lf0-x233.google.com with SMTP id t139so7138399lff.0 for ; Sun, 28 Jan 2018 13:17:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=4dwSQIQQ0QRQfOGxBAFEpwS/oqWPQRr6p+F++/FSFB0=; b=d6Sx7koTYUHZqnWP72wfQF4WLXUlBIuh98bRA0f6g+VVwibKt1wAAW5glp+606yU6y cFcbT/z7PKo0P44I5znmKkJaTSqjMgyOE2LZIAV8i/d/0M+UFxPGS2zraVkTinmJRJHW W6uj6U2OJu1NRT6UpDeC+YTS5qe3UUCfbnUPKVFc2WcUkGHMZKlttMeRvIoI1iXFCjge Z8gTuQp2gkhxvfCmh/CyqI9tCEoO4FQBizf31ODx5vvD+LcuFoSsrLsUUf36Lq66WZ0c eP0twrFpzkPXlyz7/AwqB+mS98ngwtqA2aBqKcFfXiz1doxIBOUbUsiBw8mq7jjY+EMP mxng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=4dwSQIQQ0QRQfOGxBAFEpwS/oqWPQRr6p+F++/FSFB0=; b=f0AvsvaA8kgrDD6MCv6lz7eFs++YoBstNfMH8MfGZhom9mzJJ0ICFubjExAoViqn/k tBj7utqa4rMwQbix95nZRZfG2foEekjFAOx+N8pWxiGvc7y+K69B2VQPGSHuq7NguhUs +9f/kEHuFaPhoABVyIJZQviKtF0OX9otj+hiZrOqNvh9U74Q47TBNceqPUIRKtOndztN Jh1RoJYpdCWsMWkd82B8WWFs/dUNVEEhj/wwpzGykTRsIfUSB0b1XeFefT7Jf+w8wdTx hr6IqOdS/RsC4UtWUaO17C64maJlywDimX31tZ2mu50Z08yklR1S6TfCnQy7gueXQWbj BUzw== X-Gm-Message-State: AKwxyte4h47NLlB65vlvJf7hoGRAGa5itNJgidEqKNW73iOeDlkUQKFL yfNU0WHKYZnFhGg/rh9PZoRLmiS2tTVkaDcGPXsdzQ== X-Google-Smtp-Source: AH8x227PhLG2lVdkOsxkbUWen6ytCEX+S1w/dU5EcWTPDY7xSqVt9MbY0QYE+2VGrL0K2daG10u0hX34cSOGEeFY6RA= X-Received: by 10.25.17.227 with SMTP id 96mr12696203lfr.85.1517174271445; Sun, 28 Jan 2018 13:17:51 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Received: by 10.46.58.8 with HTTP; Sun, 28 Jan 2018 13:17:10 -0800 (PST) In-Reply-To: <1517172228.2114973.1251027256.0A9C8F3C@webmail.messagingengine.com> References: <1517009079.31015.3.camel@gentoo.org> <1517172228.2114973.1251027256.0A9C8F3C@webmail.messagingengine.com> From: Gordon Pettey Date: Sun, 28 Jan 2018 15:17:10 -0600 Message-ID: Subject: Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure To: gentoo-dev@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" X-Archives-Salt: d9cb3485-23f0-473b-b986-43527c885185 X-Archives-Hash: b62c0d74fc66edccf96bcc899516d0f4 On Sun, Jan 28, 2018 at 2:43 PM, Andrew Barchuk wrote: > There's another option to use character ranges for each directory > computed in a way to have the files distributed evenly. One way to do > that is to use filename prefix of dynamic length so that each range > holds the same number of files. E.g. we would have Ab/, Ap/, Ar/ but > texlive-module-te/, texlive-module-th/, texlive-module-ti/. A similar > but simpler option is to use file names as range bounds (the same way > dictionaries use words to demarcate page bounds): each directory will > have a name of the first file located inside. This way files will be > distributed evenly and it's still easy to pick a correct directory where > a file will be located manually. > > ...snip... > > Using the approach above the files will distributed evenly among the > directories keeping the possibility to determine the directory for a > specific file by hand. It's possible if necessary to keep the directory > structure unchanged for very long time and it will likely stay > well-balanced. Picking a directory for a file is very cheap. The only > obvious downside I see is that it's necessary to know list of > directories to pick the correct one (can be mitigated by caching the > list of directories if important). If it's desirable to make directory > names shorter or to look less like file names it's fairly easy to > achieve by keeping only unique prefixes of directories. For example: To the contrary, that would not remain balanced, because your boundaries are entirely dependent on exactly what is in the tree at the moment you run your script. Now the package manager has to perform directory listing, sort and find the file name that's closest, open that directory, find the next closest filename (assuming multiple levels of hierarchy), and so on, or you have to store yet another index that duplicates information and takes additional space. Locating by distfile name hash is effectively free.