Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Zac Medico <zmedico@gentoo.org>]

[-- Attachment #1.1: Type: text/plain, Size: 1640 bytes --]

On 01/28/2018 02:00 PM, Andrew Barchuk wrote:
>> To the contrary, that would not remain balanced, because your
>> boundaries are entirely dependent on exactly what is in the tree at
>> the moment you run your script. Now the package manager has to perform
>> directory listing, sort and find the file name that's closest, open
>> that directory, find the next closest filename (assuming multiple
>> levels of hierarchy), and so on, or you have to store yet another
>> index that duplicates information and takes additional space. Locating
>> by distfile name hash is effectively free.
> 
> Sure, the tree won't be perfectly balanced but it will be pretty close.
> E.g. if texlive-* dominates the tree today it will likely continue
> dominating it for another 5 years. Statistical distribution of distfile
> names will likely be changing very slowly.
> 
> Doing a binary search through a list of couple of hundred of directories
> is really cheap. I don't see a reason to organize distfiles in a
> multi-level hierarchy: e.g. if the goal is to keep no more than 1000
> files in a folder than the limit of single level hierarchy is a million
> which is more than enough for foreseeable future. The list of 500
> directories takes 15kB when using full file names and will be couple of
> times smaller when using only unique prefixes.

In order to use that for distfiles mirrors, such that clients could know
where to fetch the files from, you'd need the mirror's http server to
redirect the request to the appropriate location (since the location
would not be predictable from the client side).
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]