Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure

[Gordon Pettey <petteyg359@gmail.com>]

Why not use a hash of the file name instead of its contents? That
seems like it would be much simpler, and that's not going to reduce
the output space for balance...

On Sat, Jan 27, 2018 at 5:41 AM, Michał Górny <mgorny@gentoo.org> wrote:
> W dniu sob, 27.01.2018 o godzinie 11∶36 +0000, użytkownik Roy Bamford
> napisał:
>> On 2018.01.27 08:30, Michał Górny wrote:
>> > W dniu pią, 26.01.2018 o godzinie 20∶48 -0500, użytkownik Michael
>> > Orlitzky napisał:
>> > > On 01/26/2018 06:24 PM, Michał Górny wrote:
>> > > >
>> > > > The alternate option of using file hash has the advantage of
>> >
>> > having
>> > > > a more balanced split.  Furthermore, since hashes are stored
>> > > > in Manifests using them is zero-cost.  However, this solution has
>> >
>> > two
>> > > > significant disadvantages:
>> > > >
>> > > > 1. The hash values are unknown for newly-downloaded distfiles, so
>> > > >    ``repoman`` (or an equivalent tool) would have to use a
>> >
>> > temporary
>> > > >    directory before locating the file in appropriate subdirectory.
>> > > >
>> > > > 2. User-provided distfiles (e.g. for fetch-restricted packages)
>> >
>> > with
>> > > >    hash mismatches would be placed in the wrong subdirectory,
>> > > >    potentially causing confusing errors.
>> > > >
>> > >
>> > > The filename proposal sounds fine, so this is only academic, but:
>> >
>> > are
>> > > these two points really disadvantages?
>> > >
>> > > What are we worried about in using a temporary directory? Copying
>> >
>> > across
>> > > filesystem boundaries? Except in rare cases, $DISTDIR itself will be
>> > > usable a temporary location (on the same filesystem), won't it?
>> >
>> > Why add the extra complexity when there's no need for one? Note that
>> > there's also the problem of resuming transfers, so in the end we're
>> > talking about permanent temporary directory where we keep unfinished
>> > transfers.
>> >
>> > > For the second point, portage is going to tell me where to put the
>> >
>> > file,
>> > > isn't it? Then no matter what garbage I download, won't portage look
>> >
>> > for
>> > > it in the right place, because where-to-put-it is determined using
>> >
>> > the
>> > > same manifest hash that determines where-to-find-it?
>> >
>> > No, it won't. Why would it? You're going to call something like:
>> >
>> >   edistadd foo.tar.gz bar.tar.gz
>> >
>> > ...and it will place the files in the right subdirectories.
>> >
>> > --
>> > Best regards,
>> > Michał Górny
>> >
>> >
>> >
>> >
>>
>> Michał,
>>
>> How does this work for fetch restricted files and finding other files
>> no longer on the mirrors?
>>
>> Its no longer a download and move it to $DISTFILES, or is it?
>> Whatever it is, users will need to do it unless files in  $DISTFILES
>> are accepted by package managers if they are not found in the main
>> structure.
>
> I've just answered that, and it's in the GLEP also. There will be
> a helper tool to make this easy. Furthermore, I think we may even make
> Portage keep accepting both locations indefinitely.
>
> As for finding files in your distdir, there's no reason why plain:
>
>   find -name 'foo.tar.gz'
>
> wouldn't work.
>
> --
> Best regards,
> Michał Górny
>
>