From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 0689E1382C5 for ; Sat, 27 Jan 2018 16:43:10 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 41801E09F2; Sat, 27 Jan 2018 16:43:04 +0000 (UTC) Received: from mail-lf0-x233.google.com (mail-lf0-x233.google.com [IPv6:2a00:1450:4010:c07::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id B570DE09E8 for ; Sat, 27 Jan 2018 16:43:03 +0000 (UTC) Received: by mail-lf0-x233.google.com with SMTP id k19so4374806lfj.1 for ; Sat, 27 Jan 2018 08:43:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-transfer-encoding; bh=3vBNeXDPxGYMdzdFb74fsMlo0KEEkf9fNQ4ic7sJ6po=; b=cA5qnh75987TN5j1TFDJBbjHx7/vRrKuhTb8RiJnT206iuWWD4nNpcnUkwvuaiMpcs z8a+iFzXYagf7wID48v7kOgGiUMqCJoJv+ADHde/cuRCDV8C+eVHpP+cjFOkYPDf8U0o qBvMNrJFsD1fV/yuqCogtCCxIZMiiQUrFADokyKB3HpHUSk4EBj6SfG5cDAZKdtkP7Y+ u/eCMlUChtOD9JW0hbVdVDWyy2UxQie5r3sEJGDTDIMnPC0iWZMggJYFjFKOL3QmelB+ n1Pol3TINE6qo0vnmT3aX39ltM67pN4GDfPmlGmhmdjyTNO97wQBduYQho61zO1f88ol inHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-transfer-encoding; bh=3vBNeXDPxGYMdzdFb74fsMlo0KEEkf9fNQ4ic7sJ6po=; b=gT3un4z/UnR9X5Osn7Pb7freV9psmBTJ6q7F9cnBqqQCUsuBvIQKtO+4IRDsTHGhbB J2kBTtIkEo6b+eD018UuFdujgH6jiFuA2HwGCpeRu2e4P+ZlEMRlDNYtyf177XAFI8bR VI1489ZOmqKrvrH8ls+WoznV5JTHXNU2s7kwxjUq6VGN1RuP2V1aeuCNxE/ejxam98xr M0pi/Mbu9+6NqBUo6wE02g311S5GHS+T680f+EaXHOiIUQ3egQ4YeO+JYO/rvwo+zc20 gNC/0JNByaJYRetBsIZzLwQl4xces4fcwASQzodJBPICjgwO/0jxLh5iMYBWEktJsMaZ ZIoA== X-Gm-Message-State: AKwxytfcEm2H2aF2OMJrviIIWOR01Uv9fARFPsK4SYAFFdYqGnI15cSm XrwV65RFfqgsSr4vCafyutdF9hXaHBYhNfn5wYO7PA== X-Google-Smtp-Source: AH8x227SAMhcqmcPqJh+Kt5wigv4q7+7VuyqMdnhLp0/k/ZqXEC7vxcp3k9K5cv73/lxhfmy4jYKyfMTAQHwSRD0rO4= X-Received: by 10.25.208.210 with SMTP id h201mr9434293lfg.22.1517071381259; Sat, 27 Jan 2018 08:43:01 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Received: by 10.46.58.8 with HTTP; Sat, 27 Jan 2018 08:42:20 -0800 (PST) In-Reply-To: <1517053318.3816.5.camel@gentoo.org> References: <1517053318.3816.5.camel@gentoo.org> From: Gordon Pettey Date: Sat, 27 Jan 2018 10:42:20 -0600 Message-ID: Subject: Re: [gentoo-dev] [pre-GLEP] Split distfile mirror directory structure To: gentoo-dev@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 9b151572-89be-4d82-bb90-6c87a491333f X-Archives-Hash: b85b6c8d97b7b78aa95b5ac32e4b4718 Why not use a hash of the file name instead of its contents? That seems like it would be much simpler, and that's not going to reduce the output space for balance... On Sat, Jan 27, 2018 at 5:41 AM, Micha=C5=82 G=C3=B3rny = wrote: > W dniu sob, 27.01.2018 o godzinie 11=E2=88=B636=E2=80=89+0000, u=C5=BCytk= ownik Roy Bamford > napisa=C5=82: >> On 2018.01.27 08:30, Micha=C5=82 G=C3=B3rny wrote: >> > W dniu pi=C4=85, 26.01.2018 o godzinie 20=E2=88=B648=E2=80=89-0500, u= =C5=BCytkownik Michael >> > Orlitzky napisa=C5=82: >> > > On 01/26/2018 06:24 PM, Micha=C5=82 G=C3=B3rny wrote: >> > > > >> > > > The alternate option of using file hash has the advantage of >> > >> > having >> > > > a more balanced split. Furthermore, since hashes are stored >> > > > in Manifests using them is zero-cost. However, this solution has >> > >> > two >> > > > significant disadvantages: >> > > > >> > > > 1. The hash values are unknown for newly-downloaded distfiles, so >> > > > ``repoman`` (or an equivalent tool) would have to use a >> > >> > temporary >> > > > directory before locating the file in appropriate subdirectory. >> > > > >> > > > 2. User-provided distfiles (e.g. for fetch-restricted packages) >> > >> > with >> > > > hash mismatches would be placed in the wrong subdirectory, >> > > > potentially causing confusing errors. >> > > > >> > > >> > > The filename proposal sounds fine, so this is only academic, but: >> > >> > are >> > > these two points really disadvantages? >> > > >> > > What are we worried about in using a temporary directory? Copying >> > >> > across >> > > filesystem boundaries? Except in rare cases, $DISTDIR itself will be >> > > usable a temporary location (on the same filesystem), won't it? >> > >> > Why add the extra complexity when there's no need for one? Note that >> > there's also the problem of resuming transfers, so in the end we're >> > talking about permanent temporary directory where we keep unfinished >> > transfers. >> > >> > > For the second point, portage is going to tell me where to put the >> > >> > file, >> > > isn't it? Then no matter what garbage I download, won't portage look >> > >> > for >> > > it in the right place, because where-to-put-it is determined using >> > >> > the >> > > same manifest hash that determines where-to-find-it? >> > >> > No, it won't. Why would it? You're going to call something like: >> > >> > edistadd foo.tar.gz bar.tar.gz >> > >> > ...and it will place the files in the right subdirectories. >> > >> > -- >> > Best regards, >> > Micha=C5=82 G=C3=B3rny >> > >> > >> > >> > >> >> Micha=C5=82, >> >> How does this work for fetch restricted files and finding other files >> no longer on the mirrors? >> >> Its no longer a download and move it to $DISTFILES, or is it? >> Whatever it is, users will need to do it unless files in $DISTFILES >> are accepted by package managers if they are not found in the main >> structure. > > I've just answered that, and it's in the GLEP also. There will be > a helper tool to make this easy. Furthermore, I think we may even make > Portage keep accepting both locations indefinitely. > > As for finding files in your distdir, there's no reason why plain: > > find -name 'foo.tar.gz' > > wouldn't work. > > -- > Best regards, > Micha=C5=82 G=C3=B3rny > >