From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1OJbHh-0001eD-Gp for garchives@archives.gentoo.org; Tue, 01 Jun 2010 23:54:13 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 1AFDAE0EF8; Tue, 1 Jun 2010 23:53:35 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by pigeon.gentoo.org (Postfix) with ESMTP id ACCB4E0EF8 for ; Tue, 1 Jun 2010 23:53:35 +0000 (UTC) Received: from [192.168.22.4] (ip68-4-152-120.oc.oc.cox.net [68.4.152.120]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTP id 9D0B01B402F for ; Tue, 1 Jun 2010 23:53:34 +0000 (UTC) Message-ID: <4C059D7B.4080804@gentoo.org> Date: Tue, 01 Jun 2010 16:53:31 -0700 From: Zac Medico User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100419 Thunderbird/3.0.4 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org MIME-Version: 1.0 To: gentoo-portage-dev@lists.gentoo.org Subject: Re: [gentoo-portage-dev] Package compression header for binhosts References: <4C047F52.30209@gentoo.org> <20100601051608.GD19306@hrair> <1275422465.24611.9.camel@hangover> <4C057DA4.5050408@gentoo.org> In-Reply-To: X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Archives-Salt: 59d75f73-9d81-4e78-8864-50098e015e91 X-Archives-Hash: 91d870c67a5200b86848daad04d0acdb On 06/01/2010 02:52 PM, Brian Harring wrote: > That bug isn't about a collision, it's about files being replaced underneath > Packages feet. Even with the tricks you've leveled the issue of things > changing under foot still is possible- you've just made the race less > likely. AFAIK the race is completely eliminated by the RCU-like snapshot mechanism. I think I like your hash-in-the-filename idea better though, since it seems simpler to implement and maintain. It can even be done with the existing version 0 format by abusing the per-package PATH attribute to refer to a filename that contains a hash (maybe something like $CATEGORY/$PF.$MD5.tbz2). I wouldn't want to abuse PATH for compression in version 0 though, since clients are only required to support tbz2. > What I was talking about was solving this issue once and for all via > restructuring, and specifically refering to the potential of an md5 > collision in the URI space- specifically what I'm implementing for pkgcore > is the ability to do stupid stuff like this- > > http://host/binpkg-store/$MD5.{txz,tbz2,tgz} That would be the MD5 of the entire file, after compression and having the xpak segment appended, right? > then have multiple views accessible just via pointing the binpkg repo remote > url at > > http://host/views/license/oss-approved/ > http://host/views/keywords/amd64/stable/ > http://host/views/raw/ # no filtering on the view of the binpkg repo, see > everything. So, the default path of a package would come from looking at the MD5 in the Packages file and then mapping that to a path? > Via restructuring where the binpkgs are stored and doing this approach, > multiple views can be had easily into the repo. An additional benefit of > this approach is that via making URI able to point outside the host, you > could combine multiple seperate repositories into one just via a view. This might also be useful for creating per-profile views while allowing packages to be shared between profiles in cases when hosting a separate build would be redundant. It might be possible to save lots of build time, disk space, and testing that way. Being able to have multiple builds of the same package with different USE settings is also solves bug 150031 [1]. >> Eventually, I'd like to see gentoo >> officially distributing binary packages, so that we'll be able to >> get a slice of the binary distribution pie. When that happens, we're >> certainly not going to want to have race conditions like these in >> our public binhosts. >> > > I'd suggest abandoning the current repository layout of Packages then, since > it's irrevocably flawed. You can hack around it via jamming timestamp/md5 > info into URI, but that's not a sane solution. Shrug, it's a handy way to solve race conditions given the existing version 0 format. It's not optimal, so we'll surely want something better in version 1. [1] http://bugs.gentoo.org/show_bug.cgi?id=150031 -- Thanks, Zac