From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1OJZNs-0003ta-Jd for garchives@archives.gentoo.org; Tue, 01 Jun 2010 21:52:28 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 700CCE0FEE; Tue, 1 Jun 2010 21:52:07 +0000 (UTC) Received: from mail-iw0-f181.google.com (mail-iw0-f181.google.com [209.85.214.181]) by pigeon.gentoo.org (Postfix) with ESMTP id 40B7DE0FEE for ; Tue, 1 Jun 2010 21:52:07 +0000 (UTC) Received: by iwn33 with SMTP id 33so822347iwn.40 for ; Tue, 01 Jun 2010 14:52:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=fZPBlYgmkj3bR/HpR0ella3MlD3aQaAVgzxoDfrt1II=; b=MxB5gtojLk7CoBG+DdU0mEB8AUVGaVRyxWfyl20YOAJGhnfNZiWAjsK7hznLfbCJbe aVnsBuCpEw99TauX82LU5L2GGQvSmWxDCp48pneGzfqhIJyhM4zaaXmUMgidD1xJkDJi bYYw3IhmiWmDTfMpVXHVJZu26du3oGYfey6Z4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=L1wQLe0tg92ILfcIxeBMz9FIi/9Q7ATFwVoRi9oN3MBNaoGS0fr9mCJjmYvJawj3jO 9cOyX7mUpEJVCNe7/mOQbUN6zrqzpor8RISLim3m9++7/S9T7Z7huk1Hltd9SBZTH2bx C+6rVnVw7snOpa4ys0O362N4QZnc0vYSqge7s= Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org MIME-Version: 1.0 Received: by 10.231.169.129 with SMTP id z1mr8591470iby.26.1275429126794; Tue, 01 Jun 2010 14:52:06 -0700 (PDT) Received: by 10.231.194.147 with HTTP; Tue, 1 Jun 2010 14:52:06 -0700 (PDT) In-Reply-To: <4C057DA4.5050408@gentoo.org> References: <4C047F52.30209@gentoo.org> <20100601051608.GD19306@hrair> <1275422465.24611.9.camel@hangover> <4C057DA4.5050408@gentoo.org> Date: Tue, 1 Jun 2010 14:52:06 -0700 Message-ID: Subject: Re: [gentoo-portage-dev] Package compression header for binhosts From: Brian Harring To: gentoo-portage-dev@lists.gentoo.org Content-Type: multipart/alternative; boundary=0016e6d26d6709eeea0487fefc75 X-Archives-Salt: a20ecdbe-1c7a-4c29-8a30-215ae4fccdfa X-Archives-Hash: f77901a3b75c592a9438833421cae4cb --0016e6d26d6709eeea0487fefc75 Content-Type: text/plain; charset=UTF-8 On Tue, Jun 1, 2010 at 2:37 PM, Zac Medico wrote: > On 06/01/2010 02:22 PM, Brian Harring wrote: > > As for zacs tool to try and generate new views of a repository via > > hardlinking/recreating the tree... frankly it's a bit of a hack. Via > > DEFAULT_URI and relying on the hash, you can make a stable repository > that > > is able to be updated in place without corrupting ongoing downloads- > simply > > put, new additions to the repo don't perturb current DL's since the md5 > is > > the same (hash collision chance is low enough that I don't care about it > > here). > > When you say "hash collision" are you talking about > http://crosbug.com/3225? Maybe that behavior is acceptable for > small-scale private use, but for large scale public repositories I'd > say it's totally unacceptable. That bug isn't about a collision, it's about files being replaced underneath Packages feet. Even with the tricks you've leveled the issue of things changing under foot still is possible- you've just made the race less likely. What I was talking about was solving this issue once and for all via restructuring, and specifically refering to the potential of an md5 collision in the URI space- specifically what I'm implementing for pkgcore is the ability to do stupid stuff like this- http://host/binpkg-store/$MD5.{txz,tbz2,tgz} then have multiple views accessible just via pointing the binpkg repo remote url at http://host/views/license/oss-approved/ http://host/views/keywords/amd64/stable/ http://host/views/raw/ # no filtering on the view of the binpkg repo, see everything. Via restructuring where the binpkgs are stored and doing this approach, multiple views can be had easily into the repo. An additional benefit of this approach is that via making URI able to point outside the host, you could combine multiple seperate repositories into one just via a view. > Eventually, I'd like to see gentoo > officially distributing binary packages, so that we'll be able to > get a slice of the binary distribution pie. When that happens, we're > certainly not going to want to have race conditions like these in > our public binhosts. > I'd suggest abandoning the current repository layout of Packages then, since it's irrevocably flawed. You can hack around it via jamming timestamp/md5 info into URI, but that's not a sane solution. ~harring --0016e6d26d6709eeea0487fefc75 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On Tue, Jun 1, 2010 at 2:37 PM, Zac Medico <zmedico@gentoo.org> wrote:
On 06/01/2010 02:22 PM, Brian Harring wrote:
> As for zacs tool to try and generate new views of a repository via
> hardlinking/recreating the tree... frankly it's a bit of a hack. = =C2=A0Via
> DEFAULT_URI and relying on the hash, you can make a stable repository = that
> is able to be updated in place without corrupting ongoing downloads- s= imply
> put, new additions to the repo don't perturb current DL's sinc= e the md5 is
> the same (hash collision chance is low enough that I don't care ab= out it
> here).

When you say "hash collision" are you talking about
http://crosbug.com/32= 25? Maybe that behavior is acceptable for
small-scale private use, but for large scale public repositories I'd say it's totally unacceptable.

That bug= isn't about a collision, it's about files being replaced underneat= h Packages feet. =C2=A0Even with the tricks you've leveled the issue of= things changing under foot still is possible- you've just made the rac= e less likely.

What I was talking about was solving this issue once an= d for all via restructuring, and specifically refering to the potential of = an md5 collision in the URI space- specifically what I'm implementing f= or pkgcore is the ability to do stupid stuff like this-


then have multiple views accessible just via pointing the binpkg repo remo= te url at

http://host/views/keywords/amd64/stable/
http://host/views/raw/ # no fil= tering on the view of the binpkg repo, see everything.

=
Via restructuring where the binpkgs are stored and doing this approach= , multiple views can be had easily into the repo. =C2=A0An additional benef= it of this approach is that via making URI able to point outside the host, = you could combine multiple seperate repositories into one just via a view.<= /div>

=C2=A0
Eventually= , I'd like to see gentoo
officially distributing binary packages, so that we'll be able to
get a slice of the binary distribution pie. When that happens, we're certainly not going to want to have race conditions like these in
our public binhosts.

I'd suggest ab= andoning the current repository layout of Packages then, since it's irr= evocably flawed. =C2=A0You can hack around it via jamming timestamp/md5 inf= o into URI, but that's not a sane solution.

~harring
--0016e6d26d6709eeea0487fefc75--