From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1SbX8S-0000Qi-1U for garchives@archives.gentoo.org; Mon, 04 Jun 2012 13:15:52 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 95836E06F9; Mon, 4 Jun 2012 13:15:37 +0000 (UTC) Received: from mail-pz0-f53.google.com (mail-pz0-f53.google.com [209.85.210.53]) by pigeon.gentoo.org (Postfix) with ESMTP id 629F6E057F for ; Mon, 4 Jun 2012 13:14:53 +0000 (UTC) Received: by dadg9 with SMTP id g9so6900779dad.40 for ; Mon, 04 Jun 2012 06:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=bcXLQX8OyGgCdu3OHpwNDJKs5mJDvw3ZZB7VcwdL47s=; b=o8Bf6x9e+C/IGRghhccBRc8DfOj8/zyd48q9boJ+OsD72gjaXxDyX4U/nob7BENOmP /VaU3H9sTknmxTDIFXExIbRx1sm4JB5Kx/qqRH/ktF49yYx+NPPdAqZQDqArIcPZSnvm xJmB5Ou+hzyUN2pLF3Veep0b5qDXY95FtNVUmX/M+4nrohlh8aIw5cWZpNTgnhL5jhHd 5VI2ARaL59joAGS1hyG7A8w0vDgmbl1XKT4kN00SUYh7AqFrl8e/8FYxjjqaeGeEQNUZ BTh2TlFonPO0JJozGQdnHWcSPxcD295d1XAFLRDz5S2/nWZbpslAFvwgG00Y22YKWMNF Gk4w== Received: by 10.68.131.10 with SMTP id oi10mr10146179pbb.122.1338815692703; Mon, 04 Jun 2012 06:14:52 -0700 (PDT) Received: from smtp.gmail.com:587 (74-95-192-101-SFBA.hfc.comcastbusiness.net. [74.95.192.101]) by mx.google.com with ESMTPS id ok6sm13286843pbb.29.2012.06.04.06.14.49 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 04 Jun 2012 06:14:51 -0700 (PDT) Received: by smtp.gmail.com:587 (sSMTP sendmail emulation); Mon, 04 Jun 2012 06:15:13 -0700 Date: Mon, 4 Jun 2012 06:15:13 -0700 From: Brian Harring To: mgorny@gentoo.org Cc: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] Re: metadata/md5-cache Message-ID: <20120604131513.GA23002@localhost> References: <20120603092204.41c66ada@pomiocik.lan> <20120603113407.35ab0787@pomiocik.lan> <20120604092710.52f57bc1@pomiocik.lan> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120604092710.52f57bc1@pomiocik.lan> User-Agent: Mutt/1.5.21 (2010-09-15) X-Archives-Salt: 7e912d70-a2c6-420c-80e8-8bbb5635af02 X-Archives-Hash: 22c176959e81d774e4941f610d12ad0b On Mon, Jun 04, 2012 at 09:27:10AM +0200, Micha?? G??rny wrote: > On Sun, 3 Jun 2012 09:48:26 +0000 > "Robin H. Johnson" wrote: > > > On Sun, Jun 03, 2012 at 11:34:07AM +0200, Micha?? G??rny wrote: > > > I means using separate proto for metadata, not necesarrily git. In > > > any case, if it comes to transferring a lot of frequently-changing > > > files, rsync is not that efficient... > > It does NOT send any of the intermediate states. > > But it does have to check all the files. Which is a pretty minimal cost in the grand scheme of things. You also need to figure out what 'efficiency' you're going to talk about here; network io, disk io, cpu io, etc. Most people in this case care about network IO; rsync's not perfect, but for reasons described below, it's the best of breed for the usage scenario. > Did I mention I'm not talking necessarily about git? Git would be sanest if you were after this; it already does point to point delta transformations sanely. No point in reinventing a VCS; if you can't force the tree back to a known good state (aka, distributed VCS), you can't apply deltas to it, which case you need an rsync like algo. > Rather anything which would just > lookup our timestamp, revision or whatever and just send what have > changed, in a packed manner. This would be reinventing git/VCS, or more likely, pretending that a timestamp file automatically means the repository is *unmodified*, and trying to do a point to point transformation on it. Where you're notion breaks down is that fun little bit about "unmodified". This is why rsync is used; it's not limited to a point to point transformation, it's able to work from any starting point *efficiently*. Either way, suggest you do some research into this- including efficiencies of rsync, git, existing snapshot delta rsync machinery (tarsync, diffball, etc), study the trade offs inherint in each. Your initial email frankly reaks of NIH, hence my suggestions to go investigate what exists now. ~harring