From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <gentoo-dev+bounces-64420-garchives=archives.gentoo.org@lists.gentoo.org>
Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80])
by finch.gentoo.org (Postfix) with ESMTP id 48F29138247
for <garchives@archives.gentoo.org>; Fri, 17 Jan 2014 19:19:57 +0000 (UTC)
Received: from pigeon.gentoo.org (localhost [127.0.0.1])
by pigeon.gentoo.org (Postfix) with SMTP id 12E68E0B42;
Fri, 17 Jan 2014 19:19:51 +0000 (UTC)
Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by pigeon.gentoo.org (Postfix) with ESMTPS id 23328E0AE5
for <gentoo-dev@lists.gentoo.org>; Fri, 17 Jan 2014 19:19:50 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
by smtp.gentoo.org (Postfix) with ESMTP id 27BDF33F91B
for <gentoo-dev@lists.gentoo.org>; Fri, 17 Jan 2014 19:19:49 +0000 (UTC)
X-Virus-Scanned: by amavisd-new using ClamAV at gentoo.org
X-Spam-Flag: NO
X-Spam-Score: -1.365
X-Spam-Level:
X-Spam-Status: No, score=-1.365 tagged_above=-999 required=5.5
tests=[AWL=-1.034, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-0.329,
SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=no
Received: from smtp.gentoo.org ([IPv6:::ffff:127.0.0.1])
by localhost (smtp.gentoo.org [IPv6:::ffff:127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id UjTk9FOMwcrg for <gentoo-dev@lists.gentoo.org>;
Fri, 17 Jan 2014 19:19:43 +0000 (UTC)
Received: from plane.gmane.org (plane.gmane.org [80.91.229.3])
(using TLSv1 with cipher AES256-SHA (256/256 bits))
(No client certificate requested)
by smtp.gentoo.org (Postfix) with ESMTPS id EE10533F8EA
for <gentoo-dev@gentoo.org>; Fri, 17 Jan 2014 19:19:42 +0000 (UTC)
Received: from list by plane.gmane.org with local (Exim 4.69)
(envelope-from <lnx-gentoo-dev@m.gmane.org>)
id 1W4Ex7-0003ym-Ki
for gentoo-dev@gentoo.org; Fri, 17 Jan 2014 20:19:37 +0100
Received: from ip68-231-22-224.ph.ph.cox.net ([68.231.22.224])
by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
id 1AlnuQ-0007hv-00
for <gentoo-dev@gentoo.org>; Fri, 17 Jan 2014 20:19:37 +0100
Received: from 1i5t5.duncan by ip68-231-22-224.ph.ph.cox.net with local (Gmexim 0.1 (Debian))
id 1AlnuQ-0007hv-00
for <gentoo-dev@gentoo.org>; Fri, 17 Jan 2014 20:19:37 +0100
X-Injected-Via-Gmane: http://gmane.org/
To: gentoo-dev@lists.gentoo.org
From: Duncan <1i5t5.duncan@cox.net>
Subject: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas
Date: Fri, 17 Jan 2014 19:19:14 +0000 (UTC)
Message-ID: <pan$d7ecf$4f65435f$499148d2$a3d10828@cox.net>
References: <20140117172730.0c504246@pomiot.lan>
Precedence: bulk
List-Post: <mailto:gentoo-dev@lists.gentoo.org>
List-Help: <mailto:gentoo-dev+help@lists.gentoo.org>
List-Unsubscribe: <mailto:gentoo-dev+unsubscribe@lists.gentoo.org>
List-Subscribe: <mailto:gentoo-dev+subscribe@lists.gentoo.org>
List-Id: Gentoo Linux mail <gentoo-dev.gentoo.org>
X-BeenThere: gentoo-dev@lists.gentoo.org
Reply-to: gentoo-dev@lists.gentoo.org
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Complaints-To: usenet@ger.gmane.org
X-Gmane-NNTP-Posting-Host: ip68-231-22-224.ph.ph.cox.net
User-Agent: Pan/0.140 (Chocolate Salty Balls; GIT 6daf184
/usr/src/portage/src/egit-src/pan2)
X-Archives-Salt: dd09e27b-4bc0-490e-8918-979c937116b9
X-Archives-Hash: f30d74894dd8b4618d147f378bc755dc
Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:
> Now some numbers. I did some tests 'converting' late gx86 daily tarballs
> to squashfs. I've used squashfs 4.2 with LZO compression since it's
> quite good and very fast.
>
> 96M portage-20140108.sqfs
[...]
> 97M portage-20140114.sqfs
> 97M portage-20140115.sqfs
>
> For deltas [...]
>
> 4,9M portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
[...]
> 8,5M portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
>
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing a
> compressed filesystem. What's important, however, is that applying it
> takes ~2.5 second on my 2 GHz Athlon64.
And... eyeballing a 6 MiB average, diffs are ~1/16 the full squashfs
size, perhaps a bit larger. So people updating once a week or even about
every 10 days would see a bandwidth savings, provided the sync script was
intelligent enough to apply updates serially.
The breakover point would be roughly an update every two weeks, or twice
a month, at which point just downloading a new full squashfs would be
easier, at about the same bandwidth.
> What do you think?
How does this, particularly the metadata cache, interact with overlays?
That's /my/ big question.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman