From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 48F29138247 for ; Fri, 17 Jan 2014 19:19:57 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 12E68E0B42; Fri, 17 Jan 2014 19:19:51 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 23328E0AE5 for ; Fri, 17 Jan 2014 19:19:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.gentoo.org (Postfix) with ESMTP id 27BDF33F91B for ; Fri, 17 Jan 2014 19:19:49 +0000 (UTC) X-Virus-Scanned: by amavisd-new using ClamAV at gentoo.org X-Spam-Flag: NO X-Spam-Score: -1.365 X-Spam-Level: X-Spam-Status: No, score=-1.365 tagged_above=-999 required=5.5 tests=[AWL=-1.034, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-0.329, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=no Received: from smtp.gentoo.org ([IPv6:::ffff:127.0.0.1]) by localhost (smtp.gentoo.org [IPv6:::ffff:127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UjTk9FOMwcrg for ; Fri, 17 Jan 2014 19:19:43 +0000 (UTC) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id EE10533F8EA for ; Fri, 17 Jan 2014 19:19:42 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1W4Ex7-0003ym-Ki for gentoo-dev@gentoo.org; Fri, 17 Jan 2014 20:19:37 +0100 Received: from ip68-231-22-224.ph.ph.cox.net ([68.231.22.224]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 17 Jan 2014 20:19:37 +0100 Received: from 1i5t5.duncan by ip68-231-22-224.ph.ph.cox.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 17 Jan 2014 20:19:37 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: gentoo-dev@lists.gentoo.org From: Duncan <1i5t5.duncan@cox.net> Subject: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas Date: Fri, 17 Jan 2014 19:19:14 +0000 (UTC) Message-ID: References: <20140117172730.0c504246@pomiot.lan> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: ip68-231-22-224.ph.ph.cox.net User-Agent: Pan/0.140 (Chocolate Salty Balls; GIT 6daf184 /usr/src/portage/src/egit-src/pan2) X-Archives-Salt: dd09e27b-4bc0-490e-8918-979c937116b9 X-Archives-Hash: f30d74894dd8b4618d147f378bc755dc Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted: > Now some numbers. I did some tests 'converting' late gx86 daily tarballs > to squashfs. I've used squashfs 4.2 with LZO compression since it's > quite good and very fast. > > 96M portage-20140108.sqfs [...] > 97M portage-20140114.sqfs > 97M portage-20140115.sqfs > > For deltas [...] > > 4,9M portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw > 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw [...] > 8,5M portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw > > As you can see, the deltas are quite large compared to the actual > changes. However, we could have expected that since we're diffing a > compressed filesystem. What's important, however, is that applying it > takes ~2.5 second on my 2 GHz Athlon64. And... eyeballing a 6 MiB average, diffs are ~1/16 the full squashfs size, perhaps a bit larger. So people updating once a week or even about every 10 days would see a bandwidth savings, provided the sync script was intelligent enough to apply updates serially. The breakover point would be roughly an update every two weeks, or twice a month, at which point just downloading a new full squashfs would be easier, at about the same bandwidth. > What do you think? How does this, particularly the metadata cache, interact with overlays? That's /my/ big question. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman