From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 4D343138247 for ; Fri, 17 Jan 2014 18:19:09 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id E8AD9E0B14; Fri, 17 Jan 2014 18:19:01 +0000 (UTC) Received: from mail-we0-f173.google.com (mail-we0-f173.google.com [74.125.82.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id BCAF7E0B0C for ; Fri, 17 Jan 2014 18:18:59 +0000 (UTC) Received: by mail-we0-f173.google.com with SMTP id t60so4858678wes.18 for ; Fri, 17 Jan 2014 10:18:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=V++DxNd3EvN7OSkF0As+Qxgpv/YR6mc/VBqYXY07SUo=; b=d6l/WHaCcGqJ4Orag3ogJn9Nc/kJdIFVLW691PhbqTnFdpqgI4H03hH0UXVwpeCaCN sdJQPCqLUn2Xx/HJIBDdxjJX3raH1zu4pZBlJU/MW8BxQ6uQG65XKU+WIIlPo+Z/h3Yz 2e+cb3y4b3FHqkU4KmKqCRA0+0fHpusE8KmJB64/51f+HbUErOVBJdy5Tw2JmNTKwKQR xyRbFMiHXAE9OatnofhLYUf5N7Bp1Y8AD7VvacRhV1WKwcAj2Sp9etJXDtKvjiityjms O9+iLyPr63USrEtGNh9gZoeQhr92HGlrU6zBoGCPVjXhoAK3y22l7he4DffDuNGDYQ7g vqzw== X-Gm-Message-State: ALoCoQkw6xn+p5LoFgasyBm4YahTEikqEFdzEyRIOGUYTWnP6z0waoa0F15kXcSE6Ubg+HpWm0oF Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 X-Received: by 10.180.36.101 with SMTP id p5mr3958913wij.15.1389982738257; Fri, 17 Jan 2014 10:18:58 -0800 (PST) Sender: antarus@scriptkitty.com Received: by 10.216.170.129 with HTTP; Fri, 17 Jan 2014 10:18:58 -0800 (PST) X-Originating-IP: [2620:0:1000:2e00:ae16:2dff:fe07:51d9] In-Reply-To: <20140117172730.0c504246@pomiot.lan> References: <20140117172730.0c504246@pomiot.lan> Date: Fri, 17 Jan 2014 10:18:58 -0800 X-Google-Sender-Auth: FjqKvgBOuk4wfHfvo8TlE-ChRME Message-ID: Subject: Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas From: Alec Warner To: Gentoo Dev Content-Type: multipart/alternative; boundary=e89a8f50345e5bc41b04f02e9367 X-Archives-Salt: d70b252e-ce3f-45a1-8fde-1cf19852d6f1 X-Archives-Hash: 4d55f3eab63702d64d9844567c95ef92 --e89a8f50345e5bc41b04f02e9367 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Fri, Jan 17, 2014 at 8:27 AM, Micha=C5=82 G=C3=B3rny = wrote: > Hello, all. > > I'm using squashfs to hold my Gentoo repositories on all of my systems > for some time. As you probably know, this allows me to save space while > keeping portage fast. However, it makes updating the tree quite > burdensome and time-consuming. > > We're already hosting daily gx86 tarballs on our mirrors, and deltas > made using diffball. Those can be used with Zac's emerge-delta-webrsync > to get daily updates done with minimal network overhead. Sadly, it > takes the whole process even more time consuming :). > > Therefore, I'd like to suggest an alternative solution that could help > out Gentoo users that use squashfs for gx86 and would like to be able > to get daily updates fast and easy. > > The idea is to host -- along with the tarballs -- daily squashfs images > of gx86 in a chosen format. Additionally, the images would come with > deltas made using xdelta3 or a similar tool. Those deltas -- with > a slight download overhead -- would allow very fast updates > of the squashfs. > > Now some numbers. I did some tests 'converting' late gx86 daily > tarballs to squashfs. I've used squashfs 4.2 with LZO compression > since it's quite good and very fast. > > 96M portage-20140108.sqfs > 96M portage-20140109.sqfs > 96M portage-20140110.sqfs > 96M portage-20140111.sqfs > 96M portage-20140112.sqfs > 96M portage-20140113.sqfs > 97M portage-20140114.sqfs > 97M portage-20140115.sqfs > > For deltas, I've used xdelta3 with max compression (-9) and djw > secondary compression (it gave ~0.1M smaller files than fgk > and ~0.5M gain than with no secondary compression). > > 4,9M portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw > 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw > 5,6M portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw > 8,9M portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw > 6,3M portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw > 7,8M portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw > 8,5M portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw > > As you can see, the deltas are quite large compared to the actual > changes. However, we could have expected that since we're diffing > a compressed filesystem. What's important, however, is that applying > it takes ~2.5 second on my 2 GHz Athlon64. > It wasn't clear to me, are these trees with metadata included? -A > > So, even with the extra download time, the update is much faster > than recreating the squashfs. And unlike some types of unionfs, > it doesn't come with extra runtime slowdown. > > What do you think? > > -- > Best regards, > Micha=C5=82 G=C3=B3rny > --e89a8f50345e5bc41b04f02e9367 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On F= ri, Jan 17, 2014 at 8:27 AM, Micha=C5=82 G=C3=B3rny <<= a href=3D"mailto:mgorny@gentoo.org" target=3D"_blank">mgorny@gentoo.org= > wrote:
Hello, all.

I'm using squashfs to hold my Gentoo repositories on all of my systems<= br> for some time. As you probably know, this allows me to save space while
keeping portage fast. However, it makes updating the tree quite
burdensome and time-consuming.

We're already hosting daily gx86 tarballs on our mirrors, and deltas made using diffball. Those can be used with Zac's emerge-delta-webrsync=
to get daily updates done with minimal network overhead. Sadly, it
takes the whole process even more time consuming :).

Therefore, I'd like to suggest an alternative solution that could help<= br> out Gentoo users that use squashfs for gx86 and would like to be able
to get daily updates fast and easy.

The idea is to host -- along with the tarballs -- daily squashfs images
of gx86 in a chosen format. Additionally, the images would come with
deltas made using xdelta3 or a similar tool. Those deltas -- with
a slight download overhead -- would allow very fast updates
of the squashfs.

Now some numbers. I did some tests 'converting' late gx86 daily
tarballs to squashfs. I've used squashfs 4.2 with LZO compression
since it's quite good and very fast.

96M =C2=A0 =C2=A0 portage-20140108.sqfs
96M =C2=A0 =C2=A0 portage-20140109.sqfs
96M =C2=A0 =C2=A0 portage-20140110.sqfs
96M =C2=A0 =C2=A0 portage-20140111.sqfs
96M =C2=A0 =C2=A0 portage-20140112.sqfs
96M =C2=A0 =C2=A0 portage-20140113.sqfs
97M =C2=A0 =C2=A0 portage-20140114.sqfs
97M =C2=A0 =C2=A0 portage-20140115.sqfs

For deltas, I've used xdelta3 with max compression (-9) and djw
secondary compression (it gave ~0.1M smaller files than fgk
and ~0.5M gain than with no secondary compression).

4,9M =C2=A0 =C2=A0portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw 6,3M =C2=A0 =C2=A0portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw 5,6M =C2=A0 =C2=A0portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw 8,9M =C2=A0 =C2=A0portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw 6,3M =C2=A0 =C2=A0portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw 7,8M =C2=A0 =C2=A0portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw 8,5M =C2=A0 =C2=A0portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
As you can see, the deltas are quite large compared to the actual
changes. However, we could have expected that since we're diffing
a compressed filesystem. What's important, however, is that applying it takes ~2.5 second on my 2 GHz Athlon64.

<= div>It wasn't clear to me, are these trees with metadata included?

-A
=C2=A0

--
Best regards,
Micha=C5=82 G=C3=B3rny

--e89a8f50345e5bc41b04f02e9367--