From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id B8204138247 for ; Fri, 17 Jan 2014 19:30:22 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id BD80AE0B2C; Fri, 17 Jan 2014 19:30:14 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id B3811E0B04 for ; Fri, 17 Jan 2014 19:30:12 +0000 (UTC) Received: from pomiot.lan (static-81-219-166-219.devs.futuro.pl [81.219.166.219]) (using SSLv3 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: mgorny) by smtp.gentoo.org (Postfix) with ESMTPSA id 8F16F33F8C0; Fri, 17 Jan 2014 19:30:09 +0000 (UTC) Date: Fri, 17 Jan 2014 20:30:00 +0100 From: =?UTF-8?B?TWljaGHFgiBHw7Nybnk=?= To: gentoo-dev@lists.gentoo.org Cc: 1i5t5.duncan@cox.net Subject: Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas Message-ID: <20140117203000.01841974@pomiot.lan> In-Reply-To: References: <20140117172730.0c504246@pomiot.lan> Organization: Gentoo X-Mailer: Claws Mail 3.9.3 (GTK+ 2.24.22; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/BN6FeY8rrZwAJQSLmxIqllS"; protocol="application/pgp-signature" X-Archives-Salt: 9f25acf3-3d3c-4b21-9b56-a1a88eb43f08 X-Archives-Hash: 65f773673f282e9a9dd5f9127e4a764c --Sig_/BN6FeY8rrZwAJQSLmxIqllS Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Dnia 2014-01-17, o godz. 19:19:14 Duncan <1i5t5.duncan@cox.net> napisa=C5=82(a): > Micha=C5=82 G=C3=B3rny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excer= pted: >=20 > > Now some numbers. I did some tests 'converting' late gx86 daily tarballs > > to squashfs. I've used squashfs 4.2 with LZO compression since it's > > quite good and very fast. > >=20 > > 96M portage-20140108.sqfs > [...] > > 97M portage-20140114.sqfs > > 97M portage-20140115.sqfs > >=20 > > For deltas [...] > >=20 > > 4,9M portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw > > 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw > [...] > > 8,5M portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw > >=20 > > As you can see, the deltas are quite large compared to the actual > > changes. However, we could have expected that since we're diffing a > > compressed filesystem. What's important, however, is that applying it > > takes ~2.5 second on my 2 GHz Athlon64. >=20 > And... eyeballing a 6 MiB average, diffs are ~1/16 the full squashfs=20 > size, perhaps a bit larger. So people updating once a week or even about= =20 > every 10 days would see a bandwidth savings, provided the sync script was= =20 > intelligent enough to apply updates serially. >=20 > The breakover point would be roughly an update every two weeks, or twice= =20 > a month, at which point just downloading a new full squashfs would be=20 > easier, at about the same bandwidth. Yes, that's the initial goal. The update code would catch that condition and perform full fetch instead. However, it may be actually beneficial to provide other durations, like weekly deltas. In my tests, the daily updates for this week summed up to almost 50M while the weekly was barely 20M. > > What do you think? >=20 > How does this, particularly the metadata cache, interact with overlays? = =20 > That's /my/ big question. The same way as usual. Nothing special happens unless you override eclasses via overlays. If you do that, you need unionfs to save the cache updates -- but then, it has no point for you to use squashfs. --=20 Best regards, Micha=C5=82 G=C3=B3rny --Sig_/BN6FeY8rrZwAJQSLmxIqllS Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQJ8BAEBCgBmBQJS2YS5XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2REJCMDdDQzRGMERBRDA2RUEwQUZFNDFC MDdBMUFFQUVGQjQ0NjRFAAoJELB6GurvtEZONIkP/2E15f8ruqyeXxzVcwHSULLL hW/Gq6kZnT/ciEO+5aznANRPDbtXsk8MwZZp6ITobwcQ+9tQUE9nF9eJk0dvleos verLKN+tkyrZYdBGGhU3POQ7pRh9qEqEnVTi3FdIOM6Jwx/R4Ihb7pfO8xL6V/YT aUCCCojBVrkA7uFMQ1mjMeest8YgbHjkZXceNIa6WYfvWZTXnM9C9tEQiPm/DxIb YNa8MUfFVam9dss2JsWtgYmHV0UWulR6XUI6dJrXA0QRtnvSE4r0gjnd5+/4cxg6 GHjrVNtA0f1c6ZAT5vKGHEg/i3uW2kyqDhur0AznQ7g7y8BdejetehdtN1NHptY5 V7IE+IC2gvJ5K/80MzkxTpQ4+QeToJ4UUW8IZF2W6OJR0ecaaHg9a6pscogS3cYJ zoQseU4ohttzylpFanei+GyC+oECxBc4nIPkfLnJIcdSrGJxpK0xC/0g1uhkJCNt 6bmfY50Y5edOAAWSxQ5YGG2sy9X9ylt1Ub829eYf64vxvbvhaa8ugZZ2GFPIoRwA rzFAgb2/JfohQQf9qhwTSzEfp+Lod9zIy5EpT7ZxPprJiTvU+Q5GlSyVecHp/jje GvgoR30bb5PD5KcRnSrlWXwyj/db+Tx9oSxrWs6UzCOn+tt+sbCVs15VtQUk09R0 ptffFvRVZtXk+qvE8NG1 =D1wN -----END PGP SIGNATURE----- --Sig_/BN6FeY8rrZwAJQSLmxIqllS--