From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 501AB138247 for ; Sat, 18 Jan 2014 08:15:16 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 95441E0AAE; Sat, 18 Jan 2014 08:15:10 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 93DF6E0A89 for ; Sat, 18 Jan 2014 08:15:09 +0000 (UTC) Received: from pomiot.lan (static-81-219-166-219.devs.futuro.pl [81.219.166.219]) (using SSLv3 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: mgorny) by smtp.gentoo.org (Postfix) with ESMTPSA id A90A433F8FF; Sat, 18 Jan 2014 08:15:07 +0000 (UTC) Date: Sat, 18 Jan 2014 09:14:56 +0100 From: =?UTF-8?B?TWljaGHFgiBHw7Nybnk=?= To: gentoo-dev@lists.gentoo.org Cc: heroxbd@gentoo.org Subject: Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas Message-ID: <20140118091456.19afcfd6@pomiot.lan> In-Reply-To: <86zjmujdvm.fsf@moguhome00.in.awa.tohoku.ac.jp> References: <20140117172730.0c504246@pomiot.lan> <20140117203000.01841974@pomiot.lan> <86zjmujdvm.fsf@moguhome00.in.awa.tohoku.ac.jp> Organization: Gentoo X-Mailer: Claws Mail 3.9.3 (GTK+ 2.24.22; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/RDf9JWnQuW7bA1m=.j25FXz"; protocol="application/pgp-signature" X-Archives-Salt: 57d88fea-078c-4e8b-bc06-f74c3379b0b1 X-Archives-Hash: f65cde13c1bf9844f1a7dda190d34441 --Sig_/RDf9JWnQuW7bA1m=.j25FXz Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Dnia 2014-01-18, o godz. 10:08:13 heroxbd@gentoo.org napisa=C5=82(a): > Micha=C5=82 G=C3=B3rny writes: >=20 > > However, it may be actually beneficial to provide other durations, like > > weekly deltas. In my tests, the daily updates for this week summed up > > to almost 50M while the weekly was barely 20M. >=20 > Is there a way to merge the deltas without the original squashfs? You mean get two deltas and merge them to a single one? I have no idea, honestly :). It may be possible but xdelta3 is hard to use. > how fast is the delta generation? With max compression on 2 GHz amd64, about 15s for a daily delta. A weekly takes 25s. > Or we provide a set of 16,8,4,2,1 day deltas, then=20 >=20 > 16d: 1 piece needed > 8d: 2 needed > 4d: 4 needed > 2d: 8 needed > 1d: 16 needed >=20 > The total of 31 pieces will cover a month (31 days) with at most 5 > deltas to be downloaded. >=20 > e.g. If the system is 19 days old, then we download a 1d, 2d, and 16d. Unnecessarily complex and not worth the effort. Please remember that we're generating deltas against compressed image with metadata cache included. Those are far from optimal, and with your solution you'll quickly run over the original squashfs size. I'm currently working on another solution that should be a bit harder for the server yet much better for users. It assumes we're keeping deltas from N days back to today, i.e.: n-1..n, n-2..n, n-3..n. Any of those deltas is much smaller than two dailies (I'll provide exact numbers when my playground finishes, that is in few hours). You just download the one matching your current image and apply it, and TADAAM, you've got today's snapshot :). --=20 Best regards, Micha=C5=82 G=C3=B3rny --Sig_/RDf9JWnQuW7bA1m=.j25FXz Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQJ8BAEBCgBmBQJS2jgEXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2REJCMDdDQzRGMERBRDA2RUEwQUZFNDFC MDdBMUFFQUVGQjQ0NjRFAAoJELB6GurvtEZOmPkP/3iZoUX/6Ue0ou5eaW6L6uPH HANV6C5EBa0+vRWk6lY3908K0p8IJcnoitVbmvhp4/LDNLDXykd0ByysFdIng1kk bZ7/IejWlGTMXWKLBG2QJFX4qs+8sejfMl4Uy3KcYgARbnA0JtOs0Pn72JhzDyOr aiegl7naOlm/tJmlexKtLRIByY0MUU4qHR7HenCRfBDCSpYfTg/89ti5O62SmZvG ZKK9Ge9Y/Rd/b2ZHiUkmkPVucTFxl5axStduq1xKvPw5udrGxGjo+FiWsE9PT0Ao cCfTYCr2xSD2yG0QV05/WBYHE2xii1StOA+Ei8tqXluELNSU2+zqpK6UETbLzwz3 s9Ek4F+625+FLFRXzRoU6JVVucfDMEOV3jO5nCIDNzN5u7L4zDmsvIZSFP74RrI9 IJ34aDSlR4FKexCTFltCge0mvnLCkr/LrdUujMsie4v+9aBLA04GA8b+8mqRgW60 ZC/gVm60j1aX0tB4hKnp/5416h8orywkg5PqbBQlv+O1hndiGqIMoxuG3BVcUaWp PcMCZ07fYO8rC85JdkLTujUHWe61nuxBwW2mwwd+uAXXuVBZftrWamB2Q0y17gM/ uu8encFHRAdG2WWgfuqtwRaq1Kn7GfhUBpTZTXRRyxmFMH4I2NaqFFSuKRKWDBB3 DZSerETRSRAIEULTdgVO =/3oz -----END PGP SIGNATURE----- --Sig_/RDf9JWnQuW7bA1m=.j25FXz--