[gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2486 bytes --]

Hello, all.

I'm using squashfs to hold my Gentoo repositories on all of my systems
for some time. As you probably know, this allows me to save space while
keeping portage fast. However, it makes updating the tree quite
burdensome and time-consuming.

We're already hosting daily gx86 tarballs on our mirrors, and deltas
made using diffball. Those can be used with Zac's emerge-delta-webrsync
to get daily updates done with minimal network overhead. Sadly, it
takes the whole process even more time consuming :).

Therefore, I'd like to suggest an alternative solution that could help
out Gentoo users that use squashfs for gx86 and would like to be able
to get daily updates fast and easy.

The idea is to host -- along with the tarballs -- daily squashfs images
of gx86 in a chosen format. Additionally, the images would come with
deltas made using xdelta3 or a similar tool. Those deltas -- with
a slight download overhead -- would allow very fast updates
of the squashfs.

Now some numbers. I did some tests 'converting' late gx86 daily
tarballs to squashfs. I've used squashfs 4.2 with LZO compression
since it's quite good and very fast.

96M	portage-20140108.sqfs
96M	portage-20140109.sqfs
96M	portage-20140110.sqfs
96M	portage-20140111.sqfs
96M	portage-20140112.sqfs
96M	portage-20140113.sqfs
97M	portage-20140114.sqfs
97M	portage-20140115.sqfs

For deltas, I've used xdelta3 with max compression (-9) and djw
secondary compression (it gave ~0.1M smaller files than fgk
and ~0.5M gain than with no secondary compression).

4,9M	portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
6,3M	portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
5,6M	portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
8,9M	portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
6,3M	portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
7,8M	portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
8,5M	portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw

As you can see, the deltas are quite large compared to the actual
changes. However, we could have expected that since we're diffing
a compressed filesystem. What's important, however, is that applying
it takes ~2.5 second on my 2 GHz Athlon64.

So, even with the extra download time, the update is much faster
than recreating the squashfs. And unlike some types of unionfs,
it doesn't come with extra runtime slowdown.

What do you think?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 17/01/14 11:27 AM, Michał Górny wrote:
> Hello, all.
> 
> I'm using squashfs to hold my Gentoo repositories on all of my
> systems for some time. As you probably know, this allows me to save
> space while keeping portage fast. However, it makes updating the
> tree quite burdensome and time-consuming.
> 
> We're already hosting daily gx86 tarballs on our mirrors, and
> deltas made using diffball. Those can be used with Zac's
> emerge-delta-webrsync to get daily updates done with minimal
> network overhead. Sadly, it takes the whole process even more time
> consuming :).
> 
> Therefore, I'd like to suggest an alternative solution that could
> help out Gentoo users that use squashfs for gx86 and would like to
> be able to get daily updates fast and easy.
> 
> The idea is to host -- along with the tarballs -- daily squashfs
> images of gx86 in a chosen format. Additionally, the images would
> come with deltas made using xdelta3 or a similar tool. Those deltas
> -- with a slight download overhead -- would allow very fast
> updates of the squashfs.
> 
> Now some numbers. I did some tests 'converting' late gx86 daily 
> tarballs to squashfs. I've used squashfs 4.2 with LZO compression 
> since it's quite good and very fast.
> 
> 96M	portage-20140108.sqfs 96M	portage-20140109.sqfs 96M
> portage-20140110.sqfs 96M	portage-20140111.sqfs 96M
> portage-20140112.sqfs 96M	portage-20140113.sqfs 97M
> portage-20140114.sqfs 97M	portage-20140115.sqfs
> 
> For deltas, I've used xdelta3 with max compression (-9) and djw 
> secondary compression (it gave ~0.1M smaller files than fgk and
> ~0.5M gain than with no secondary compression).
> 
> 4,9M	portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw 6,3M
> portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw 5,6M
> portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw 8,9M
> portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw 6,3M
> portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw 7,8M
> portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw 8,5M
> portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> 
> As you can see, the deltas are quite large compared to the actual 
> changes. However, we could have expected that since we're diffing a
> compressed filesystem. What's important, however, is that applying 
> it takes ~2.5 second on my 2 GHz Athlon64.
> 
> So, even with the extra download time, the update is much faster 
> than recreating the squashfs. And unlike some types of unionfs, it
> doesn't come with extra runtime slowdown.
> 
> What do you think?
> 

PLEASE DO!  This sounds fantastic, and is something i've been
considering proposing for some time.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlLZWoMACgkQ2ugaI38ACPAb0gEAwFPWdtI5J8l9QH1YTrKe2Mbm
OwPL4wGg9ORaHv0FkVcA/iLsP/z3uz3sJoWciR5ZCZ73HblyDgH/flAhakNhl3NZ
=ool3
-----END PGP SIGNATURE-----

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 179 bytes --]

Michał Górny wrote:
> What do you think?

Excellent. I think it could quickly become the prefered protage
storage format, although a loopback mount is needed.


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

Dnia 2014-01-17, o godz. 17:51:41
Peter Stuge <peter@stuge.se> napisał(a):

> Michał Górny wrote:
> > What do you think?
> 
> Excellent. I think it could quickly become the prefered protage
> storage format, although a loopback mount is needed.

You can use sys-fs/squashfuse :).

Though honestly I'd prefer if portage was able to take squashfs image
path in repos.conf and mount it locally for build-time. This will have
the extra advantage that we wouldn't have to worry about remounting it
after sync.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

Michał Górny wrote:
> > > What do you think?
> > 
> > Excellent. I think it could quickly become the prefered protage
> > storage format, although a loopback mount is needed.
> 
> You can use sys-fs/squashfuse :).
> 
> Though honestly I'd prefer if portage was able to take squashfs image
> path in repos.conf and mount it locally for build-time. This will have
> the extra advantage that we wouldn't have to worry about remounting it
> after sync.

Or read it directly without mounting.


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[vivo75]

On 01/17/14 17:27, Michał Górny wrote:
> Hello, all.
>
> I'm using squashfs to hold my Gentoo repositories on all of my systems
> for some time. As you probably know, this allows me to save space while
> keeping portage fast. However, it makes updating the tree quite
> burdensome and time-consuming.
Me too and you have my total support (maybe I've even proposed this
before to te list)

[snip]

>
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing
> a compressed filesystem. What's important, however, is that applying
> it takes ~2.5 second on my 2 GHz Athlon64.

Have you tried to give an order (always the same) to the compressed files?
It could give an advantage, tough it may be limited to 2^16 files the
option is -sort <sort_file>


thanks for it,
Francesco

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2808 bytes --]

On Fri, Jan 17, 2014 at 8:27 AM, Michał Górny <mgorny@gentoo.org> wrote:

> Hello, all.
>
> I'm using squashfs to hold my Gentoo repositories on all of my systems
> for some time. As you probably know, this allows me to save space while
> keeping portage fast. However, it makes updating the tree quite
> burdensome and time-consuming.
>
> We're already hosting daily gx86 tarballs on our mirrors, and deltas
> made using diffball. Those can be used with Zac's emerge-delta-webrsync
> to get daily updates done with minimal network overhead. Sadly, it
> takes the whole process even more time consuming :).
>
> Therefore, I'd like to suggest an alternative solution that could help
> out Gentoo users that use squashfs for gx86 and would like to be able
> to get daily updates fast and easy.
>
> The idea is to host -- along with the tarballs -- daily squashfs images
> of gx86 in a chosen format. Additionally, the images would come with
> deltas made using xdelta3 or a similar tool. Those deltas -- with
> a slight download overhead -- would allow very fast updates
> of the squashfs.
>
> Now some numbers. I did some tests 'converting' late gx86 daily
> tarballs to squashfs. I've used squashfs 4.2 with LZO compression
> since it's quite good and very fast.
>
> 96M     portage-20140108.sqfs
> 96M     portage-20140109.sqfs
> 96M     portage-20140110.sqfs
> 96M     portage-20140111.sqfs
> 96M     portage-20140112.sqfs
> 96M     portage-20140113.sqfs
> 97M     portage-20140114.sqfs
> 97M     portage-20140115.sqfs
>
> For deltas, I've used xdelta3 with max compression (-9) and djw
> secondary compression (it gave ~0.1M smaller files than fgk
> and ~0.5M gain than with no secondary compression).
>
> 4,9M    portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> 6,3M    portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> 5,6M    portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
> 8,9M    portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
> 6,3M    portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
> 7,8M    portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
> 8,5M    portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
>
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing
> a compressed filesystem. What's important, however, is that applying
> it takes ~2.5 second on my 2 GHz Athlon64.
>

It wasn't clear to me, are these trees with metadata included?

-A


>
> So, even with the extra download time, the update is much faster
> than recreating the squashfs. And unlike some types of unionfs,
> it doesn't come with extra runtime slowdown.
>
> What do you think?
>
> --
> Best regards,
> Michał Górny
>

[-- Attachment #2: Type: text/html, Size: 3535 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2949 bytes --]

Dnia 2014-01-17, o godz. 10:18:58
Alec Warner <antarus@gentoo.org> napisał(a):

> On Fri, Jan 17, 2014 at 8:27 AM, Michał Górny <mgorny@gentoo.org> wrote:
> 
> > Hello, all.
> >
> > I'm using squashfs to hold my Gentoo repositories on all of my systems
> > for some time. As you probably know, this allows me to save space while
> > keeping portage fast. However, it makes updating the tree quite
> > burdensome and time-consuming.
> >
> > We're already hosting daily gx86 tarballs on our mirrors, and deltas
> > made using diffball. Those can be used with Zac's emerge-delta-webrsync
> > to get daily updates done with minimal network overhead. Sadly, it
> > takes the whole process even more time consuming :).
> >
> > Therefore, I'd like to suggest an alternative solution that could help
> > out Gentoo users that use squashfs for gx86 and would like to be able
> > to get daily updates fast and easy.
> >
> > The idea is to host -- along with the tarballs -- daily squashfs images
> > of gx86 in a chosen format. Additionally, the images would come with
> > deltas made using xdelta3 or a similar tool. Those deltas -- with
> > a slight download overhead -- would allow very fast updates
> > of the squashfs.
> >
> > Now some numbers. I did some tests 'converting' late gx86 daily
> > tarballs to squashfs. I've used squashfs 4.2 with LZO compression
> > since it's quite good and very fast.
> >
> > 96M     portage-20140108.sqfs
> > 96M     portage-20140109.sqfs
> > 96M     portage-20140110.sqfs
> > 96M     portage-20140111.sqfs
> > 96M     portage-20140112.sqfs
> > 96M     portage-20140113.sqfs
> > 97M     portage-20140114.sqfs
> > 97M     portage-20140115.sqfs
> >
> > For deltas, I've used xdelta3 with max compression (-9) and djw
> > secondary compression (it gave ~0.1M smaller files than fgk
> > and ~0.5M gain than with no secondary compression).
> >
> > 4,9M    portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> > 6,3M    portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> > 5,6M    portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
> > 8,9M    portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
> > 6,3M    portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
> > 7,8M    portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
> > 8,5M    portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> >
> > As you can see, the deltas are quite large compared to the actual
> > changes. However, we could have expected that since we're diffing
> > a compressed filesystem. What's important, however, is that applying
> > it takes ~2.5 second on my 2 GHz Athlon64.
> >
> 
> It wasn't clear to me, are these trees with metadata included?

Yes, full metadata with md5-cache. That's the same thing you get via
'emerge --sync'. And that's why the deltas are so big -- I recall three
big cache updates this week.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

[gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Duncan]

Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:

> Now some numbers. I did some tests 'converting' late gx86 daily tarballs
> to squashfs. I've used squashfs 4.2 with LZO compression since it's
> quite good and very fast.
> 
> 96M	portage-20140108.sqfs
[...]
> 97M	portage-20140114.sqfs
> 97M	portage-20140115.sqfs
> 
> For deltas [...]
> 
> 4,9M	portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> 6,3M	portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
[...]
> 8,5M	portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> 
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing a
> compressed filesystem. What's important, however, is that applying it
> takes ~2.5 second on my 2 GHz Athlon64.

And... eyeballing a 6 MiB average, diffs are ~1/16 the full squashfs 
size, perhaps a bit larger.  So people updating once a week or even about 
every 10 days would see a bandwidth savings, provided the sync script was 
intelligent enough to apply updates serially.

The breakover point would be roughly an update every two weeks, or twice 
a month, at which point just downloading a new full squashfs would be 
easier, at about the same bandwidth.

> What do you think?

How does this, particularly the metadata cache, interact with overlays?  
That's /my/ big question.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2119 bytes --]

Dnia 2014-01-17, o godz. 19:19:14
Duncan <1i5t5.duncan@cox.net> napisał(a):

> Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:
> 
> > Now some numbers. I did some tests 'converting' late gx86 daily tarballs
> > to squashfs. I've used squashfs 4.2 with LZO compression since it's
> > quite good and very fast.
> > 
> > 96M	portage-20140108.sqfs
> [...]
> > 97M	portage-20140114.sqfs
> > 97M	portage-20140115.sqfs
> > 
> > For deltas [...]
> > 
> > 4,9M	portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> > 6,3M	portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> [...]
> > 8,5M	portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> > 
> > As you can see, the deltas are quite large compared to the actual
> > changes. However, we could have expected that since we're diffing a
> > compressed filesystem. What's important, however, is that applying it
> > takes ~2.5 second on my 2 GHz Athlon64.
> 
> And... eyeballing a 6 MiB average, diffs are ~1/16 the full squashfs 
> size, perhaps a bit larger.  So people updating once a week or even about 
> every 10 days would see a bandwidth savings, provided the sync script was 
> intelligent enough to apply updates serially.
> 
> The breakover point would be roughly an update every two weeks, or twice 
> a month, at which point just downloading a new full squashfs would be 
> easier, at about the same bandwidth.

Yes, that's the initial goal. The update code would catch that
condition and perform full fetch instead.

However, it may be actually beneficial to provide other durations, like
weekly deltas. In my tests, the daily updates for this week summed up
to almost 50M while the weekly was barely 20M.

> > What do you think?
> 
> How does this, particularly the metadata cache, interact with overlays?  
> That's /my/ big question.

The same way as usual. Nothing special happens unless you override
eclasses via overlays. If you do that, you need unionfs to save
the cache updates -- but then, it has no point for you to use squashfs.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Markos Chandras]

On 01/17/2014 04:27 PM, Michał Górny wrote:
> Hello, all.
> 
> I'm using squashfs to hold my Gentoo repositories on all of my systems
> for some time. As you probably know, this allows me to save space while
> keeping portage fast. However, it makes updating the tree quite
> burdensome and time-consuming.
> 
> We're already hosting daily gx86 tarballs on our mirrors, and deltas
> made using diffball. Those can be used with Zac's emerge-delta-webrsync
> to get daily updates done with minimal network overhead. Sadly, it
> takes the whole process even more time consuming :).
> 
> Therefore, I'd like to suggest an alternative solution that could help
> out Gentoo users that use squashfs for gx86 and would like to be able
> to get daily updates fast and easy.
> 
> The idea is to host -- along with the tarballs -- daily squashfs images
> of gx86 in a chosen format. Additionally, the images would come with
> deltas made using xdelta3 or a similar tool. Those deltas -- with
> a slight download overhead -- would allow very fast updates
> of the squashfs.
> 
> Now some numbers. I did some tests 'converting' late gx86 daily
> tarballs to squashfs. I've used squashfs 4.2 with LZO compression
> since it's quite good and very fast.
> 
> 96M	portage-20140108.sqfs
> 96M	portage-20140109.sqfs
> 96M	portage-20140110.sqfs
> 96M	portage-20140111.sqfs
> 96M	portage-20140112.sqfs
> 96M	portage-20140113.sqfs
> 97M	portage-20140114.sqfs
> 97M	portage-20140115.sqfs
> 
> For deltas, I've used xdelta3 with max compression (-9) and djw
> secondary compression (it gave ~0.1M smaller files than fgk
> and ~0.5M gain than with no secondary compression).
> 
> 4,9M	portage-20140108.sqfs-portage-20140109.sqfs.vcdiff.djw
> 6,3M	portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> 5,6M	portage-20140110.sqfs-portage-20140111.sqfs.vcdiff.djw
> 8,9M	portage-20140111.sqfs-portage-20140112.sqfs.vcdiff.djw
> 6,3M	portage-20140112.sqfs-portage-20140113.sqfs.vcdiff.djw
> 7,8M	portage-20140113.sqfs-portage-20140114.sqfs.vcdiff.djw
> 8,5M	portage-20140114.sqfs-portage-20140115.sqfs.vcdiff.djw
> 
> As you can see, the deltas are quite large compared to the actual
> changes. However, we could have expected that since we're diffing
> a compressed filesystem. What's important, however, is that applying
> it takes ~2.5 second on my 2 GHz Athlon64.
> 
> So, even with the extra download time, the update is much faster
> than recreating the squashfs. And unlike some types of unionfs,
> it doesn't come with extra runtime slowdown.
> 
> What do you think?
> 

+1 I like the idea

-- 
Regards,
Markos Chandras

[gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Duncan]

Michał Górny posted on Fri, 17 Jan 2014 20:30:00 +0100 as excerpted:

> Dnia 2014-01-17, o godz. 19:19:14 Duncan <1i5t5.duncan@cox.net>
> napisał(a):
> 
>> Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:
>> 
>> > 96M	portage-20140108.sqfs

>> > For deltas [...]
>> > 
>> > 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw

>> > applying it takes ~2.5 second on my 2 GHz Athlon64.
>> 
>> diffs are ~1/16 the full squashfs size[.] So people updating once a
>> week [or] 10 days would see a bandwidth savings, provided the sync
>> script was intelligent enough to apply updates serially.
>> 
>> The breakover point would be roughly an update every two weeks, or
>> twice a month
> 
> However, it may be actually beneficial to provide other durations, like
> weekly deltas. In my tests, the daily updates for this week summed up to
> almost 50M while the weekly was barely 20M.

That's useful additional data.  Thanks.

And yes, a weekly delta would be quite useful, taking the breakover point 
out to about a month or so.  Practically speaking, I'd guess most 
gentooers update once a month or more, so that should cover the vast 
majority.  Beyond a month, just downloading a new full squashfs makes as 
much sense anyway, and as the cutover would be automated, users on the 
borderline wouldn't have to worry about whether they should just do the 
normal sync or download an entirely new tarball, as they now need to do, 
if they even bother at all.  For those users, it'd be an even BIGGER win.

=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2094 bytes --]

Dnia 2014-01-17, o godz. 23:31:32
Duncan <1i5t5.duncan@cox.net> napisał(a):

> Michał Górny posted on Fri, 17 Jan 2014 20:30:00 +0100 as excerpted:
> 
> > Dnia 2014-01-17, o godz. 19:19:14 Duncan <1i5t5.duncan@cox.net>
> > napisał(a):
> > 
> >> Michał Górny posted on Fri, 17 Jan 2014 17:27:30 +0100 as excerpted:
> >> 
> >> > 96M	portage-20140108.sqfs
> 
> >> > For deltas [...]
> >> > 
> >> > 6,3M portage-20140109.sqfs-portage-20140110.sqfs.vcdiff.djw
> 
> >> > applying it takes ~2.5 second on my 2 GHz Athlon64.
> >> 
> >> diffs are ~1/16 the full squashfs size[.] So people updating once a
> >> week [or] 10 days would see a bandwidth savings, provided the sync
> >> script was intelligent enough to apply updates serially.
> >> 
> >> The breakover point would be roughly an update every two weeks, or
> >> twice a month
> > 
> > However, it may be actually beneficial to provide other durations, like
> > weekly deltas. In my tests, the daily updates for this week summed up to
> > almost 50M while the weekly was barely 20M.
> 
> That's useful additional data.  Thanks.
> 
> And yes, a weekly delta would be quite useful, taking the breakover point 
> out to about a month or so.  Practically speaking, I'd guess most 
> gentooers update once a month or more, so that should cover the vast 
> majority.  Beyond a month, just downloading a new full squashfs makes as 
> much sense anyway, and as the cutover would be automated, users on the 
> borderline wouldn't have to worry about whether they should just do the 
> normal sync or download an entirely new tarball, as they now need to do, 
> if they even bother at all.  For those users, it'd be an even BIGGER win.

Well, I've got even a better idea and I'll try to provide direct deltas
from N days back to the newest snapshot. Since the tree changes a lot,
those deltas will be smaller to download ;).

Now, does anyone have an old portage-XXXXYYZZ.tar.{bz2,xz} snapshot? I
need the official one from our mirrors, preferably 3-4 months old.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[heroxbd]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

Michał Górny <mgorny@gentoo.org> writes:

> However, it may be actually beneficial to provide other durations, like
> weekly deltas. In my tests, the daily updates for this week summed up
> to almost 50M while the weekly was barely 20M.

Is there a way to merge the deltas without the original squashfs? how
fast is the delta generation? It could be done on the server side on the
fly and cached.

Or we provide a set of 16,8,4,2,1 day deltas, then 

 16d: 1 piece needed
 8d: 2 needed
 4d: 4 needed
 2d: 8 needed
 1d: 16 needed

The total of 31 pieces will cover a month (31 days) with at most 5
deltas to be downloaded.

e.g. If the system is 19 days old, then we download a 1d, 2d, and 16d.

Cheers,
Benda

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Thomas D.]

Hi,

Michał Górny wrote:
> Now, does anyone have an old portage-XXXXYYZZ.tar.{bz2,xz} snapshot? I
> need the official one from our mirrors, preferably 3-4 months old.

<https://dl.dropboxusercontent.com/s/ldh8ie2zzdpnc57/portage-20121228.tar.bz2>

<https://dl.dropboxusercontent.com/s/v0sm7xc5k5vtu48/portage-20131018.tar.bz2>


-- 
Regards,
Thomas

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Alex Xu]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

On 17/01/14 08:08 PM, heroxbd@gentoo.org wrote:
> Michał Górny <mgorny@gentoo.org> writes:
> 
>> However, it may be actually beneficial to provide other durations, like
>> weekly deltas. In my tests, the daily updates for this week summed up
>> to almost 50M while the weekly was barely 20M.
> 
> Is there a way to merge the deltas without the original squashfs?

Uh... what? How would that work?

> how fast is the delta generation?

Very.

> It could be done on the server side on the fly and cached.

Too much work.


> Or we provide a set of 16,8,4,2,1 day deltas, then 
> 
>  16d: 1 piece needed
>  8d: 2 needed
>  4d: 4 needed
>  2d: 8 needed
>  1d: 16 needed
> 
> The total of 31 pieces will cover a month (31 days) with at most 5
> deltas to be downloaded.
> 
> e.g. If the system is 19 days old, then we download a 1d, 2d, and 16d.
> 

This doesn't really help. Consider that deltas require both a *start*
and *end*. It's not as simple as adding it all up, since you would need
a 19-3d, then 3-1d, then 1-0d.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[heroxbd]

[-- Attachment #1: Type: text/plain, Size: 385 bytes --]

Alex Xu <alex_y_xu@yahoo.ca> writes:

> This doesn't really help. Consider that deltas require both a *start*
> and *end*. It's not as simple as adding it all up, since you would need
> a 19-3d, then 3-1d, then 1-0d.

In this case, 19-18d, 18-16d, 16-0d, instead. The largest delta is
always used for the most recent. That's why we need 8 pieces of 2d
delta and 16 pieces of 1d delta.

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1750 bytes --]

Dnia 2014-01-18, o godz. 10:08:13
heroxbd@gentoo.org napisał(a):

> Michał Górny <mgorny@gentoo.org> writes:
> 
> > However, it may be actually beneficial to provide other durations, like
> > weekly deltas. In my tests, the daily updates for this week summed up
> > to almost 50M while the weekly was barely 20M.
> 
> Is there a way to merge the deltas without the original squashfs?

You mean get two deltas and merge them to a single one? I have no idea,
honestly :). It may be possible but xdelta3 is hard to use.

> how fast is the delta generation?

With max compression on 2 GHz amd64, about 15s for a daily delta.
A weekly takes 25s.

> Or we provide a set of 16,8,4,2,1 day deltas, then 
> 
>  16d: 1 piece needed
>  8d: 2 needed
>  4d: 4 needed
>  2d: 8 needed
>  1d: 16 needed
> 
> The total of 31 pieces will cover a month (31 days) with at most 5
> deltas to be downloaded.
> 
> e.g. If the system is 19 days old, then we download a 1d, 2d, and 16d.

Unnecessarily complex and not worth the effort.

Please remember that we're generating deltas against compressed image
with metadata cache included. Those are far from optimal, and with your
solution you'll quickly run over the original squashfs size.

I'm currently working on another solution that should be a bit harder
for the server yet much better for users. It assumes we're keeping
deltas from N days back to today, i.e.: n-1..n, n-2..n, n-3..n.

Any of those deltas is much smaller than two dailies (I'll provide
exact numbers when my playground finishes, that is in few hours). You
just download the one matching your current image and apply it,
and TADAAM, you've got today's snapshot :).

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

Dnia 2014-01-18, o godz. 02:28:52
"Thomas D." <whissi@whissi.de> napisał(a):

> Hi,
> 
> Michał Górny wrote:
> > Now, does anyone have an old portage-XXXXYYZZ.tar.{bz2,xz} snapshot? I
> > need the official one from our mirrors, preferably 3-4 months old.
> 
> <https://dl.dropboxusercontent.com/s/v0sm7xc5k5vtu48/portage-20131018.tar.bz2>

Well, thanks but a bit too late :). I already looked through all of our
mirrors and found July 2013 on one of them. And I used the deltas to
update it to... guess what... exactly 18 October :D. Coincidence?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Christopher Head]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

On Fri, 17 Jan 2014 19:53:44 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 2014-01-17, o godz. 10:18:58
> Alec Warner <antarus@gentoo.org> napisał(a):
> 
> > On Fri, Jan 17, 2014 at 8:27 AM, Michał Górny <mgorny@gentoo.org>
> > wrote:
> > 
> > > Hello, all.
> > >
> > > I'm using squashfs to hold my Gentoo repositories on all of my
> > > systems for some time. As you probably know, this allows me to
> > > save space while keeping portage fast. However, it makes updating
> > > the tree quite burdensome and time-consuming.
> 
> Yes, full metadata with md5-cache. That's the same thing you get via
> 'emerge --sync'. And that's why the deltas are so big -- I recall
> three big cache updates this week.
> 

I would absolutely use this on my machines.
-- 
Christopher Head

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

Hi again.

If someone is interested in the results of my tests and benchmarks,
I've uploaded the initial version of my article on the topic in our
dev-space.

http://dev.gentoo.org/~mgorny/tmp/squashfs-deltas.pdf

I am terribly busy with the uni right now so it will take some time
before I continue working on it. I will try to provide a final
specification for the first attempt at the idea and ask infra if they
are ready to sacrifice the hardware for it.

Further possible improvements:

1. switch to LZ4 (stronger compression, even faster) -- will require
a newer kernel (3.14?),

2. dedicated SquashFS delta tool -- I'm working on it but
the format seems to be poorly documented so it will take some time :).

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Steev Klimaszewski]

On Sun, 2014-01-26 at 21:00 +0100, Michał Górny wrote:
> Hi again.
> 
> If someone is interested in the results of my tests and benchmarks,
> I've uploaded the initial version of my article on the topic in our
> dev-space.
> 
> http://dev.gentoo.org/~mgorny/tmp/squashfs-deltas.pdf
> 
> I am terribly busy with the uni right now so it will take some time
> before I continue working on it. I will try to provide a final
> specification for the first attempt at the idea and ask infra if they
> are ready to sacrifice the hardware for it.
> 
> Further possible improvements:
> 
> 1. switch to LZ4 (stronger compression, even faster) -- will require
> a newer kernel (3.14?),
> 

While the stronger compression, and being faster is definitely nice,
having portage on squashfs is really nice on ARM devices, however the
number of them that have a decently running kernel newer than 3.8 are
few and far between, so I'd like to ask that this be held off as long as
possible.  I know these are just possible improvements, but doing so
would definitely alienate a really good place where this would shine.

> 2. dedicated SquashFS delta tool -- I'm working on it but
> the format seems to be poorly documented so it will take some time :).
> 

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[vivo75]

On 01/27/14 08:35, Steev Klimaszewski wrote:
> On Sun, 2014-01-26 at 21:00 +0100, Michał Górny wrote:
>> Hi again.
>>
>> If someone is interested in the results of my tests and benchmarks,
>> I've uploaded the initial version of my article on the topic in our
>> dev-space.
>>
>> http://dev.gentoo.org/~mgorny/tmp/squashfs-deltas.pdf
>>
>> I am terribly busy with the uni right now so it will take some time
>> before I continue working on it. I will try to provide a final
>> specification for the first attempt at the idea and ask infra if they
>> are ready to sacrifice the hardware for it.
>>
>> Further possible improvements:
>>
>> 1. switch to LZ4 (stronger compression, even faster) -- will require
>> a newer kernel (3.14?),
>>
it should be in kernel 3.11 "windows for workgroups" release (check anyway)

> While the stronger compression, and being faster is definitely nice,
> having portage on squashfs is really nice on ARM devices, however the
> number of them that have a decently running kernel newer than 3.8 are
> few and far between, so I'd like to ask that this be held off as long as
> possible.  I know these are just possible improvements, but doing so
> would definitely alienate a really good place where this would shine.

yes, there are good reasons also for amd64

>
>> 2. dedicated SquashFS delta tool -- I'm working on it but
>> the format seems to be poorly documented so it will take some time :).
>>
>
>

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1517 bytes --]

Dnia 2014-01-27, o godz. 01:35:52
Steev Klimaszewski <steev@gentoo.org> napisał(a):

> On Sun, 2014-01-26 at 21:00 +0100, Michał Górny wrote:
> > Hi again.
> > 
> > If someone is interested in the results of my tests and benchmarks,
> > I've uploaded the initial version of my article on the topic in our
> > dev-space.
> > 
> > http://dev.gentoo.org/~mgorny/tmp/squashfs-deltas.pdf
> > 
> > I am terribly busy with the uni right now so it will take some time
> > before I continue working on it. I will try to provide a final
> > specification for the first attempt at the idea and ask infra if they
> > are ready to sacrifice the hardware for it.
> > 
> > Further possible improvements:
> > 
> > 1. switch to LZ4 (stronger compression, even faster) -- will require
> > a newer kernel (3.14?),
> > 
> 
> While the stronger compression, and being faster is definitely nice,
> having portage on squashfs is really nice on ARM devices, however the
> number of them that have a decently running kernel newer than 3.8 are
> few and far between, so I'd like to ask that this be held off as long as
> possible.  I know these are just possible improvements, but doing so
> would definitely alienate a really good place where this would shine.

I think that if we decide to do the switch, we will host multiple
formats at least for some time. It won't be really helpful to prevent
people from upgrading their kernel due to new repo archive format :).

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

Dnia 2014-01-27, o godz. 14:53:34
"vivo75@gmail.com" <vivo75@gmail.com> napisał(a):

> On 01/27/14 08:35, Steev Klimaszewski wrote:
> > On Sun, 2014-01-26 at 21:00 +0100, Michał Górny wrote:
> >> Hi again.
> >>
> >> If someone is interested in the results of my tests and benchmarks,
> >> I've uploaded the initial version of my article on the topic in our
> >> dev-space.
> >>
> >> http://dev.gentoo.org/~mgorny/tmp/squashfs-deltas.pdf
> >>
> >> I am terribly busy with the uni right now so it will take some time
> >> before I continue working on it. I will try to provide a final
> >> specification for the first attempt at the idea and ask infra if they
> >> are ready to sacrifice the hardware for it.
> >>
> >> Further possible improvements:
> >>
> >> 1. switch to LZ4 (stronger compression, even faster) -- will require
> >> a newer kernel (3.14?),
> >>
> it should be in kernel 3.11 "windows for workgroups" release (check anyway)

That's just the LZ4 library code. We additionally need the SquashFS
support code. It has been introduced in squashfs-tools lately
(4.2_p20140119 has it, though disabled by ebuild) and I don't see it
in the kernel's master branch yet.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] RFC: Hosting daily gx86 squashfs images and deltas

[Jeroen Roovers]

On Mon, 27 Jan 2014 16:52:09 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> That's just the LZ4 library code. We additionally need the SquashFS
> support code. It has been introduced in squashfs-tools lately
> (4.2_p20140119 has it, though disabled by ebuild) and I don't see it
> in the kernel's master branch yet.

I'll be glad to add that squashfs-tools support for you shortly.


Regards,
     jer