Re: [gentoo-dev] GLEP61 - Manifest2 compression

["Robin H. Johnson" <robbat2@gentoo.org>]

On Sun, Feb 07, 2010 at 05:02:22PM -0800, Brian Harring wrote:
> On Sun, Jan 31, 2010 at 10:04:40AM +0000, Robin H. Johnson wrote:
> > Changes:
> > - This GLEP can stand independently of GLEP58.
> > - Add XZ to compression types list.
> > - Move cutoff to 32KiB. Provide size example w/ 32KiB+gzip.
> > - Split specification into generation and validation.
> One concern w/ this glep- the intention seems to be to reduce on disk 
> space requirements but the addition of compression raises questions 
> for rsync transferance of the manifests.
> 
> Have you done any testing to quantify how much of an increase in rsync 
> bandwidth this will add?  Specifically thinking about the metamanifest 
> on this one.
The top-level MetaManifest, in the case of fully split (eg a Manifest in
every 1st-level directory $CAT/Manifest and the other dirs), is only
33KiB.

21 existing packages have Manifests larger than 32KiB, texlive stuff
come in the worst here.

I do agree, that depending on the block alignment, there is an increase
in transfer size in some cases, but I have not conducted rigorous tests
to work out long-term statistics on changes.

The more aggressively that Manifests are added to each subdirectory
tree, the less Manifest2 compression is actually required, as the
individual Manifests are more likely to fit within the size limit.

With the 1st-level case again, here's a size count breakdown for the new
(Meta)Manifests:
>=32KiB: 27
>=64KiB: 10
>=128KiB: 3 (dev-perl @ 179KiB, metadata/ @ 5.2MiB, profiles/ @ 300KiB)

I think the best course of action is to end up generating the compressed
MetaManifests when we start generated the MetaManifests themselves, but
not placing them into the tree yet. Instead simply use them to measure
rsync transfer size impact on the generation server and produce
statistics to see if the cutoff could benefit from being altered, or if
the disk space should be wasted in favour of smaller transfer size.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85