From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1NeM66-0003kp-UW for garchives@archives.gentoo.org; Mon, 08 Feb 2010 05:23:47 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id D46A0E0AB4; Mon, 8 Feb 2010 05:23:32 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by pigeon.gentoo.org (Postfix) with ESMTP id 525A4E08E6 for ; Mon, 8 Feb 2010 05:23:08 +0000 (UTC) Received: from mail.isohunt.com (b01.ext.isohunt.com [208.71.112.51]) by smtp.gentoo.org (Postfix) with ESMTP id C6FDE1B41E7 for ; Mon, 8 Feb 2010 05:23:07 +0000 (UTC) Received: (qmail 25953 invoked from network); 8 Feb 2010 05:23:05 -0000 Received: from tsi-static.orbis-terrarum.net (HELO grubbs.orbis-terrarum.net) (76.10.188.108) by mail.isohunt.com (qpsmtpd/0.33-dev on beta01) with (CAMELLIA256-SHA encrypted) ESMTPS; Mon, 08 Feb 2010 05:23:05 +0000 Received: (qmail 17788 invoked by uid 10000); 8 Feb 2010 05:23:03 -0000 Date: Mon, 8 Feb 2010 05:23:03 +0000 From: "Robin H. Johnson" To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] GLEP61 - Manifest2 compression Message-ID: References: <20100208010222.GB6052@hrair> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100208010222.GB6052@hrair> User-Agent: Mutt/1.5.20 (2009-06-14) X-Archives-Salt: e92aa878-868e-43f8-92f4-dd1f1b741f59 X-Archives-Hash: 26d3d2e8edbfe635840e519e98d9b75d On Sun, Feb 07, 2010 at 05:02:22PM -0800, Brian Harring wrote: > On Sun, Jan 31, 2010 at 10:04:40AM +0000, Robin H. Johnson wrote: > > Changes: > > - This GLEP can stand independently of GLEP58. > > - Add XZ to compression types list. > > - Move cutoff to 32KiB. Provide size example w/ 32KiB+gzip. > > - Split specification into generation and validation. > One concern w/ this glep- the intention seems to be to reduce on disk > space requirements but the addition of compression raises questions > for rsync transferance of the manifests. > > Have you done any testing to quantify how much of an increase in rsync > bandwidth this will add? Specifically thinking about the metamanifest > on this one. The top-level MetaManifest, in the case of fully split (eg a Manifest in every 1st-level directory $CAT/Manifest and the other dirs), is only 33KiB. 21 existing packages have Manifests larger than 32KiB, texlive stuff come in the worst here. I do agree, that depending on the block alignment, there is an increase in transfer size in some cases, but I have not conducted rigorous tests to work out long-term statistics on changes. The more aggressively that Manifests are added to each subdirectory tree, the less Manifest2 compression is actually required, as the individual Manifests are more likely to fit within the size limit. With the 1st-level case again, here's a size count breakdown for the new (Meta)Manifests: >=32KiB: 27 >=64KiB: 10 >=128KiB: 3 (dev-perl @ 179KiB, metadata/ @ 5.2MiB, profiles/ @ 300KiB) I think the best course of action is to end up generating the compressed MetaManifests when we start generated the MetaManifests themselves, but not placing them into the tree yet. Instead simply use them to measure rsync transfer size impact on the generation server and produce statistics to see if the cutoff could benefit from being altered, or if the disk space should be wasted in favour of smaller transfer size. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee & Infrastructure Lead E-Mail : robbat2@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85