From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1NeMY9-0005vK-Ff for garchives@archives.gentoo.org; Mon, 08 Feb 2010 05:52:45 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 31888E0CBE; Mon, 8 Feb 2010 05:52:40 +0000 (UTC) Received: from mail-pz0-f195.google.com (mail-pz0-f195.google.com [209.85.222.195]) by pigeon.gentoo.org (Postfix) with ESMTP id 3F3D3E0CAA for ; Mon, 8 Feb 2010 05:52:28 +0000 (UTC) Received: by pzk33 with SMTP id 33so3135852pzk.2 for ; Sun, 07 Feb 2010 21:52:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:date:from:to:cc :subject:message-id:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=9rTNpFfgMZjMbybItUNHzjE9k1BZhSnECzh3QwOuTEo=; b=ktdGw5PoBJsdQTE+Yszy1KS/Mqhd/RQGtJ4bMcxEmZdEVjIxk2w6q5mkIKLeP9hYiH FDX1j3kl4/RxyGHWK6YTW1IFVQvsfLLY2FMMMWycQEgoTRLIEbFIyfhenX3x0ZfOCrZp W5eYKuWXS/8/4hT3es/roW29958NXbpCQ6Obc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=H3a1qAEteG5DEVLyq/ZaL875c2e6ktxArS1Wyk2S2l17HhAm5x3mkVz4h56P84w7Wc tiGWhaFAS5b8CSwOicGP7tWYxUWQt5aX1Y1c1vm81+Binnfa1g64Q9SKj3EpOOwY2ugU hyOkzE42cdS9vMw1siUH4xrrjtMy10GSL3NRk= Received: by 10.143.26.16 with SMTP id d16mr3948887wfj.287.1265608346511; Sun, 07 Feb 2010 21:52:26 -0800 (PST) Received: from smtp.gmail.com (c-98-210-130-131.hsd1.ca.comcast.net [98.210.130.131]) by mx.google.com with ESMTPS id 20sm6194504pzk.5.2010.02.07.21.52.22 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 07 Feb 2010 21:52:24 -0800 (PST) Received: by smtp.gmail.com (sSMTP sendmail emulation); Sun, 07 Feb 2010 21:50:37 -0800 Date: Sun, 7 Feb 2010 21:50:37 -0800 From: Brian Harring To: gentoo-dev@lists.gentoo.org Cc: robbat2@gentoo.org Subject: Re: [gentoo-dev] GLEP61 - Manifest2 compression Message-ID: <20100208055037.GD6052@hrair> References: <20100208010222.GB6052@hrair> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZARJHfwaSJQLOEUz" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) X-Archives-Salt: 74f5ca92-a624-4cfc-9cc0-0b688dfa3a25 X-Archives-Hash: e39b00a55daef90f706da1c6af1e79de --ZARJHfwaSJQLOEUz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Feb 08, 2010 at 05:23:03AM +0000, Robin H. Johnson wrote: > On Sun, Feb 07, 2010 at 05:02:22PM -0800, Brian Harring wrote: > > On Sun, Jan 31, 2010 at 10:04:40AM +0000, Robin H. Johnson wrote: > > > Changes: > > > - This GLEP can stand independently of GLEP58. > > > - Add XZ to compression types list. > > > - Move cutoff to 32KiB. Provide size example w/ 32KiB+gzip. > > > - Split specification into generation and validation. > > One concern w/ this glep- the intention seems to be to reduce on disk= =20 > > space requirements but the addition of compression raises questions=20 > > for rsync transferance of the manifests. > >=20 > > Have you done any testing to quantify how much of an increase in rsync= =20 > > bandwidth this will add? Specifically thinking about the metamanifest= =20 > > on this one. > > I think the best course of action is to end up generating the compressed > MetaManifests when we start generated the MetaManifests themselves, but > not placing them into the tree yet. Instead simply use them to measure > rsync transfer size impact on the generation server and produce > statistics to see if the cutoff could benefit from being altered, or if > the disk space should be wasted in favour of smaller transfer size. Works for me- I do strongly suspect that if we use compression we'll=20 be purely trading disk space decrease for the cost of transfering each=20 changed compressed manifest in full though... rsync + compression do=20 not get along at all. Either way, stats would be useful when you've got time. ~brian --ZARJHfwaSJQLOEUz Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.13 (GNU/Linux) iEYEARECAAYFAktvpi0ACgkQsiLx3HvNzgfHygCglvj1yWc6HttMUY/4O6Jk6zpP Vn0AnAteGQjX4kCUHR4MsrAZoDDrJQKO =wt/U -----END PGP SIGNATURE----- --ZARJHfwaSJQLOEUz--