From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.gentoo.org ([140.105.134.102] helo=robin.gentoo.org) by nuthatch.gentoo.org with esmtp (Exim 4.50) id 1EQ7Bc-0001zN-Gf for garchives@archives.gentoo.org; Thu, 13 Oct 2005 17:48:13 +0000 Received: from robin.gentoo.org (localhost [127.0.0.1]) by robin.gentoo.org (8.13.5/8.13.5) with SMTP id j9DHbRsQ010011; Thu, 13 Oct 2005 17:37:27 GMT Received: from smtp.gentoo.org (smtp.gentoo.org [134.68.220.30]) by robin.gentoo.org (8.13.5/8.13.5) with ESMTP id j9DHbQkV023305 for ; Thu, 13 Oct 2005 17:37:26 GMT Received: from cpe-65-26-255-237.wi.res.rr.com ([65.26.255.237] helo=nightcrawler) by smtp.gentoo.org with esmtpa (Exim 4.43) id 1EQ7Aj-0004DF-P1 for gentoo-portage-dev@lists.gentoo.org; Thu, 13 Oct 2005 17:47:18 +0000 Date: Thu, 13 Oct 2005 12:47:11 -0500 From: Brian Harring To: gentoo-portage-dev@lists.gentoo.org Subject: Re: [gentoo-portage-dev] exclude cache from sync, was:Cache rewrite backport Message-ID: <20051013174711.GD21343@nightcrawler> References: <200510131912.43167.BastianBalthazarBux@pnpitalia.it> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="VuQYccsttdhdIfIP" Content-Disposition: inline In-Reply-To: <200510131912.43167.BastianBalthazarBux@pnpitalia.it> User-Agent: Mutt/1.5.8i X-Archives-Salt: 89727c37-5bc3-4bf2-8c3b-b66195dcdcae X-Archives-Hash: fcd276354f83eec6f6f3b89d51dc9fe6 --VuQYccsttdhdIfIP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Oct 13, 2005 at 07:12:42PM +0200, Francesco R. wrote: > Elaborating on the previous subject an idea come in mind, > avoid the $PORTDIR/metadata/cache sync, It's 80 MB, 20000 files >=20 > So I've tryed the first python patch of my life, obviously it's also the= =20 > first portage one (subliminal message, take the result with care and=20 > check it yourself) >=20 > how? the patch add the option "--withregen", the to=20 > exclude /metadata/cache from the sync and to issue a regen of the cache= =20 > before to update the metadata. > calling "emerge --withregen --sync" do the work. >=20 > The following test has run with a dual opteron as rsync server and a=20 > Athlon XP 2500+ with an old and slow disk, cabled with gigabit=20 > ethernet. The rsync is forced removing the timestamp. >=20 > The portage tree has been synced _before_ starting the bench, so the=20 > overhead of rsync is really reduced at the bare minimum, check the=20 > differences between the files on server and client. >=20 > The surprising result is that recreating the cache is faster than rsync= =20 > it. (real time) Nuke the cache and try it ;) Being slightly sarcastic there, fastest I've managed to get a=20 full regen down to was around 22 minutes for ebd, with stable being=20 around 34m http://dev.gentoo.org/~ferringb/ebuild-daemon/stats/paired-stats I'd expect the gap has narrowed a bit since those timing runs,=20 although it should still be sizable. Either way, the longer timing=20 run (stable ebuild.sh) is the one to note :) So... that --metadata run has an extra stat call per check best case,=20 but the cost of getting a cache entry is pretty heavily different. For metadata, just is a file read/write, for regen, exec(bash -c=20 ebuild.sh) which, via quicky commandline test (that sets a floor for=20 it), time bash /usr/lib/portage/bin/ebuild.sh , you're looking at=20 around .1s per call. Pretty heavy difference on updates, eg, worst case- something a user=20 hits on first sync, or nuking of the tree :) > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 1st try, portage [2.0.53_rc5] + cac|1st try, portage [2.0.53_rc5] + cac > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > sync with cache and _no_ regen |sync without cache and regen after > |Regenerating cache entries... > |done regen! > | > real 6m23.727s |real 4m14.837s > user 0m12.373s |user 0m18.681s > sys 0m13.849s |sys 0m7.744s > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 2nd try, portage [2.0.53_rc5] + cac|2nd try, portage [2.0.53_rc5] + cac > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > sync with cache and _no_ regen |sync without cache and regen after > |Regenerating cache entries... > |done regen! > | > real 6m53.649s |real 2m40.794s > user 0m12.361s |user 0m18.361s > sys 0m14.117s |sys 0m6.800s > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > 3rd try, portage [2.0.53_rc5] + cac|3rd try, portage [2.0.53_rc5] + cac > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > sync with cache and _no_ regen |sync without cache and regen after > |Regenerating cache entries... > |done regen! > | > real 6m46.973s |real 4m19.261s > user 0m12.605s |user 0m18.593s > sys 0m13.733s |sys 0m7.648s > | That said, I'm curious wth is triggering the 2x sys activity, which=20 probably is to blame for the ~90s difference Anyone game for profiling a --metadata run, vs --regen run via=20 profile.run? ~harring --VuQYccsttdhdIfIP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDTp2fvdBxRoA3VU0RArPvAJ9hWeQTuyr5K7K9tc4qkAWdWL1zNACeKCFy 7R4+x2DJarAK6bBmGs0rCBc= =eOBg -----END PGP SIGNATURE----- --VuQYccsttdhdIfIP-- -- gentoo-portage-dev@gentoo.org mailing list