From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 4EE38138247 for ; Fri, 10 Jan 2014 01:19:59 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 4D98DE0C9E; Fri, 10 Jan 2014 01:19:54 +0000 (UTC) Received: from gerard.telenet-ops.be (gerard.telenet-ops.be [195.130.132.48]) by pigeon.gentoo.org (Postfix) with ESMTP id D6894E0C71 for ; Fri, 10 Jan 2014 01:19:52 +0000 (UTC) Received: from TOMWIJ-GENTOO ([94.226.55.127]) by gerard.telenet-ops.be with bizsmtp id C1Kr1n00U2khLEN0H1Ks1h; Fri, 10 Jan 2014 02:19:52 +0100 Date: Fri, 10 Jan 2014 02:19:03 +0100 From: Tom Wijsman To: patrick@gentoo.org Cc: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] Portage QOS Message-ID: <20140110021903.22d9a77b@TOMWIJ-GENTOO> In-Reply-To: <52CF3F59.8010008@gentoo.org> References: <52ce4eab.463f700a.4b43.16bd@mx.google.com> <52ce9994.24f5980a.0660.342e@mx.google.com> <6345949.JsNcU8lWSX@cschwan-laptop> <52cebfa2.aa78980a.7a02.42e5@mx.google.com> <86r48g8zdc.fsf@moguhome00.in.awa.tohoku.ac.jp> <52CF3F59.8010008@gentoo.org> X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.22; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/kY_66AYo8B6msziA+vLMFIG"; protocol="application/pgp-signature" X-Archives-Salt: 4c9e21ec-f060-4a0c-ad27-bd3c481c6c07 X-Archives-Hash: 6fb63b700ef1184273758af7c7b9727d --Sig_/kY_66AYo8B6msziA+vLMFIG Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Fri, 10 Jan 2014 08:31:21 +0800 Patrick Lauer wrote: > On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote: > > Igor writes: > >=20 > >> The ebuilds have approximately the same time to install, the > >> failure rate is about the same, emerge is getting slower. > >=20 > > I am curious about the slowness of emerge. > >=20 > > How about profile the portage and rewrite the time-crucial part in > > C/C++, or ideally, borrowing the counterpart from paludis? How > > feasible is that? >=20 > Last I checked paludis wasn't faster - on average portage was a few > percents faster. Besides that, a different Python version can also differ; I've found Python 3.3 to be faster for both emerge and repoman. Maybe PyPy can end up even being faster than that, but that's another subjective story; though introducing multiple threads in Portage would be really nice, but the GIL sits in the way: https://wiki.python.org/moin/GlobalInterpreterLock http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-why > For python things you really want python or C instead of C++... Why is this a Python thing? What's the reason to exclude a language? > So, what you wanted to ask was: > "Which parts of pkgcore can be migrated into portage?" Or rather: "What does it take to migrate parts of pkgcore into portage?" > > I guess the dep-tree calculation is the slowest part. > Yes, it's doing lots of silly dynamic things (backtracking), Exactly, without backtracking (which I can live without) it takes around half a minute as compared to a lot of minutes; when it started to take almost half an hour it was time to completely nuke backtracking. Now I happily live without ever needing to enable backtracking again. :) > and portage codebase on average is not designed for speed. Yes, inspecting the profiler results has me worried; though I find half a minute acceptable for the amount of packages that are involved on my system, so, I'm less concerned but it is still interesting that there is the possibility to do things faster. It's just some work to actually get it to do that, which requires much more dedication, time and knowledge than doing an occasional commit. > As a first step I would recommend profiling it and removing unneeded > stuff (do less work!), rewriting parts in C is a lot of work and not > needed for the first round of speedups. See my other mail <20140110020218.0f6244d5@TOMWIJ-GENTOO> for recent profiling images; we should indeed start with dealing with the low hanging fruit (there are quite some 0-2% speed improvements possible), as that can get us to speed it up faster than trying to deal with some algorithmic complexity that needs far more insight to redesign, let alone to properly re-implement it. --=20 With kind regards, Tom Wijsman (TomWij) Gentoo Developer E-mail address : TomWij@gentoo.org GPG Public Key : 6D34E57D GPG Fingerprint : C165 AF18 AB4C 400B C3D2 ABF0 95B2 1FCD 6D34 E57D --Sig_/kY_66AYo8B6msziA+vLMFIG Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJSz0qHAAoJEJWyH81tNOV9cz0H/3rOLrjyru5Kkz/EyHoy408b 5g7xUl0El6ZIJURkNtri7Ng91B75X/riwAim5W+nfTT8uLQY3Mmj2jT4eyx7yiij sOAUKTPpZ4fArElwVPkK5NhvLcfMdoj/nbMq4CxVBlzVRGLUGB+cy0QWIxw0h5l7 JHXugpBMXAEHJ8tcsJoOx8zOpdo+O4TweYNksj+3gXv7jiF9Tm/+QiMsE/pK6ORW 8DDIB2+PUeH0oXzO3aZk/EkSJ4K6Kz67NjIqVWQCWbQBPU6phSls0ISNbKluMv0P gShvHdGZAWoM1XIlzkow+OtqM8kylNSgUJYs9ddX9dI2JDc8XwQ73vpVuPGA6lM= =EmC/ -----END PGP SIGNATURE----- --Sig_/kY_66AYo8B6msziA+vLMFIG--