From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 7840E138247 for ; Fri, 10 Jan 2014 01:52:28 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id A8F06E0D4E; Fri, 10 Jan 2014 01:52:22 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 9EEF2E0D29 for ; Fri, 10 Jan 2014 01:52:21 +0000 (UTC) Received: from localhost (unknown [100.42.98.7]) (using SSLv3 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: chutzpah) by smtp.gentoo.org (Postfix) with ESMTPSA id 7972933F587; Fri, 10 Jan 2014 01:52:20 +0000 (UTC) Date: Thu, 9 Jan 2014 17:52:16 -0800 From: Patrick McLean To: gentoo-dev@lists.gentoo.org Cc: TomWij@gentoo.org, patrick@gentoo.org Subject: Re: [gentoo-dev] Portage QOS Message-ID: <20140109175216.5c558257@gentoo.org> In-Reply-To: <20140110021903.22d9a77b@TOMWIJ-GENTOO> References: <52ce4eab.463f700a.4b43.16bd@mx.google.com> <52ce9994.24f5980a.0660.342e@mx.google.com> <6345949.JsNcU8lWSX@cschwan-laptop> <52cebfa2.aa78980a.7a02.42e5@mx.google.com> <86r48g8zdc.fsf@moguhome00.in.awa.tohoku.ac.jp> <52CF3F59.8010008@gentoo.org> <20140110021903.22d9a77b@TOMWIJ-GENTOO> X-Mailer: Claws Mail 3.9.3 (GTK+ 2.24.22; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Archives-Salt: 5f6df894-b67e-4204-9cb5-d13cf1258232 X-Archives-Hash: 03772e39cda93d2a2f0145df86af0d2a On Fri, 10 Jan 2014 02:19:03 +0100 Tom Wijsman wrote: > On Fri, 10 Jan 2014 08:31:21 +0800 > Patrick Lauer wrote: > > > On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote: > > > Igor writes: > > > > > >> The ebuilds have approximately the same time to install, the > > >> failure rate is about the same, emerge is getting slower. > > > > > > I am curious about the slowness of emerge. > > > > > > How about profile the portage and rewrite the time-crucial part in > > > C/C++, or ideally, borrowing the counterpart from paludis? How > > > feasible is that? > > > > Last I checked paludis wasn't faster - on average portage was a few > > percents faster. > > > For python things you really want python or C instead of C++... > > Why is this a Python thing? What's the reason to exclude a language? > > > So, what you wanted to ask was: > > "Which parts of pkgcore can be migrated into portage?" > > Or rather: "What does it take to migrate parts of pkgcore into > portage?" Why not just switch to using pkgcore as the default package manager. radhermit has been doing a lot of work lately getting EAPI 5 support added, and generally fixing bugs etc. Since we no longer have anyone intimately familiar with the portage code, we should just switch to a more modern and readable system. Rather than having random people trying to learn the convoluted portage code, let's concentrate on getting pkgcore to a point where we can replace portage with it. > > > I guess the dep-tree calculation is the slowest part. > > Yes, it's doing lots of silly dynamic things (backtracking), > > Exactly, without backtracking (which I can live without) it takes > around half a minute as compared to a lot of minutes; when it started > to take almost half an hour it was time to completely nuke > backtracking. > > Now I happily live without ever needing to enable backtracking > again. :) > > > and portage codebase on average is not designed for speed. > > Yes, inspecting the profiler results has me worried; though I find > half a minute acceptable for the amount of packages that are involved > on my system, so, I'm less concerned but it is still interesting that > there is the possibility to do things faster. It's just some work to > actually get it to do that, which requires much more dedication, time > and knowledge than doing an occasional commit. > > > As a first step I would recommend profiling it and removing unneeded > > stuff (do less work!), rewriting parts in C is a lot of work and not > > needed for the first round of speedups. > > See my other mail <20140110020218.0f6244d5@TOMWIJ-GENTOO> for recent > profiling images; we should indeed start with dealing with the low > hanging fruit (there are quite some 0-2% speed improvements possible), > as that can get us to speed it up faster than trying to deal with some > algorithmic complexity that needs far more insight to redesign, let > alone to properly re-implement it. >