Re: [gentoo-dev] Portage QOS

[Tom Wijsman <TomWij@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 3042 bytes --]

On Fri, 10 Jan 2014 08:31:21 +0800
Patrick Lauer <patrick@gentoo.org> wrote:

> On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote:
> > Igor <lanthruster@gmail.com> writes:
> > 
> >> The ebuilds have approximately the same time to install, the
> >> failure rate is about the same, emerge is getting slower.
> > 
> > I am curious about the slowness of emerge.
> > 
> > How about profile the portage and rewrite the time-crucial part in
> > C/C++, or ideally, borrowing the counterpart from paludis? How
> > feasible is that?
> 
> Last I checked paludis wasn't faster - on average portage was a few
> percents faster.

Besides that, a different Python version can also differ; I've found
Python 3.3 to be faster for both emerge and repoman. Maybe PyPy can end
up even being faster than that, but that's another subjective story;
though introducing multiple threads in Portage would be really nice,
but the GIL sits in the way:

https://wiki.python.org/moin/GlobalInterpreterLock
http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-why

> For python things you really want  python or C instead of C++...

Why is this a Python thing? What's the reason to exclude a language?

> So, what you wanted to ask was:
> "Which parts of pkgcore can be migrated into portage?"

Or rather: "What does it take to migrate parts of pkgcore into portage?"

> > I guess the dep-tree calculation is the slowest part.
> Yes, it's doing lots of silly dynamic things (backtracking),

Exactly, without backtracking (which I can live without) it takes
around half a minute as compared to a lot of minutes; when it started
to take almost half an hour it was time to completely nuke backtracking.

Now I happily live without ever needing to enable backtracking again. :)

> and portage codebase on average is not designed for speed.

Yes, inspecting the profiler results has me worried; though I find half
a minute acceptable for the amount of packages that are involved on my
system, so, I'm less concerned but it is still interesting that there
is the possibility to do things faster. It's just some work to actually
get it to do that, which requires much more dedication, time and
knowledge than doing an occasional commit.

> As a first step I would recommend profiling it and removing unneeded
> stuff (do less work!), rewriting parts in C is a lot of work and not
> needed for the first round of speedups.

See my other mail <20140110020218.0f6244d5@TOMWIJ-GENTOO> for recent
profiling images; we should indeed start with dealing with the low
hanging fruit (there are quite some 0-2% speed improvements possible),
as that can get us to speed it up faster than trying to deal with some
algorithmic complexity that needs far more insight to redesign, let
alone to properly re-implement it.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]