Re: [gentoo-portage-dev] Add caching to a few commonly used functions

[Chun-Yu Shei <cshei@google.com>]

Hi Fabian,

Just eyeballing htop's RES column while emerge is running, the max
value I see during "emerge -uDvpU --with-bdeps=y @world" increases
from 272 MB to 380 MB, so it looks to be around 110 MB of extra memory
usage on my system (with 1,094 total packages installed).

Chun-Yu

On Sat, Jun 27, 2020, 12:35 AM Fabian Groffen <grobian@gentoo.org> wrote:
>
> Hi Chun-Yu,
>
> On 26-06-2020 23:34:12 -0700, Chun-Yu Shei wrote:
> > Hi,
> >
> > I was recently interested in whether portage could be speed up, since
> > dependency resolution can sometimes take a while on slower machines.
> > After generating some flame graphs with cProfile and vmprof, I found 3
> > functions which seem to be called extremely frequently with the same
> > arguments: catpkgsplit, use_reduce, and match_from_list.  In the first
> > two cases, it was simple to cache the results in dicts, while
> > match_from_list was a bit trickier, since it seems to be a requirement
> > that it return actual entries from the input "candidate_list".  I also
> > ran into some test failures if I did the caching after the
> > mydep.unevaluated_atom.use and mydep.repo checks towards the end of the
> > function, so the caching is only done up to just before that point.
> >
> > The catpkgsplit change seems to definitely be safe, and I'm pretty sure
> > the use_reduce one is too, since anything that could possibly change the
> > result is hashed.  I'm a bit less certain about the match_from_list one,
> > although all tests are passing.
> >
> > With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world"
> > speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup.  "emerge
> > -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec
> > (2.5% improvement).  Since the upgrade case is far more common, this
> > would really help in daily use, and it shaves about 30 seconds off
> > the time you have to wait to get to the [Yes/No] prompt (from ~90s to
> > 60s) on my old Sandy Bridge laptop when performing normal upgrades.
> >
> > Hopefully, at least some of these patches can be incorporated, and please
> > let me know if any changes are necessary.
>
> This sounds like a good job to me!  Do you have any idea what the added
> memory pressure for these changes are?
>
> Thanks,
> Fabian
> >
> > Thanks,
> > Chun-Yu
> >
> >
> >
>
> --
> Fabian Groffen
> Gentoo on a different level