Re: [gentoo-portage-dev] Performance tuning and parallelisation

[Ed W <lists@wildgooses.com>]

[-- Attachment #1: Type: text/plain, Size: 1975 bytes --]

On 26/08/2021 17:38, Marco Sirabella wrote:
> -
>
> Hi Ed,
>
> I’ve taken a dabble at trying to track down portage’s bottlenecks (and have stopped for the time
> being at solving them :/ )
>
>     Can anyone give me a leg up on how I could benchmark this further and look for the hotspot?
>     Perhaps someone understand the architecture of this point more intimately and could point at
>     whether there are opportunities to do some of the processing on mass, rather than per file?
>
> From my notes at the timem, it looks like yappi <https://pypi.org/project/yappi/> worked a bit
> better than python’s built in cProfile for me because it properly dove into async calls. I used
> snakeviz <https://jiffyclub.github.io/snakeviz/> for visualizing the profile results.
>
> I was taking a look at depclean, but I found similarly that a lot of duplicate process was being
> done due to encapsulated abstractions not being able to communicate that the same thing was being
> done multiple times eg removing each package processes a massive json structure for each package
> removed, although I opted to work on the more-understandable unicode conversions.
>
> My stalled progress can be found here: #700 <https://github.com/gentoo/portage/pull/700>. Lost the
> drive to continue for now unfortunately :<
>
> Good luck! Looking forward to your optimizations
>
> – Marco Sirabella
>

Hi All, thanks for the replies. Wow, Marco, that patch touches a lot of stuff...!

OK, I will start by trying to get the profilers going and work from there...

(Alec, to avoid replying separately: Thanks for your notes. Yes, I am not clear which of the
install/merge phases specifically is the culprit, but it feels like something in that area is
"slow", especially when run under qemu user mode. I think using qmerge won't work for my build
script, but great idea to use it for benchmarking to narrow things down - thanks)

Thanks all

Ed W


[-- Attachment #2: Type: text/html, Size: 3490 bytes --]