Hi Ed,

I’ve taken a dabble at trying to track down portage’s bottlenecks (and have stopped for the time being at solving them :/ )

Can anyone give me a leg up on how I could benchmark this further and look for the hotspot? Perhaps someone understand the architecture of this point more intimately and could point at whether there are opportunities to do some of the processing on mass, rather than per file?

From my notes at the timem, it looks like yappi worked a bit better than python’s built in cProfile for me because it properly dove into async calls. I used snakeviz for visualizing the profile results.

I was taking a look at depclean, but I found similarly that a lot of duplicate process was being done due to encapsulated abstractions not being able to communicate that the same thing was being done multiple times eg removing each package processes a massive json structure for each package removed, although I opted to work on the more-understandable unicode conversions.

My stalled progress can be found here: #700. Lost the drive to continue for now unfortunately :<

Good luck! Looking forward to your optimizations

– Marco Sirabella

Hi All, thanks for the replies. Wow, Marco, that patch touches a lot of stuff...!

OK, I will start by trying to get the profilers going and work from there...

(Alec, to avoid replying separately: Thanks for your notes. Yes, I am not clear which of the install/merge phases specifically is the culprit, but it feels like something in that area is "slow", especially when run under qemu user mode. I think using qmerge won't work for my build script, but great idea to use it for benchmarking to narrow things down - thanks)

Thanks all

Ed W