On Monday, 27 November 2023 15:39:33 GMT Peter Humphreey wrote:
> Hello list,
> 
> I still can't see how portage limits the load. Today I'm emerging
> libreoffice, and it's spending almost the whole time working with 4 CPU
> threads. But:
> 
> $ grep -e '\-j' -e distcc /etc/portage/make.conf
> EMERGE_DEFAULT_OPTS="--jobs=18 --load-average=30 --backtrack=200 --
> autounmask=n --keep-going  --nospinner"
> FEATURES="distcc userfetch buildpkg network-sandbox parallel-install sandbox
> userpriv usersandbox"
> MAKEOPTS="-j18"
> 
> I found a suggestion to use distcc in the installation handbook, which I
> hadn't seen there before, so I went searching for it and found how to do it.
> It usually works well, in this case starting 18 packages before starting LO
> itself. grep -rw doesn't find '4' anywere relevant under /etc/portage/ .
> Other times it just doesn't help at all.
> 
> What am I missing?

In absence of other contributions I'll offer a theoretical explanation, based 
on random observations on my systems.

You have specified as many as 18 packages to be emerged in parallel x up to 18 
make jobs each.  The result of [18 x 18 = 324] is to be limited by a total 
load average of 30.

If there were more than 18 packages listed to be emerged and there were no 
dependencies between them to restrict how many could start emerging in 
parallel, you would observe =<18 packages being emerged in parallel.  This 
alone will not breach the load limit of 30.

Let's assume all 18 packages had a large codebase to need at least 18 make 
jobs each.  Sooner or later you'd have 18 parallel emerges all trying to run 
18 make jobs.

Were this to occur the load limit restriction would kick in and you would see 
only up to 30 jobs listed in top, with individual package processes 
alternating in the top list of make threads.

Here's my hypothesis explaining your own observation with libreoffice.  As a 
package or more finished emerging, libreoffice's turn comes up.  Soon 
libreoffice starts to execute make jobs, but any of the following may apply:

1. There are only 4 out of 30 jobs available, because other packages are 
already using 26, throughout your window of observation.
2. Libreoffice sequencing of make jobs is mostly linear with succeeding make 
jobs waiting on output from their predecessors.
3. Libreoffice source code is not optimised for high parallelism - I recall 
when it was hardcoded at -j1 just a few years ago.  Before this restriction 
was added, any bug reporters were advised to try again after limiting make to 
-j1.

Next time I'm building libreoffice on a beefier system I'll keep an eye out 
for the number of jobs to see what it gets up to.