Re: [gentoo-user] long compiles

[Peter Humphrey <peter@prh.myzen.co.uk>]

On Wednesday, 13 September 2023 16:14:09 BST Michael wrote:

> I recall this being discussed in a previous thread, but if your CPU has 24
> threads and you've set:
> 
> EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32
> MAKEOPTS="-j14"
> 
> You will be asking emerge to run up to 4 x 14 = 56 threads, which could
> potentially eat up to 2G of RAM each, i.e. 112G of RAM.  This will exhaust
> your 64G of RAM, not taking into account whatever else the OS will be trying
> to run at the time.

Yes, I understand that, but I've spent a lot of time watching, and 
instrumenting, and in practice the load doesn't exceed about 33.

> The --load-average=32 is normally expected to be a floating point number

That stipulation has only appeared recently. I have tried adding a '.0' to the 
number, and it's made no noticeaible difference. Besides, I could attempt 
pedantry and declare that the set of all integers is a proper subset of all 
real numbers.  ;-)

> Of course, not all emerges use make and you may never or rarely emerge 4 x
> monster packages in parallel to need 2G of RAM for each thread at the same
> time.

I have actually had three or four big packages running together, but my 
observation is that they don't pump the load up too far.

> If only we had at our disposal some AI algorithm to calculate dynamically
> each time we run emerge the optimal combo of parallel emerge jobs and
> number of make tasks, so as to achieve the highest total time saving Vs
> energy spent! Or just the highest total time saving.  ;-)

Yes, that's what we need, all right.

> I haven't performed any meaningful comparisons to determine where the
> greatest gains are to be achieved.  Parallel emerges of many small
> packages, or a large number of make jobs for big packages.  The combination
> would change each time according to the individual packages waiting for an
> update.  In my use case, instinctively feels more beneficial reducing the
> time I have to wait for huge packages like qtwebengine to finish, than
> accelerating the updates of half a dozen smaller packages.

That is the difficulty. I do often rebuild a new system, not trusting the 
existing one any longer, and a lot of time is spent fiddling with four tiny 
packages at a time in the early and middle stages, then benefitting from the 
limit of 4 once the desktop packages begin.

> Therefore, as a rule I leave EMERGE_DEFAULT_OPTS unset.  I set MAKEOPTS jobs
> to the number of CPU threads +1 and the load average at 95%, so I can
> continue using the PC without any noticeable latency. On PCs where RAM is
> constrained I reduce the MAKEOPTS in /etc/portage/ package.env for any large
> packages which are bound to start swapping and thrashing the disk.

Interesting. Do you mean 95% of the jobs figure? I'll do some more 
experimenting.

Meanwhile, perhaps I'll run new builds in two stages: the first without any 
desktop packages - I do have sets defined to enable that - and the second with 
them. I can't do that with emerge -e though.

-- 
Regards,
Peter.