[gentoo-user] Portage improved

[gentoo-user] Portage improved

[Peter Humphrey]

Greetings,

I see that the latest version of portage has improved the handling of giant 
packages. Today I had a few dozen kde-frameworks packages to install, together 
with webkit-gtk. That job was near the top of the list, so it was started 
before most of the kde ones. I have this in make.conf:

$ grep '\-j' /etc/portage/make.conf
EMERGE_DEFAULT_OPTS="--jobs=24 --load-average=30 [...]"
MAKEOPTS="-j16 -l16"

(64GB, 24 CPU threads)

The CPU load then rose quickly past 40 as the kde packages got stuck in, up to 
24 total jobs. I was surprised and pleased to see that packages were made to 
pause until cores became available. This process was progressive until the 
load came down again.

That's a big improvement, and I'd like to commend the team on it.

Just one little fly in the ointment: the status string became too long to show 
properly. Perhaps shorter phrases could be used, or numbers limited to two 
digits, or the 80-character line limit exceeded while running in an x-term.

Anyway, well done the team!

-- 
Regards,
Peter.

Re: [gentoo-user] Portage improved

[Peter Humphrey]

On Saturday 14 September 2024 13:05:26 BST Peter Humphrey wrote:

> Just one little fly in the ointment: the status string became too long to
> show properly. Perhaps shorter phrases could be used, or numbers limited to
> two digits, or the 80-character line limit exceeded while running in an
> x-term.

And indeed the line limit has been relaxed: it uses the full 90 characters of 
my Konsoles.

> Anyway, well done the team!

-- 
Regards,
Peter.

Re: [gentoo-user] Portage improved

[Peter Humphrey]

On Friday 11 October 2024 11:49:18 GMT I wrote:
> On Saturday 14 September 2024 13:05:26 BST Peter Humphrey wrote:
> > Just one little fly in the ointment: the status string became too long to
> > show properly. Perhaps shorter phrases could be used, or numbers limited
> > to two digits, or the 80-character line limit exceeded while running in an
> > x-term.
> 
> And indeed the line limit has been relaxed: it uses the full 90 characters
> of my Konsoles.

More to do yet though. The merge-wait idea works well with humongous packages 
like nodejs and the web engines, but it's too cautious with ordinary ones. 
I've been building (too many) new systems over the last fortnight, and when 
there are, say, a hundred packages to compile, the load steadies out at about 
4, with several more in merge-wait. This is with i24 l30 in make.conf. 

Just a bit of tuning, then...

-- 
Regards,
Peter.

Re: [gentoo-user] Portage improved

[Wols Lists]

On 31/10/2024 11:56, Peter Humphrey wrote:
> the load steadies out at about
> 4, with several more in merge-wait. This is with i24 l30 in make.conf.

How many cores does your CPU have. I've found that load is an 
approximation to "how many cores are running at 100%".

It's very noticeable running xosview, that as soon as load goes above 4 
on my 4-core system, response goes through the floor. Which is probably 
because a large chunk of CPU time suddenly gets diverted to scheduling 
and task switching, rather than doing useful work.

Cheers,
Wol

Re: [gentoo-user] Portage improved

[Peter Humphrey]

On Sunday 3 November 2024 12:56:05 GMT Wols Lists wrote:
> On 31/10/2024 11:56, Peter Humphrey wrote:
> > the load steadies out at about
> > 4, with several more in merge-wait. This is with i24 l30 in make.conf.
> 
> How many cores does your CPU have. I've found that load is an
> approximation to "how many cores are running at 100%".

24, which is why I have -i24 -l30 in make.conf. The machine also has 6GB RAM.

-- 
Regards,
Peter.

Re: [gentoo-user] Portage improved

[Peter Humphrey]

On Sunday 3 November 2024 23:18:37 GMT Peter Humphrey wrote:
> On Sunday 3 November 2024 12:56:05 GMT Wols Lists wrote:
> > On 31/10/2024 11:56, Peter Humphrey wrote:
> > > the load steadies out at about
> > > 4, with several more in merge-wait. This is with i24 l30 in make.conf.
> > 
> > How many cores does your CPU have. I've found that load is an
> > approximation to "how many cores are running at 100%".
> 
> 24, which is why I have -i24 -l30 in make.conf. The machine also has 6GB
> RAM.

Er... 64GB.

-- 
Regards,
Peter.

Re: [gentoo-user] Portage improved

[Peter Humphrey]

On Thursday 31 October 2024 11:56:41 GMT Peter Humphrey wrote:
> On Friday 11 October 2024 11:49:18 GMT I wrote:
> > On Saturday 14 September 2024 13:05:26 BST Peter Humphrey wrote:
> > > Just one little fly in the ointment: the status string became too long
> > > to
> > > show properly. Perhaps shorter phrases could be used, or numbers limited
> > > to two digits, or the 80-character line limit exceeded while running in
> > > an
> > > x-term.
> > 
> > And indeed the line limit has been relaxed: it uses the full 90 characters
> > of my Konsoles.
> 
> More to do yet though. The merge-wait idea works well with humongous
> packages like nodejs and the web engines, but it's too cautious with
> ordinary ones. I've been building (too many) new systems over the last
> fortnight, and when there are, say, a hundred packages to compile, the load
> steadies out at about 4, with several more in merge-wait. This is with i24
> l30 in make.conf.

There's also a logic problem. When a package enters the Installing phase, it's 
shown instead as Merge Wait, or it's added to that queue.

-- 
Regards,
Peter.

Re: [gentoo-user] Portage improved

[Peter Humphrey]

On Tuesday 5 November 2024 06:58:37 GMT I wrote:
> On Thursday 31 October 2024 11:56:41 GMT Peter Humphrey wrote:
> > On Friday 11 October 2024 11:49:18 GMT I wrote:
> > > On Saturday 14 September 2024 13:05:26 BST Peter Humphrey wrote:
> > > > Just one little fly in the ointment: the status string became too long
> > > > to
> > > > show properly. Perhaps shorter phrases could be used, or numbers
> > > > limited
> > > > to two digits, or the 80-character line limit exceeded while running
> > > > in
> > > > an
> > > > x-term.
> > > 
> > > And indeed the line limit has been relaxed: it uses the full 90
> > > characters
> > > of my Konsoles.
> > 
> > More to do yet though. The merge-wait idea works well with humongous
> > packages like nodejs and the web engines, but it's too cautious with
> > ordinary ones. I've been building (too many) new systems over the last
> > fortnight, and when there are, say, a hundred packages to compile, the
> > load
> > steadies out at about 4, with several more in merge-wait. This is with i24
> > l30 in make.conf.
> 
> There's also a logic problem. When a package enters the Installing phase,
> it's shown instead as Merge Wait, or it's added to that queue.

Here's an example of it working with medium-sized programs:

...
>>> Emerging (79 of 102) dev-util/gdbus-codegen-2.78.6::gentoo
>>> Emerging (80 of 102) dev-util/glib-utils-2.78.6::gentoo
>>> Emerging (81 of 102) sys-apps/sysvinit-3.11::gentoo
>>> Emerging (82 of 102) net-misc/curl-8.10.1-r1::gentoo
>>> Installing (80 of 102) dev-util/glib-utils-2.78.6::gentoo
>>> Installing (79 of 102) dev-util/gdbus-codegen-2.78.6::gentoo
>>> Completed (80 of 102) dev-util/glib-utils-2.78.6::gentoo
>>> Completed (79 of 102) dev-util/gdbus-codegen-2.78.6::gentoo
>>> Installing (81 of 102) sys-apps/sysvinit-3.11::gentoo
>>> Completed (81 of 102) sys-apps/sysvinit-3.11::gentoo
>>> Installing (82 of 102) net-misc/curl-8.10.1-r1::gentoo
>>> Completed (82 of 102) net-misc/curl-8.10.1-r1::gentoo
>>> Emerging (83 of 102) dev-libs/elfutils-0.192-r1::gentoo
>>> Emerging (84 of 102) sys-devel/gcc-14.2.1_p20241026::gentoo
>>> Jobs: 82 of 102 complete, 1 running, 1 merge wait         Load avg: 17.1, 11.2, 8.5

Better than load average 4, but nowhere near full capacity (-i24 -l30).

-- 
Regards,
Peter.