[gentoo-user] long compiles

[gentoo-user] long compiles

[Alan McKinnon]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

After my long time away from Gentoo, I thought perhaps some packages that
always took ages to compile would have improved. I needed to change to
~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what I did
and let emerge do it's thing.

chromium has been building since 10:14, it's now 21:16 and still going so 9
hours at least on this machine to build a browser - almost as bad as
openoffice at it's worst (regularly took 12 hours). Nodejs also took a
while, but I didn't record time.


What other packages have huge build times?

-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

[-- Attachment #2: Type: text/html, Size: 875 bytes --]

Re: [gentoo-user] long compiles

[Dale]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

Alan McKinnon wrote:
> After my long time away from Gentoo, I thought perhaps some packages
> that always took ages to compile would have improved. I needed to
> change to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So
> that's what I did and let emerge do it's thing.
>
> chromium has been building since 10:14, it's now 21:16 and still going
> so 9 hours at least on this machine to build a browser - almost as bad
> as openoffice at it's worst (regularly took 12 hours). Nodejs also
> took a while, but I didn't record time.
>
>
> What other packages have huge build times?
>
> -- 
> Alan McKinnon
> alan dot mckinnon at gmail dot com


I have some software you don't likely use that takes a while but one
that is common is qtwebengine or something.  If it's not that one, it's
qtweb something.  It takes about 4 hours, sometimes 5 or so. 

I think the software takes longer to compile so that we will build new
rigs.  ROFL 

Dale

:-)  :-) 

[-- Attachment #2: Type: text/html, Size: 2020 bytes --]

Re: [gentoo-user] long compiles

[Siddhanth Rathod]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

Chromium and qtwebengine have the longest build times that I have
encountered

On Tue, 12 Sep 2023 at 1:01 AM, Dale <rdalek1967@gmail.com> wrote:

> Alan McKinnon wrote:
>
> After my long time away from Gentoo, I thought perhaps some packages that
> always took ages to compile would have improved. I needed to change to
> ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what I did
> and let emerge do it's thing.
>
> chromium has been building since 10:14, it's now 21:16 and still going so
> 9 hours at least on this machine to build a browser - almost as bad as
> openoffice at it's worst (regularly took 12 hours). Nodejs also took a
> while, but I didn't record time.
>
>
> What other packages have huge build times?
>
> --
> Alan McKinnon
> alan dot mckinnon at gmail dot com
>
>
>
> I have some software you don't likely use that takes a while but one that
> is common is qtwebengine or something.  If it's not that one, it's qtweb
> something.  It takes about 4 hours, sometimes 5 or so.
>
> I think the software takes longer to compile so that we will build new
> rigs.  ROFL
>
> Dale
>
> :-)  :-)
>

[-- Attachment #2: Type: text/html, Size: 2378 bytes --]

Re: [gentoo-user] long compiles

[Ramon Fischer]

[-- Attachment #1.1.1: Type: text/plain, Size: 2418 bytes --]

Hi Alan,

just quick and dirty, I am too tired for formalities. :) The following 
list contains packages, that may be too big for tmpfs and are most 
probably very time consuming to compile:

     $ < /etc/portage/package.env/no_tmpfs.conf
     # custom - 20181121 - rfischer: list packages, which are too big 
for tmpfs
     #app-editors/neovim no_tmpfs.conf
     #app-emulation/qemu-kv no_tmpfs.conf
     #app-office/libreoffice no_tmpfs.conf
     #dev-db/mysql no_tmpfs.conf
     #dev-java/icedtea no_tmpfs.conf
     #dev-lang/ghc no_tmpfs.conf
     #dev-lang/ghc no_tmpfs.conf
     #dev-lang/mono no_tmpfs.conf
     #dev-lang/rust no_tmpfs.conf
     #dev-lang/spidermonkey no_tmpfs.conf
     #dev-libs/libpcre no_tmpfs.conf
     #dev-qt/qtwebengine no_tmpfs.conf #throttle_make_emerge.conf
     #mail-client/thunderbird no_tmpfs.conf
     #media-libs/opencv no_tmpfs.conf
     #media-libs/opencv no_tmpfs.conf
     #net-libs/nodejs no_tmpfs.conf
     #net-misc/openssh no_tmpfs.conf
     #sci-libs/tensorflow no_tmpfs.conf
     #sys-apps/iproute2 no_tmpfs.conf
     #sys-devel/clang no_tmpfs.conf
     #sys-devel/gcc no_tmpfs.conf
     #www-client/chromium no_tmpfs.conf #throttle_make_emerge.conf
     #www-client/firefox no_tmpfs.conf
     #www-client/ungoogled-chromium no_tmpfs.conf #throttle_make_emerge.conf

See also:

* 
https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs#Considering_tmpfs.27_size
* 
https://wiki.gentoo.org/wiki/Q_applets#Extracting_information_from_emerge_logs_.28qlop.29

Sleeps away.
-Ramon

On 11/09/2023 21:19, Alan McKinnon wrote:
> After my long time away from Gentoo, I thought perhaps some packages 
> that always took ages to compile would have improved. I needed to 
> change to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So 
> that's what I did and let emerge do it's thing.
>
> chromium has been building since 10:14, it's now 21:16 and still going 
> so 9 hours at least on this machine to build a browser - almost as bad 
> as openoffice at it's worst (regularly took 12 hours). Nodejs also 
> took a while, but I didn't record time.
>
>
> What other packages have huge build times?
>
> -- 
> Alan McKinnon
> alan dot mckinnon at gmail dot com

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 8969 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-user] long compiles

[Ramon Fischer]

[-- Attachment #1.1.1: Type: text/plain, Size: 3440 bytes --]

In addition to the reference to "qlop":

     $ qlop ungoogled-chromium | tail
     2022-08-04T19:58:22 >>> www-client/ungoogled-chromium: 9:06:54
     2022-08-05T14:27:44 >>> www-client/ungoogled-chromium: 16:19:06
     2022-08-25T11:45:37 >>> www-client/ungoogled-chromium: 8:01:54
     2022-09-01T10:03:19 >>> www-client/ungoogled-chromium: 15:27:22
     2022-09-06T16:29:49 >>> www-client/ungoogled-chromium: 9:46:16
     2022-09-14T17:48:16 >>> www-client/ungoogled-chromium: 9:30:29
     2022-10-08T03:40:44 >>> www-client/ungoogled-chromium: 1:52:16
     2022-10-21T17:58:43 >>> www-client/ungoogled-chromium: 9:24:55
     2022-12-16T17:47:27 >>> www-client/ungoogled-chromium: 20:56:08
     2023-03-20T14:12:02 <<< www-client/ungoogled-chromium: 2s

Since I am using "ccache", the compilation time is sometimes doubled.

ZzZzZzzz
-Ramon

On 11/09/2023 21:42, Ramon Fischer wrote:
> Hi Alan,
>
> just quick and dirty, I am too tired for formalities. :) The following 
> list contains packages, that may be too big for tmpfs and are most 
> probably very time consuming to compile:
>
>     $ < /etc/portage/package.env/no_tmpfs.conf
>     # custom - 20181121 - rfischer: list packages, which are too big 
> for tmpfs
>     #app-editors/neovim no_tmpfs.conf
>     #app-emulation/qemu-kv no_tmpfs.conf
>     #app-office/libreoffice no_tmpfs.conf
>     #dev-db/mysql no_tmpfs.conf
>     #dev-java/icedtea no_tmpfs.conf
>     #dev-lang/ghc no_tmpfs.conf
>     #dev-lang/ghc no_tmpfs.conf
>     #dev-lang/mono no_tmpfs.conf
>     #dev-lang/rust no_tmpfs.conf
>     #dev-lang/spidermonkey no_tmpfs.conf
>     #dev-libs/libpcre no_tmpfs.conf
>     #dev-qt/qtwebengine no_tmpfs.conf #throttle_make_emerge.conf
>     #mail-client/thunderbird no_tmpfs.conf
>     #media-libs/opencv no_tmpfs.conf
>     #media-libs/opencv no_tmpfs.conf
>     #net-libs/nodejs no_tmpfs.conf
>     #net-misc/openssh no_tmpfs.conf
>     #sci-libs/tensorflow no_tmpfs.conf
>     #sys-apps/iproute2 no_tmpfs.conf
>     #sys-devel/clang no_tmpfs.conf
>     #sys-devel/gcc no_tmpfs.conf
>     #www-client/chromium no_tmpfs.conf #throttle_make_emerge.conf
>     #www-client/firefox no_tmpfs.conf
>     #www-client/ungoogled-chromium no_tmpfs.conf 
> #throttle_make_emerge.conf
>
> See also:
>
> * 
> https://wiki.gentoo.org/wiki/Portage_TMPDIR_on_tmpfs#Considering_tmpfs.27_size
> * 
> https://wiki.gentoo.org/wiki/Q_applets#Extracting_information_from_emerge_logs_.28qlop.29
>
> Sleeps away.
> -Ramon
>
> On 11/09/2023 21:19, Alan McKinnon wrote:
>> After my long time away from Gentoo, I thought perhaps some packages 
>> that always took ages to compile would have improved. I needed to 
>> change to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So 
>> that's what I did and let emerge do it's thing.
>>
>> chromium has been building since 10:14, it's now 21:16 and still 
>> going so 9 hours at least on this machine to build a browser - almost 
>> as bad as openoffice at it's worst (regularly took 12 hours). Nodejs 
>> also took a while, but I didn't record time.
>>
>>
>> What other packages have huge build times?
>>
>> -- 
>> Alan McKinnon
>> alan dot mckinnon at gmail dot com
>

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 8969 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-user] long compiles

[Alan McKinnon]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

qtwebengine! yes that one took forever also. It also said my 16G of RAM was
smaller than the 16G it needed. Weird.

Anyways I enabled a swapfile and left it to run overnight

Alan

On Mon, Sep 11, 2023 at 9:31 PM Dale <rdalek1967@gmail.com> wrote:

> Alan McKinnon wrote:
>
> After my long time away from Gentoo, I thought perhaps some packages that
> always took ages to compile would have improved. I needed to change to
> ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what I did
> and let emerge do it's thing.
>
> chromium has been building since 10:14, it's now 21:16 and still going so
> 9 hours at least on this machine to build a browser - almost as bad as
> openoffice at it's worst (regularly took 12 hours). Nodejs also took a
> while, but I didn't record time.
>
>
> What other packages have huge build times?
>
> --
> Alan McKinnon
> alan dot mckinnon at gmail dot com
>
>
>
> I have some software you don't likely use that takes a while but one that
> is common is qtwebengine or something.  If it's not that one, it's qtweb
> something.  It takes about 4 hours, sometimes 5 or so.
>
> I think the software takes longer to compile so that we will build new
> rigs.  ROFL
>
> Dale
>
> :-)  :-)
>


-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

[-- Attachment #2: Type: text/html, Size: 2474 bytes --]

Re: [gentoo-user] long compiles

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote:

> chromium has been building since 10:14, it's now 21:16 and still going
> so 9 hours at least on this machine to build a browser - almost as bad
> as openoffice at it's worst (regularly took 12 hours). Nodejs also took
> a while, but I didn't record time.

Chromium is definitely the worst, and strangely variable. The last few
compiles have taken between 6 and 14 hours. Since it takes longer than
everything else to build, it is usually compiling on its own, so parallel
emerges aren't a factor.

Qtwebengine is also bad, not surprising as it is a cut down Chromium.
Emerging world with --exclude then timing build to coincide with sleep
helps, although I haven't quite reached the age where I need 14 hours of
sleep a day.


-- 
Neil Bothwick

If it isn't broken, I can fix it.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] long compiles

[Alan McKinnon]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick <neil@digimed.co.uk> wrote:

> On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote:
>
> > chromium has been building since 10:14, it's now 21:16 and still going
> > so 9 hours at least on this machine to build a browser - almost as bad
> > as openoffice at it's worst (regularly took 12 hours). Nodejs also took
> > a while, but I didn't record time.
>
> Chromium is definitely the worst, and strangely variable. The last few
> compiles have taken between 6 and 14 hours. Since it takes longer than
> everything else to build, it is usually compiling on its own, so parallel
> emerges aren't a factor.
>
> Qtwebengine is also bad, not surprising as it is a cut down Chromium.
> Emerging world with --exclude then timing build to coincide with sleep
> helps, although I haven't quite reached the age where I need 14 hours of
> sleep a day.
>
>
> --
> Neil Bothwick
>
> If it isn't broken, I can fix it.
>

Yup, that jibes with what I see. Oh well, just means that the need for
overnight compiles did not go away haha

Thanks to every one else that replied too - everyone said much the same
thing so I figured one replay to rule them all was the best way


Alan
-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

[-- Attachment #2: Type: text/html, Size: 1885 bytes --]

Re: [gentoo-user] long compiles

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2156 bytes --]

On Monday, 11 September 2023 21:21:47 BST Alan McKinnon wrote:
> On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick <neil@digimed.co.uk> wrote:
> > On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote:
> > > chromium has been building since 10:14, it's now 21:16 and still going
> > > so 9 hours at least on this machine to build a browser - almost as bad
> > > as openoffice at it's worst (regularly took 12 hours). Nodejs also took
> > > a while, but I didn't record time.
> > 
> > Chromium is definitely the worst, and strangely variable. The last few
> > compiles have taken between 6 and 14 hours. Since it takes longer than
> > everything else to build, it is usually compiling on its own, so parallel
> > emerges aren't a factor.
> > 
> > Qtwebengine is also bad, not surprising as it is a cut down Chromium.
> > Emerging world with --exclude then timing build to coincide with sleep
> > helps, although I haven't quite reached the age where I need 14 hours of
> > sleep a day.
> > 
> > 
> > --
> > Neil Bothwick
> > 
> > If it isn't broken, I can fix it.
> 
> Yup, that jibes with what I see. Oh well, just means that the need for
> overnight compiles did not go away haha
> 
> Thanks to every one else that replied too - everyone said much the same
> thing so I figured one replay to rule them all was the best way
> 
> 
> Alan

As the old saying goes, "there ain't no substitute to cubic inches".  Moar 
cores and moar RAM is almost always the solution, but with laptops and older 
PCs in general overnight builds soon become inevitable.  Selectively reducing 
jobs and adding swap, or for packages like rust placing /var/tmp/portage on 
the disk becomes necessary.

A solution I use for older/smaller laptops is to build binaries on a more 
powerful PC and emerge these in turn on the weaker PCs.

There's also the option of using bin alternatives where available, e.g. 
google-chrome, firefox-bin, libreoffice-bin.

Finally, there is a small scale project to provide systemd based binaries as 
an alternative to building your own:

https://wiki.gentoo.org/wiki/Experimental_binary_package_host

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] long compiles

[Alan McKinnon]

[-- Attachment #1: Type: text/plain, Size: 2729 bytes --]

On Mon, Sep 11, 2023 at 11:23 PM Michael <confabulate@kintzios.com> wrote:

> On Monday, 11 September 2023 21:21:47 BST Alan McKinnon wrote:
> > On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick <neil@digimed.co.uk>
> wrote:
> > > On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote:
> > > > chromium has been building since 10:14, it's now 21:16 and still
> going
> > > > so 9 hours at least on this machine to build a browser - almost as
> bad
> > > > as openoffice at it's worst (regularly took 12 hours). Nodejs also
> took
> > > > a while, but I didn't record time.
> > >
> > > Chromium is definitely the worst, and strangely variable. The last few
> > > compiles have taken between 6 and 14 hours. Since it takes longer than
> > > everything else to build, it is usually compiling on its own, so
> parallel
> > > emerges aren't a factor.
> > >
> > > Qtwebengine is also bad, not surprising as it is a cut down Chromium.
> > > Emerging world with --exclude then timing build to coincide with sleep
> > > helps, although I haven't quite reached the age where I need 14 hours
> of
> > > sleep a day.
> > >
> > >
> > > --
> > > Neil Bothwick
> > >
> > > If it isn't broken, I can fix it.
> >
> > Yup, that jibes with what I see. Oh well, just means that the need for
> > overnight compiles did not go away haha
> >
> > Thanks to every one else that replied too - everyone said much the same
> > thing so I figured one replay to rule them all was the best way
> >
> >
> > Alan
>
> As the old saying goes, "there ain't no substitute to cubic inches".  Moar
> cores and moar RAM is almost always the solution, but with laptops and
> older
> PCs in general overnight builds soon become inevitable.  Selectively
> reducing
> jobs and adding swap, or for packages like rust placing /var/tmp/portage
> on
> the disk becomes necessary.
>
> A solution I use for older/smaller laptops is to build binaries on a more
> powerful PC and emerge these in turn on the weaker PCs.
>
> There's also the option of using bin alternatives where available, e.g.
> google-chrome, firefox-bin, libreoffice-bin.
>
> Finally, there is a small scale project to provide systemd based binaries
> as
> an alternative to building your own:
>
> https://wiki.gentoo.org/wiki/Experimental_binary_package_host
>

As it turns out this laptop is the most powerful machine I have available,
my large collection of previous work laptops are getting older and older.

Although, I *could* create a ginormous build host on one of the
virtualization clusters at work hahaha :-)

That link looks interesting, I'll check it out, thanks!


-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

[-- Attachment #2: Type: text/html, Size: 3715 bytes --]

Re: [gentoo-user] long compiles

[Ramon Fischer]

[-- Attachment #1.1.1: Type: text/plain, Size: 3366 bytes --]

You may also want to take a look at "distcc", with which you can set up 
compiler farms; this can be even combined with "ccache":

     https://wiki.gentoo.org/wiki/Distcc#With_ccache

-Ramon

On 11/09/2023 23:46, Alan McKinnon wrote:
>
>
> On Mon, Sep 11, 2023 at 11:23 PM Michael <confabulate@kintzios.com> wrote:
>
>     On Monday, 11 September 2023 21:21:47 BST Alan McKinnon wrote:
>     > On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick
>     <neil@digimed.co.uk> wrote:
>     > > On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote:
>     > > > chromium has been building since 10:14, it's now 21:16 and
>     still going
>     > > > so 9 hours at least on this machine to build a browser -
>     almost as bad
>     > > > as openoffice at it's worst (regularly took 12 hours).
>     Nodejs also took
>     > > > a while, but I didn't record time.
>     > >
>     > > Chromium is definitely the worst, and strangely variable. The
>     last few
>     > > compiles have taken between 6 and 14 hours. Since it takes
>     longer than
>     > > everything else to build, it is usually compiling on its own,
>     so parallel
>     > > emerges aren't a factor.
>     > >
>     > > Qtwebengine is also bad, not surprising as it is a cut down
>     Chromium.
>     > > Emerging world with --exclude then timing build to coincide
>     with sleep
>     > > helps, although I haven't quite reached the age where I need
>     14 hours of
>     > > sleep a day.
>     > >
>     > >
>     > > --
>     > > Neil Bothwick
>     > >
>     > > If it isn't broken, I can fix it.
>     >
>     > Yup, that jibes with what I see. Oh well, just means that the
>     need for
>     > overnight compiles did not go away haha
>     >
>     > Thanks to every one else that replied too - everyone said much
>     the same
>     > thing so I figured one replay to rule them all was the best way
>     >
>     >
>     > Alan
>
>     As the old saying goes, "there ain't no substitute to cubic
>     inches".  Moar
>     cores and moar RAM is almost always the solution, but with laptops
>     and older
>     PCs in general overnight builds soon become inevitable.
>     Selectively reducing
>     jobs and adding swap, or for packages like rust placing
>     /var/tmp/portage on
>     the disk becomes necessary.
>
>     A solution I use for older/smaller laptops is to build binaries on
>     a more
>     powerful PC and emerge these in turn on the weaker PCs.
>
>     There's also the option of using bin alternatives where available,
>     e.g.
>     google-chrome, firefox-bin, libreoffice-bin.
>
>     Finally, there is a small scale project to provide systemd based
>     binaries as
>     an alternative to building your own:
>
>     https://wiki.gentoo.org/wiki/Experimental_binary_package_host
>
>
> As it turns out this laptop is the most powerful machine I have 
> available, my large collection of previous work laptops are getting 
> older and older.
>
> Although, I *could* create a ginormous build host on one of the 
> virtualization clusters at work hahaha :-)
>
> That link looks interesting, I'll check it out, thanks!
>
>
> -- 
> Alan McKinnon
> alan dot mckinnon at gmail dot com

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 8969 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-user] long compiles

[Alan McKinnon]

[-- Attachment #1: Type: text/plain, Size: 3947 bytes --]

On Tue, Sep 12, 2023 at 1:05 AM Ramon Fischer <Ramon_Fischer@hotmail.de>
wrote:

> You may also want to take a look at "distcc", with which you can set up
> compiler farms; this can be even combined with "ccache":
>
>      https://wiki.gentoo.org/wiki/Distcc#With_ccache
>
> -Ramon
>


Hi Ramon,

distcc is way more than I need. I'm not complaining about long compile
times and wanting a solution, I was more curious about which packages these
days take long compared to when I was last here 5/6 years ago

Alan




>
> On 11/09/2023 23:46, Alan McKinnon wrote:
> >
> >
> > On Mon, Sep 11, 2023 at 11:23 PM Michael <confabulate@kintzios.com>
> wrote:
> >
> >     On Monday, 11 September 2023 21:21:47 BST Alan McKinnon wrote:
> >     > On Mon, Sep 11, 2023 at 10:05 PM Neil Bothwick
> >     <neil@digimed.co.uk> wrote:
> >     > > On Mon, 11 Sep 2023 21:19:27 +0200, Alan McKinnon wrote:
> >     > > > chromium has been building since 10:14, it's now 21:16 and
> >     still going
> >     > > > so 9 hours at least on this machine to build a browser -
> >     almost as bad
> >     > > > as openoffice at it's worst (regularly took 12 hours).
> >     Nodejs also took
> >     > > > a while, but I didn't record time.
> >     > >
> >     > > Chromium is definitely the worst, and strangely variable. The
> >     last few
> >     > > compiles have taken between 6 and 14 hours. Since it takes
> >     longer than
> >     > > everything else to build, it is usually compiling on its own,
> >     so parallel
> >     > > emerges aren't a factor.
> >     > >
> >     > > Qtwebengine is also bad, not surprising as it is a cut down
> >     Chromium.
> >     > > Emerging world with --exclude then timing build to coincide
> >     with sleep
> >     > > helps, although I haven't quite reached the age where I need
> >     14 hours of
> >     > > sleep a day.
> >     > >
> >     > >
> >     > > --
> >     > > Neil Bothwick
> >     > >
> >     > > If it isn't broken, I can fix it.
> >     >
> >     > Yup, that jibes with what I see. Oh well, just means that the
> >     need for
> >     > overnight compiles did not go away haha
> >     >
> >     > Thanks to every one else that replied too - everyone said much
> >     the same
> >     > thing so I figured one replay to rule them all was the best way
> >     >
> >     >
> >     > Alan
> >
> >     As the old saying goes, "there ain't no substitute to cubic
> >     inches".  Moar
> >     cores and moar RAM is almost always the solution, but with laptops
> >     and older
> >     PCs in general overnight builds soon become inevitable.
> >     Selectively reducing
> >     jobs and adding swap, or for packages like rust placing
> >     /var/tmp/portage on
> >     the disk becomes necessary.
> >
> >     A solution I use for older/smaller laptops is to build binaries on
> >     a more
> >     powerful PC and emerge these in turn on the weaker PCs.
> >
> >     There's also the option of using bin alternatives where available,
> >     e.g.
> >     google-chrome, firefox-bin, libreoffice-bin.
> >
> >     Finally, there is a small scale project to provide systemd based
> >     binaries as
> >     an alternative to building your own:
> >
> >     https://wiki.gentoo.org/wiki/Experimental_binary_package_host
> >
> >
> > As it turns out this laptop is the most powerful machine I have
> > available, my large collection of previous work laptops are getting
> > older and older.
> >
> > Although, I *could* create a ginormous build host on one of the
> > virtualization clusters at work hahaha :-)
> >
> > That link looks interesting, I'll check it out, thanks!
> >
> >
> > --
> > Alan McKinnon
> > alan dot mckinnon at gmail dot com
>
> --
> GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
>
>

-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

[-- Attachment #2: Type: text/html, Size: 5841 bytes --]

[gentoo-user] Re: long compiles

[Nikos Chantziaras]

On 11/09/2023 22:19, Alan McKinnon wrote:
> chromium has been building since 10:14, it's now 21:16 and still going 
> so 9 hours at least on this machine to build a browser - almost as bad 
> as openoffice at it's worst (regularly took 12 hours). Nodejs also took 
> a while, but I didn't record time.

What's your CPU and how much RAM? Even on my older system I had (an 
4-core i5 2500K) libreoffice took like 2 hours or so to build.


> What other packages have huge build times?

IIRC, dev-qt/qtwebengine is one of the heaviest when it comes to build 
times.

Anyway, a nice way to cut down on build times is to build on tmpfs. To 
do that however with heavy packages like that, I had to upgrade to 32GB 
RAM. There was a large price drop in the memory market a couple months 
ago, so I snatched a 32GB DDR4 3600 kit (2x16GB) for like 80€. So now 
with plenty of RAM, I configured a 14GB tmpfs in /var/tmp/portage. I 
never hit swap when emerging.

[gentoo-user] Re: long compiles

[Nikos Chantziaras]

On 11/09/2023 23:21, Alan McKinnon wrote:
> Yup, that jibes with what I see. Oh well, just means that the need for 
> overnight compiles did not go away haha

Ever since I added the following to my make.conf:

PORTAGE_NICENESS=19
PORTAGE_IONICE_COMMAND="sh -c \"schedtool -D \${PID} && ionice -c 3 -p 
\${PID}\""

I never needed overnight compiles again. Make sure sys-process/schedtool 
is installed. As long as you have plenty of RAM so the system doesn't 
swap, you can use the system normally even while building monster 
packages. I can even play video games without issue while portage is 
emerging now.

Re: [gentoo-user] long compiles

[Peter Humphrey]

On Monday, 11 September 2023 22:22:28 BST Michael wrote:

> There's also the option of using bin alternatives where available, e.g.
> google-chrome, firefox-bin, libreoffice-bin.

...and rust-bin, which is now the default in at least some desktop profiles.

-- 
Regards,
Peter.

Re: [gentoo-user] long compiles

[Jacques Montier]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

Le mar. 12 sept. 2023, 12:14, Peter Humphrey <peter@prh.myzen.co.uk> a
écrit :

> On Monday, 11 September 2023 22:22:28 BST Michael wrote:
>
> > There's also the option of using bin alternatives where available, e.g.
> > google-chrome, firefox-bin, libreoffice-bin.
>
> ...and rust-bin, which is now the default in at least some desktop
> profiles.
>
> --
> Regards,
> Peter.
>
>
>
>
> Hello,


I could get rid of webkit-gtk with pleasure and i use rust-bin.
I tried libreoffice-bin but encountered some issues with pdf import, so i
went back to libreoffice.

Cheers,

Jacques



>
>

[-- Attachment #2: Type: text/html, Size: 1538 bytes --]

Re: [gentoo-user] Re: long compiles

[Alan McKinnon]

[-- Attachment #1: Type: text/plain, Size: 1587 bytes --]

On Tue, Sep 12, 2023 at 11:19 AM Nikos Chantziaras <realnc@gmail.com> wrote:

> On 11/09/2023 22:19, Alan McKinnon wrote:
> > chromium has been building since 10:14, it's now 21:16 and still going
> > so 9 hours at least on this machine to build a browser - almost as bad
> > as openoffice at it's worst (regularly took 12 hours). Nodejs also took
> > a while, but I didn't record time.
>
> What's your CPU and how much RAM? Even on my older system I had (an
> 4-core i5 2500K) libreoffice took like 2 hours or so to build.
>
>
> > What other packages have huge build times?
>
> IIRC, dev-qt/qtwebengine is one of the heaviest when it comes to build
> times.
>
> Anyway, a nice way to cut down on build times is to build on tmpfs. To
> do that however with heavy packages like that, I had to upgrade to 32GB
> RAM. There was a large price drop in the memory market a couple months
> ago, so I snatched a 32GB DDR4 3600 kit (2x16GB) for like 80€. So now
> with plenty of RAM, I configured a 14GB tmpfs in /var/tmp/portage. I
> never hit swap when emerging.
>

That's not an option for me, this is a corporate laptop with 16G RAM and a
case I may not open :-)
I'm not interested in a remote build host or distcc either

But anyways, this is not really about how to deal with long compiles, I was
asking what current packages take a long time after a 5 year absence.

The answer is what it was always - browsers and libreoffice. I do recall
icu being a bit of a beast back then


Alan




-- 
Alan McKinnon
alan dot mckinnon at gmail dot com

[-- Attachment #2: Type: text/html, Size: 2229 bytes --]

[gentoo-user] Re: long compiles

[Nuno Silva]

On 2023-09-12, Alan McKinnon wrote:
[...]
> But anyways, this is not really about how to deal with long compiles, I was
> asking what current packages take a long time after a 5 year absence.
>
> The answer is what it was always - browsers and libreoffice. I do recall
> icu being a bit of a beast back then

I remember insn-attrtab.c making the GCC compilation swap a lot :-)

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=29442

-- 
Nuno Silva

Re: [gentoo-user] long compiles

[Wol]

On 11/09/2023 20:46, Alan McKinnon wrote:
> qtwebengine! yes that one took forever also. It also said my 16G of RAM 
> was smaller than the 16G it needed. Weird.
> 
> Anyways I enabled a swapfile and left it to run overnight

16GB physical ram <> 16GB usable ram for the compile ...

I concur with others that tmpfs is the way to go - I don't think my 
system is set up that way just now, but I always have googols of swap 
(twice max physical ram per disk) so I just declare a huge ramdisk for 
compiling on.

My current system has four ram slots, two maxed out with 16GB chips 
each, so that makes 128GB swap partitions per disk (4 of them) equals 
512GB swap ...

(Yes the people at openSUSE said I was being stupid with that much swap)

But declare a 128GB ramdisk, and it'll spill over as required, but 
anything that fits in ram will compile very quick.

And yes, I also used to have my systems configured so they had one 
shared portage area, matching make.conf, and were set up to "use binary 
if it exists, else build one". I had the opposite problem though, my 
nice fast system had a habit of crashing, so I used the old slow one to 
build most things, because it was more reliable. Hey ho.

There's all sorts of tricks, some work for some people, others work for 
others.

Cheers,
Wol

Re: [gentoo-user] Re: long compiles

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 589 bytes --]

On Tue, 12 Sep 2023 21:01:48 +0200, Alan McKinnon wrote:

> But anyways, this is not really about how to deal with long compiles, I
> was asking what current packages take a long time after a 5 year
> absence.
> 
> The answer is what it was always - browsers and libreoffice. I do recall
> icu being a bit of a beast back then

LibreOffice doesn't seem too bad these days. icu and boost are a pain
because of the number of other packages they rebuild.


-- 
Neil Bothwick

Light travels faster than sound. This is why some people appear bright
until you hear them speak.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] long compiles

[Peter Humphrey]

On Tuesday, 12 September 2023 22:08:49 BST Wol wrote:

> There's all sorts of tricks, some work for some people, others work for
> others.

Quite so. Here I have two swap partitions: 8GB priority 20 on NVME and 50GB 
priority 10 on SSD. I've never noticed either of them being used, so I suppose 
I could dispense with the smaller one. When I bought the machine I wanted it 
to be as powerful as I could reasonably justify (to run BOINC projects, making 
my own small contribution to the state of knowlege in astrophysics), so it has 
64 GB RAM in its four slots.

Tmpfs earns its keep here. I don't set limits on it, preferring to let the 
kernel manage it itself. One tmpfs is on /tmp, the other on /var/tmp/portage. 
With all that RAM to play in, swap is rarely used, if ever.

A thought on compiling, which I hope some devs will read: I was tempted to 
push the system hard at first, with load average and jobs as high as I thought 
I could set them. I've come to believe, though, that job control by portage 
and /usr/bin/make is weak at very high loads, because I would usually find that 
a few packages had failed to compile; also that some complex programs were 
sometimes unstable. Therefore I've had to throttle the system to be sure(r) of 
correctness. Seems a waste. Thus:

$ grep '\-j' /etc/portage/make.conf
EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32 [...]"
MAKEOPTS="-j14"

That 14 will revert to its previous 12 if I find things going bump in the night 
again, or perhaps go up a bit more if not.

-- 
Regards,
Peter.

Re: [gentoo-user] long compiles

[Wols Lists]

On 13/09/2023 12:28, Peter Humphrey wrote:
> A thought on compiling, which I hope some devs will read: I was tempted to
> push the system hard at first, with load average and jobs as high as I thought
> I could set them. I've come to believe, though, that job control by portage
> and /usr/bin/make is weak at very high loads, because I would usually find that
> a few packages had failed to compile; also that some complex programs were
> sometimes unstable. Therefore I've had to throttle the system to be sure(r) of
> correctness. Seems a waste. Thus:

Bear in mind a lot of systems are thermally limited and can't run at 
full pelt anyway ...

You might find it's actually better (and more efficient) to run at lower 
loading. Certainly following the kernel lists you get the impression 
that the CPU regularly goes into thermal throttling under heavy load, 
and also that using a couple of cores lightly is more efficient than 
using one core heavily.

It's so difficult to know what's best ... (because too many people make 
decisions based on their interests, and then when you come along their 
decisions may conflict with each other and certainly conflict with you ...)

Cheers,
Wol

Re: [gentoo-user] long compiles

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]

Am Wed, Sep 13, 2023 at 12:50:20PM +0100 schrieb Wols Lists:

> Bear in mind a lot of systems are thermally limited and can't run at full
> pelt anyway ...

Usually those are space-constrained systems like mini PCs or laptops. 
Typical Desktops shouldn’t be limited; even the stock CPU coolers should be 
capable of dissipating all heat, as long as the case has enough air flow.

> You might find it's actually better (and more efficient) to run at lower
> loading. Certainly following the kernel lists you get the impression that
> the CPU regularly goes into thermal throttling under heavy load, and also
> that using a couple of cores lightly is more efficient than using one core
> heavily.

At least very current CPUs tend to go into so high clock speeds that they 
become quite inefficient. If you set a 105 W Ryzen 7700X to 65 W eco mode in 
the BIOS (which means that the actual maximum intake goes down from 144 W to 
84 W), you reduce consuption by a third, but only lose ~15 % in performance.

At very low figures (15 W), Ryzen 5000 and 7000 CPUs are almost as efficient 
as Apple M1.

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Einstein is dead.  Newton is dead.  I’m feeling sick, too.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] long compiles

[Peter Humphrey]

On Wednesday, 13 September 2023 12:50:20 BST Wols Lists wrote:
> On 13/09/2023 12:28, Peter Humphrey wrote:
> > A thought on compiling, which I hope some devs will read: I was tempted to
> > push the system hard at first, with load average and jobs as high as I
> > thought I could set them. I've come to believe, though, that job control
> > by portage and /usr/bin/make is weak at very high loads, because I would
> > usually find that a few packages had failed to compile; also that some
> > complex programs were sometimes unstable. Therefore I've had to throttle
> > the system to be sure(r) of correctness. Seems a waste.

> Bear in mind a lot of systems are thermally limited and can't run at
> full pelt anyway ...

No doubt, but apparently not this box: I run it 24x7 with all 24 CPU threads 
fully loaded with floating-point calculations, which make a good deal more heat 
than 'mere' compiling with (I assume) integer arithmetic.   :)

> You might find it's actually better (and more efficient) to run at lower
> loading. Certainly following the kernel lists you get the impression
> that the CPU regularly goes into thermal throttling under heavy load,
> and also that using a couple of cores lightly is more efficient than
> using one core heavily.

See above; besides, I have to limit the load anyway when compiling, for the 
reasons I gave last time.

> It's so difficult to know what's best ... (because too many people make
> decisions based on their interests, and then when you come along their
> decisions may conflict with each other and certainly conflict with you ...)

I agree with you there Wol, even without the parenthesis.  :)

-- 
Regards,
Peter.

Re: [gentoo-user] long compiles

[Michael]

[-- Attachment #1: Type: text/plain, Size: 3026 bytes --]

On Wednesday, 13 September 2023 13:41:00 BST Peter Humphrey wrote:
> On Wednesday, 13 September 2023 12:50:20 BST Wols Lists wrote:
> > On 13/09/2023 12:28, Peter Humphrey wrote:
> > > A thought on compiling, which I hope some devs will read: I was tempted
> > > to
> > > push the system hard at first, with load average and jobs as high as I
> > > thought I could set them. I've come to believe, though, that job control
> > > by portage and /usr/bin/make is weak at very high loads, because I would
> > > usually find that a few packages had failed to compile; also that some
> > > complex programs were sometimes unstable. Therefore I've had to throttle
> > > the system to be sure(r) of correctness. Seems a waste.
> > 
> > Bear in mind a lot of systems are thermally limited and can't run at
> > full pelt anyway ...
> 
> No doubt, but apparently not this box: I run it 24x7 with all 24 CPU threads
> fully loaded with floating-point calculations, which make a good deal more
> heat than 'mere' compiling with (I assume) integer arithmetic.   :)

I recall this being discussed in a previous thread, but if your CPU has 24 
threads and you've set:

EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32
MAKEOPTS="-j14"

You will be asking emerge to run up to 4 x 14 = 56 threads, which could 
potentially eat up to 2G of RAM each, i.e. 112G of RAM.  This will exhaust 
your 64G of RAM, not taking into account whatever else the OS will be trying 
to run at the time. The --load-average=32 is normally expected to be a 
floating point number indicating a percentage of load x the number of cores; 
e.g. for 12 cores you could set it at 12 x 0.9 = 10.8 to limit the load to 90% 
so as maintain some system responsiveness.

Of course, not all emerges use make and you may never or rarely emerge 4 x 
monster packages in parallel to need 2G of RAM for each thread at the same 
time.

If only we had at our disposal some AI algorithm to calculate dynamically each 
time we run emerge the optimal combo of parallel emerge jobs and number of 
make tasks, so as to achieve the highest total time saving Vs energy spent!  
Or just the highest total time saving.  ;-)

I haven't performed any meaningful comparisons to determine where the greatest 
gains are to be achieved.  Parallel emerges of many small packages, or a large 
number of make jobs for big packages.  The combination would change each time 
according to the individual packages waiting for an update.  In my use case, 
instinctively feels more beneficial reducing the time I have to wait for huge 
packages like qtwebengine to finish, than accelerating the updates of half a 
dozen smaller packages.  Therefore, as a rule I leave EMERGE_DEFAULT_OPTS 
unset.  I set MAKEOPTS jobs to the number of CPU threads +1 and the load 
average at 95%, so I can continue using the PC without any noticeable latency.  
On PCs where RAM is constrained I reduce the MAKEOPTS in /etc/portage/
package.env for any large packages which are bound to start swapping and 
thrashing the disk.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] long compiles

[Kristian Poul Herkild]

Hi.

Nothing compares to Chromium (browser) in terms of compilation times. On 
my system with 12 core threads it takes about 8 hours to compile - which 
is 4 times longer than 10 years ago with 2 core threads ;)

Libreoffice takes a few hours, but less than half of chromium. Nothing 
gets close to Chromium. But otherwise webkitgtk and qtwebengine are to 
big ones - but still about a quarter of Chromium.

Kristian Poul Herkild

Den 11.09.2023 kl. 21.19 skrev Alan McKinnon:
> After my long time away from Gentoo, I thought perhaps some packages 
> that always took ages to compile would have improved. I needed to change 
> to ~amd64 anyway (dumb n00b mistake leaving it at amd64). So that's what 
> I did and let emerge do it's thing.
> 
> chromium has been building since 10:14, it's now 21:16 and still going 
> so 9 hours at least on this machine to build a browser - almost as bad 
> as openoffice at it's worst (regularly took 12 hours). Nodejs also took 
> a while, but I didn't record time.
> 
> 
> What other packages have huge build times?
> 
> -- 
> Alan McKinnon
> alan dot mckinnon at gmail dot com

[gentoo-user] Re: long compiles

[Grant Edwards]

On 2023-09-13, Kristian Poul Herkild <kristian@herkild.dk> wrote:

> Nothing compares to Chromium (browser) in terms of compilation times. On 
> my system with 12 core threads it takes about 8 hours to compile - which 
> is 4 times longer than 10 years ago with 2 core threads ;)

About a year ago I finally gave up building Chromium and switched to
www-client/google-chrome.  It got to the point where it sometimes took
longer to build Chromium than it did for the next version to come out.

--
Grant

Re: [gentoo-user] Re: long compiles

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

On Wed, 13 Sep 2023 21:20:20 -0000 (UTC), Grant Edwards wrote:

> About a year ago I finally gave up building Chromium and switched to
> www-client/google-chrome.  It got to the point where it sometimes took
> longer to build Chromium than it did for the next version to come out.

That's why I run stable Chromium on an otherwise testing system.


-- 
Neil Bothwick

We all know what comes after 'X', said Tom, wisely.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] long compiles

[Peter Humphrey]

On Wednesday, 13 September 2023 16:14:09 BST Michael wrote:

> I recall this being discussed in a previous thread, but if your CPU has 24
> threads and you've set:
> 
> EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32
> MAKEOPTS="-j14"
> 
> You will be asking emerge to run up to 4 x 14 = 56 threads, which could
> potentially eat up to 2G of RAM each, i.e. 112G of RAM.  This will exhaust
> your 64G of RAM, not taking into account whatever else the OS will be trying
> to run at the time.

Yes, I understand that, but I've spent a lot of time watching, and 
instrumenting, and in practice the load doesn't exceed about 33.

> The --load-average=32 is normally expected to be a floating point number

That stipulation has only appeared recently. I have tried adding a '.0' to the 
number, and it's made no noticeaible difference. Besides, I could attempt 
pedantry and declare that the set of all integers is a proper subset of all 
real numbers.  ;-)

> Of course, not all emerges use make and you may never or rarely emerge 4 x
> monster packages in parallel to need 2G of RAM for each thread at the same
> time.

I have actually had three or four big packages running together, but my 
observation is that they don't pump the load up too far.

> If only we had at our disposal some AI algorithm to calculate dynamically
> each time we run emerge the optimal combo of parallel emerge jobs and
> number of make tasks, so as to achieve the highest total time saving Vs
> energy spent! Or just the highest total time saving.  ;-)

Yes, that's what we need, all right.

> I haven't performed any meaningful comparisons to determine where the
> greatest gains are to be achieved.  Parallel emerges of many small
> packages, or a large number of make jobs for big packages.  The combination
> would change each time according to the individual packages waiting for an
> update.  In my use case, instinctively feels more beneficial reducing the
> time I have to wait for huge packages like qtwebengine to finish, than
> accelerating the updates of half a dozen smaller packages.

That is the difficulty. I do often rebuild a new system, not trusting the 
existing one any longer, and a lot of time is spent fiddling with four tiny 
packages at a time in the early and middle stages, then benefitting from the 
limit of 4 once the desktop packages begin.

> Therefore, as a rule I leave EMERGE_DEFAULT_OPTS unset.  I set MAKEOPTS jobs
> to the number of CPU threads +1 and the load average at 95%, so I can
> continue using the PC without any noticeable latency. On PCs where RAM is
> constrained I reduce the MAKEOPTS in /etc/portage/ package.env for any large
> packages which are bound to start swapping and thrashing the disk.

Interesting. Do you mean 95% of the jobs figure? I'll do some more 
experimenting.

Meanwhile, perhaps I'll run new builds in two stages: the first without any 
desktop packages - I do have sets defined to enable that - and the second with 
them. I can't do that with emerge -e though.

-- 
Regards,
Peter.

Re: [gentoo-user] long compiles

[Michael]

[-- Attachment #1: Type: text/plain, Size: 4304 bytes --]

On Thursday, 14 September 2023 10:49:35 BST Peter Humphrey wrote:
> On Wednesday, 13 September 2023 16:14:09 BST Michael wrote:
> > I recall this being discussed in a previous thread, but if your CPU has 24
> > threads and you've set:
> > 
> > EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=32
> > MAKEOPTS="-j14"
> > 
> > You will be asking emerge to run up to 4 x 14 = 56 threads, which could
> > potentially eat up to 2G of RAM each, i.e. 112G of RAM.  This will exhaust
> > your 64G of RAM, not taking into account whatever else the OS will be
> > trying to run at the time.
> 
> Yes, I understand that, but I've spent a lot of time watching, and
> instrumenting, and in practice the load doesn't exceed about 33.
> 
> > The --load-average=32 is normally expected to be a floating point number
> 
> That stipulation has only appeared recently. I have tried adding a '.0' to
> the number, and it's made no noticeaible difference. Besides, I could
> attempt pedantry and declare that the set of all integers is a proper
> subset of all real numbers.  ;-)
> 
> > Of course, not all emerges use make and you may never or rarely emerge 4 x
> > monster packages in parallel to need 2G of RAM for each thread at the same
> > time.
> 
> I have actually had three or four big packages running together, but my
> observation is that they don't pump the load up too far.
> 
> > If only we had at our disposal some AI algorithm to calculate dynamically
> > each time we run emerge the optimal combo of parallel emerge jobs and
> > number of make tasks, so as to achieve the highest total time saving Vs
> > energy spent! Or just the highest total time saving.  ;-)
> 
> Yes, that's what we need, all right.
> 
> > I haven't performed any meaningful comparisons to determine where the
> > greatest gains are to be achieved.  Parallel emerges of many small
> > packages, or a large number of make jobs for big packages.  The
> > combination
> > would change each time according to the individual packages waiting for an
> > update.  In my use case, instinctively feels more beneficial reducing the
> > time I have to wait for huge packages like qtwebengine to finish, than
> > accelerating the updates of half a dozen smaller packages.
> 
> That is the difficulty. I do often rebuild a new system, not trusting the
> existing one any longer, and a lot of time is spent fiddling with four tiny
> packages at a time in the early and middle stages, then benefitting from the
> limit of 4 once the desktop packages begin.

I very rarely reinstall.  So rarely, I can't remember the last time I did it.


> > Therefore, as a rule I leave EMERGE_DEFAULT_OPTS unset.  I set MAKEOPTS
> > jobs to the number of CPU threads +1 and the load average at 95%, so I
> > can continue using the PC without any noticeable latency. On PCs where
> > RAM is constrained I reduce the MAKEOPTS in /etc/portage/ package.env for
> > any large packages which are bound to start swapping and thrashing the
> > disk.
> Interesting. Do you mean 95% of the jobs figure? I'll do some more
> experimenting.

Yes.  Example:

A CPU with 4 cores and 8 threads is set like this:

MAKEOPTS="-j8 -l7.2"

This allows up to 8 make jobs at a time, with an average load being kept at 
90% to leave some breathing space for day to day desktop work and minimise too 
much/frequent swapping:

8 x 0.9 = 7.2

Looking at top shows 7 to 8 compiling jobs at a time.  The PC in question has 
16G of RAM.  I've observed small packages rarely if ever reach 2G of RAM per 
job.  So, I set the above MAKEOPTS in make.conf, but bypass it for large 
packages in /etc/portage/env/ by setting, e.g.:

/etc/portage/env/j4.conf

with MAKEOPTS="-j4" in it and specify this for each large package in /etc/
portage/package.env, e.g.:

www-client/qtwebengine -j4.conf
www-client/firefox -j4.conf
dev-lang/rust j4.conf
etc.


> Meanwhile, perhaps I'll run new builds in two stages: the first without any
> desktop packages - I do have sets defined to enable that - and the second
> with them. I can't do that with emerge -e though.

Unlike MAKEOPTS, the EMERGE_DEFAULT_OPTS variable cannot be set in /etc/
portage/package.env.  Therefore, I leave the latter unset in make.conf and 
specify it on the CLI when I'm about to update a lot of smaller packages and 
remember to do it. ;-)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]