[gentoo-user] "emerge --jobs=1 ..." vs "MAKEOPTS=-j1 emerge ..."

[gentoo-user] "emerge --jobs=1 ..." vs "MAKEOPTS=-j1 emerge ..."

[Dr Rainer Woitok]

Greetings,

is there any difference between running "emerge --jobs=1 ..."  and runn-
ing "MAKEOPTS=-j1 emerge ..."?

Sincerely,
  Rainer

[gentoo-user] Re: "emerge --jobs=1 ..." vs "MAKEOPTS=-j1 emerge ..."

[Holger Hoffstätte]

On 2020-06-21 11:53, Dr Rainer Woitok wrote:
> Greetings,
> 
> is there any difference between running "emerge --jobs=1 ..."  and runn-
> ing "MAKEOPTS=-j1 emerge ..."?

--jobs=1 starts one package build at a time, possibly using many parallel
processes - depending on MAKEOPTS and how the build works.

MAKEOPTS sets the number of parallel processes within one package build.

They are complementary, and you can combine them. However frequently
a series of packages have dependencies (a before b before c), so starting
multiple --jobs may not have any effect. Or it may, e.g. in the case of
rebuilds or independent packages that often only use one process to build,
e.g. perl modules or something smaller. --jobs can really make a difference
since otherwise you would suffer from portage/script/shell overhead relative
to the actual work being performed.

make offers the -l parameter that constrains the number of processes by
load, and adjusts them accordingly. As an example, with 8 cores you could
use MAKEOPTS="-j8 -l8" to enable inter-ebuild parallelism and a max. load
for one package, then start several --jobs to build independent packages
without waiting for each other.

Hope this helps.

-h

Re: [gentoo-user] "emerge --jobs=1 ..." vs "MAKEOPTS=-j1 emerge ..."

[Rich Freeman]

On Sun, Jun 21, 2020 at 5:53 AM Dr Rainer Woitok
<rainer.woitok@gmail.com> wrote:
>
> is there any difference between running "emerge --jobs=1 ..."  and runn-
> ing "MAKEOPTS=-j1 emerge ..."?
>

Yes.

--jobs=1 tells emerge to build one package at a time.

-j1 tells make to compile one file at a time when building a package.

So, if you have 10 independent packages containing 10 independent C
files, --jobs=10 and -j10 will build all 100 C files in parallel.
--jobs=1 and -j10 will build each package in series, with all 10 C
files in each package in parallel.  --jobs=10 and -j1 will build all
10 packages in parallel, but each one will build one C file at a time.

If you have the cores to spare each is useful, because rarely are
either packages or the files within them completely independent.  So
running both in parallel maximizes the opportunity to keep your cores
busy, of course at the cost of memory if you spawn too many.  It is
hard to perfectly predict the optimal combination since portage can't
tell how parallel any particular package is.

Setting -j1 is necessary in some build systems due to bugs in the
dependency expressions inside them.  This SHOULD be done by packages
automatically, but sometimes it is missed.  Bugs should be filed in
these cases to get the ebuilds fixed, but in the meantime setting -j1
can fix this.  (This was recently discussed on this list for that pam
dependency - which has already been fixed in the repo so you don't
actually need to worry about it now.)  It shouldn't really be
necessary to set --jobs=1 unless you're running out of RAM, because it
is pretty rare to have portage dependency issues and if you do chances
are the package will break either way (because the unspecified
dependency just won't be built at all - unlike with make where
generally everything does get built just in the wrong order).

Hopefully this helps...

-- 
Rich