[gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[gottlieb]

For a long time I have had in make.conf

EMERGE_DEFAULT_OPTS="--ask --deep --tree --verbose --jobs --load-average=5"
MAKEOPTS="--jobs --load-average=5"

(for previous processors the 5 was 3).

It seems that this configuration fails for several packages (or tickles
bugs in their ebuilds/Makefiles).

Lately whenever a build fails I change to

MAKEOPTS="--jobs=1"

and this very often "fixes" the problem.

It is not clear that any time saved by having jobs=5 compensates for
having to redo builds.  So my question is do people

1. keep jobs=1 in MAKEOPTS
2. have jobs=n in MAKEOPTS but degrade on error as I do
3. have jobs=n and file bugs when it fails.

thanks,
allan

PS I do not change EMERGE_DEFAULT_OPTS

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 928 bytes --]

On Mon, 25 Mar 2013 16:57:09 -0400, gottlieb@nyu.edu wrote:

> It is not clear that any time saved by having jobs=5 compensates for
> having to redo builds.  So my question is do people
> 
> 1. keep jobs=1 in MAKEOPTS
> 2. have jobs=n in MAKEOPTS but degrade on error as I do
> 3. have jobs=n and file bugs when it fails.
> 
> thanks,
> allan
> 
> PS I do not change EMERGE_DEFAULT_OPTS

I have "--jobs --load-average=12" in EMERGE_DEFAULT_OPTS and override
this for known problematic packages in /etc/portage/package.env.

For example:

% cat /etc/portage/package.env/libreoffice 
app-office/libreoffice j4.conf

% cat /etc/portage/env/j4.conf 
MAKEOPTS="-j4"

Running MAKEOPTS="-j1" as default on a multi-core processor seems an
awful waste of resources, unless it is needed for something else, in
which case I don't run emerge at all.


-- 
Neil Bothwick

Excuse for the day: daemons did it

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[gottlieb]

On Mon, Mar 25 2013, Neil Bothwick wrote:

> On Mon, 25 Mar 2013 16:57:09 -0400, gottlieb@nyu.edu wrote:
>
>> It is not clear that any time saved by having jobs=5 compensates for
>> having to redo builds.  So my question is do people
>> 
>> 1. keep jobs=1 in MAKEOPTS
>> 2. have jobs=n in MAKEOPTS but degrade on error as I do
>> 3. have jobs=n and file bugs when it fails.
>> 
>> thanks,
>> allan
>> 
>> PS I do not change EMERGE_DEFAULT_OPTS
>
> I have "--jobs --load-average=12" in EMERGE_DEFAULT_OPTS and override
> this for known problematic packages in /etc/portage/package.env.
>
> For example:
>
> % cat /etc/portage/package.env/libreoffice 
> app-office/libreoffice j4.conf
>
> % cat /etc/portage/env/j4.conf 
> MAKEOPTS="-j4"

I see.  Clever.

Do you file bugs when you need to restrict MAKEOPTS?

Also I am somewhat surprised you can run libreoffice
with MAKEOPTS="-j4".  I seem to remember that being one of the ones I
had to degrade.

Thanks for the tip, I will use it.

allan

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Dale]

gottlieb@nyu.edu wrote:
> Also I am somewhat surprised you can run libreoffice with
> MAKEOPTS="-j4". I seem to remember that being one of the ones I had to
> degrade. Thanks for the tip, I will use it. allan 

I update libreoffice whenever it needs it and I have this setting:

MAKEOPTS="-j16"

It hasn't failed me in a long time, except for the time I ran out of
space.  Maybe things have been fixed so it can build with that setting
where it couldn't before?  May want to try it sometime when you got time
on your hands.

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[gottlieb]

On Mon, Mar 25 2013, Dale wrote:

> gottlieb@nyu.edu wrote:
>> Also I am somewhat surprised you can run libreoffice with
>> MAKEOPTS="-j4". I seem to remember that being one of the ones I had to
>> degrade. Thanks for the tip, I will use it. allan 
>
> I update libreoffice whenever it needs it and I have this setting:
>
> MAKEOPTS="-j16"
>
> It hasn't failed me in a long time, except for the time I ran out of
> space.  Maybe things have been fixed so it can build with that setting
> where it couldn't before?  May want to try it sometime when you got time
> on your hands.
>
> Dale

Thanks.  My data may well be old.
allan

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

On Mon, 25 Mar 2013 22:29:00 -0400, gottlieb@nyu.edu wrote:

> > % cat /etc/portage/package.env/libreoffice 
> > app-office/libreoffice j4.conf
> >
> > % cat /etc/portage/env/j4.conf 
> > MAKEOPTS="-j4"  
> 
> I see.  Clever.
> 
> Do you file bugs when you need to restrict MAKEOPTS?

If I need to restrict it to -j1, it is builds with -j4 I consider the
"bug" to be self-inflicted :)

> Also I am somewhat surprised you can run libreoffice with
> MAKEOPTS="-j4".  I seem to remember that being one of the ones I had to
> degrade.

I try with -j4, if that fails, I try -j1. ISTR OpenOffice.org used to
have -j1 forced in the ebuild. Perhaps the time the LO guys spent on code
cleanup covered more than removing a ton of German comments to no longer
used functions :-O


-- 
Neil Bothwick

@@@@ There are two kinds of people in this world: Those who are
good with words, and those who are... erm... thingy @@@@

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Nilesh Govindrajan]

On Tue, Mar 26, 2013 at 2:04 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Mon, 25 Mar 2013 22:29:00 -0400, gottlieb@nyu.edu wrote:
>
>> > % cat /etc/portage/package.env/libreoffice
>> > app-office/libreoffice j4.conf
>> >
>> > % cat /etc/portage/env/j4.conf
>> > MAKEOPTS="-j4"
>>
>> I see.  Clever.
>>
>> Do you file bugs when you need to restrict MAKEOPTS?
>
> If I need to restrict it to -j1, it is builds with -j4 I consider the
> "bug" to be self-inflicted :)
>
>> Also I am somewhat surprised you can run libreoffice with
>> MAKEOPTS="-j4".  I seem to remember that being one of the ones I had to
>> degrade.
>
> I try with -j4, if that fails, I try -j1. ISTR OpenOffice.org used to
> have -j1 forced in the ebuild. Perhaps the time the LO guys spent on code
> cleanup covered more than removing a ton of German comments to no longer
> used functions :-O
>
>
> --
> Neil Bothwick
>
> @@@@ There are two kinds of people in this world: Those who are
> good with words, and those who are... erm... thingy @@@@

I'd done this experiment sometime ago, and I had sent a mail here as
well regarding which load average does make account for.
A couple of packages started failing compile and it turns out that
they don't work well with the infinite jobs that make --jobs spawns.

It's better to limit the number of jobs to 2*CPUs (or cores) with a
load control like --load-average=N where N is number of CPUs.

https://bugs.gentoo.org/show_bug.cgi?id=452942

--
Nilesh Govindrajan
http://nileshgr.com

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Tue, 26 Mar 2013 20:43:25 +0530, Nilesh Govindrajan wrote:

> I'd done this experiment sometime ago, and I had sent a mail here as
> well regarding which load average does make account for.
> A couple of packages started failing compile and it turns out that
> they don't work well with the infinite jobs that make --jobs spawns.
> 
> It's better to limit the number of jobs to 2*CPUs (or cores) with a
> load control like --load-average=N where N is number of CPUs.

That makes sense, I've altered my settings to

% grep jobs /etc/portage/make.conf
MAKEOPTS="--jobs 16 --load 10"
EMERGE_DEFAULT_OPTS="--alphabetical --jobs --load-average 12"

and two of the previously troublesome packages, libreoffice and
virtualbox, now work with no override.


-- 
Neil Bothwick

One difference between a man and a machine is that a machine is quiet
when well oiled.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[gottlieb]

On Tue, Mar 26 2013, Neil Bothwick wrote:

> On Tue, 26 Mar 2013 20:43:25 +0530, Nilesh Govindrajan wrote:
>
>> It's better to limit the number of jobs to 2*CPUs (or cores) with a
>> load control like --load-average=N where N is number of CPUs.

I have two i7-3520Ms.  Each has hyperthreading so "counts" as 2.
In particular /proc/cpuinfo describes 4 "cpus".

Nilesh, Does that mean you recommend

EMERGE_DEFAULT_OPTS="--jobs --load-average=4"
MAKEOPTS="--jobs=8 --load-average=4"

In particular I am not sure if your recommendation for load-average
applied to EMERGE_DEFAULT_OPTS as well or if you were just discussing
MAKEOPTS.

thanks,
allan

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Nilesh Govindrajan]

On Wed, Mar 27, 2013 at 7:50 PM,  <gottlieb@nyu.edu> wrote:
> On Tue, Mar 26 2013, Neil Bothwick wrote:
>
>> On Tue, 26 Mar 2013 20:43:25 +0530, Nilesh Govindrajan wrote:
>>
>>> It's better to limit the number of jobs to 2*CPUs (or cores) with a
>>> load control like --load-average=N where N is number of CPUs.
>
> I have two i7-3520Ms.  Each has hyperthreading so "counts" as 2.
> In particular /proc/cpuinfo describes 4 "cpus".
>
> Nilesh, Does that mean you recommend
>
> EMERGE_DEFAULT_OPTS="--jobs --load-average=4"
> MAKEOPTS="--jobs=8 --load-average=4"
>
> In particular I am not sure if your recommendation for load-average
> applied to EMERGE_DEFAULT_OPTS as well or if you were just discussing
> MAKEOPTS.
>
> thanks,
> allan
>

It was for MAKEOPTS. If you have a really powerful processor, consider
splitting it for emerge options and make options. Would be faster.

@Walter, I'm also on a dual core machine, and as per my observation
over long emerges, load doesn't cross 2.2.
I have also observed that if it is limited to 2, system seems to be
under utilized, because make checks the 1 minute average instead of 15
minute average (well, it doesn't make sense otherwise).

This could apply to bigger processors too, so if you want full
utilization, slightly extrapolate the load average limit. Say by
20-30%.

--
Nilesh Govindrajan
http://nileshgr.com

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Walter Dnes]

On Wed, Mar 27, 2013 at 08:07:06PM +0530, Nilesh Govindrajan wrote

> @Walter, I'm also on a dual core machine, and as per my observation
> over long emerges, load doesn't cross 2.2.
> I have also observed that if it is limited to 2, system seems to be
> under utilized, because make checks the 1 minute average instead of 15
> minute average (well, it doesn't make sense otherwise).
> 
> This could apply to bigger processors too, so if you want full
> utilization, slightly extrapolate the load average limit. Say by
> 20-30%.

  OK, I'll go with...

MAKEOPTS="-j2 --load-average=3"

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Peter Humphrey]

[-- Attachment #1: Type: text/plain, Size: 565 bytes --]

On Wednesday 27 March 2013 18:16:22 Walter Dnes wrote:

>   OK, I'll go with...
> 
> MAKEOPTS="-j2 --load-average=3"

This box is an i5 with four single-threaded CPUs and I limit the average 
load to 8. Since emerge is running at niceness=3 the desktop remains 
responsive throughout. I used not to limit the load at all and KDE was still 
fine to work with. I sometimes think that with modern systems there's no need 
to impose limits of my own since the kernel can cope well by itself.

In fact I'm going to remove the load limit and see how I get on.

-- 
Peter

[-- Attachment #2: Type: text/html, Size: 2807 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Stroller]

On 28 March 2013, at 14:03, Peter Humphrey wrote:
> ...
> This box is an i5 with four single-threaded CPUs  …

Your usage of the term "CPUs" is making me twitch.

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Peter Humphrey]

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]

On Thursday 28 March 2013 19:28:47 Stroller wrote:
> On 28 March 2013, at 14:03, Peter Humphrey wrote:
> > ...
> > This box is an i5 with four single-threaded CPUs  …
> 
> Your usage of the term "CPUs" is making me twitch.

What would you have said?

And it wasn't usage, it was use.
-- 
Peter

[-- Attachment #2: Type: text/html, Size: 2540 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Mateusz Kowalczyk]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 29/03/13 00:40, Peter Humphrey wrote:
> On Thursday 28 March 2013 19:28:47 Stroller wrote:
>> On 28 March 2013, at 14:03, Peter Humphrey wrote:
>>> ... This box is an i5 with four single-threaded CPUs  …
>> 
>> Your usage of the term "CPUs" is making me twitch.
> 
> What would you have said?
> 
> And it wasn't usage, it was use.
> 
I can only imagine he was pointing out that you have a single CPU with
four cores in it.

- -- 
Mateusz K.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iQIcBAEBAgAGBQJRVO1fAAoJEM1mucMq2pqXaS8QAKPfQ72PlpiAKdf2xKmXKOnU
3sH2108QfVWxo3Aw+aQlUwSXukkwwJY7vZWRXHsYyU58A03UmBx2rQnYD+cXeUlN
pm2HcIyp5qypX2oISK7i8Xg38Op/MCedtVU24uOZLYsQ9YUrBuHOhbKTmI5PvpAP
hmNRbUFx+pdYtLJsvR11L+kgNXC/T8ZIyw3vouktEidf7igRXVs5oCZyaCLfxDXk
XT8V1HHxbvU7Y5kUKoVAuWAHtxKqnVCyqpz/G1gLsSVrksfP4Spv+7qWe87yiBdu
fd8CBlc6A6cArz82owim9YdllKMa2T6/ohptOkLyrWhU7Q4piGD48ky55dqlYzB6
Zsa6DuiUEmIYfWTviR1pXgFsMTCpkmj2mLSZ5xLL45/I1U2+9NsbiKxnwe/8GfpU
oUY2owjkpfxoaOyWZ0YCBgXEIuWpzJllifv5B18XroC9QSWPCckhOoR9JnrfS6QW
1EVsomK5moJ/Qqe4Xtx0XZj8fUmOnelV2h/NoNRt0kdQFacT8+y6xTGCdOb+FLxf
e8qUUtOiEv5zSbHO1cHuqzmHdXcr7euZI881Vp9nFbdvbmXD3hrQybakdVzq9gt/
cRg+JvE4pGyUBtNdVy8vWW4PWf7rw0gOqF3PpuJB2siFqSZs0fYREIzRFucVFj5/
EWl9A8gv/BAZv6D6S4/L
=2eKx
-----END PGP SIGNATURE-----

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Peter Humphrey]

[-- Attachment #1: Type: text/plain, Size: 217 bytes --]

On Friday 29 March 2013 01:24:48 Mateusz Kowalczyk wrote:

> I can only imagine he was pointing out that you have a single CPU with
> four cores in it.

You're right, of course. I should have said /cores/.

-- 
Peter

[-- Attachment #2: Type: text/html, Size: 2013 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Nilesh Govindrajan]

On Fri, Mar 29, 2013 at 7:29 AM, Peter Humphrey
<peter@humphrey.ukfsn.org> wrote:
> On Friday 29 March 2013 01:24:48 Mateusz Kowalczyk wrote:
>
>
>
>> I can only imagine he was pointing out that you have a single CPU with
>
>> four cores in it.
>
>
>
> You're right, of course. I should have said /cores/.
>
>
>
> --
>
> Peter
>
>

Cores or CPUs.. in this context it's *almost*, __NOT EXACTLY__ same.

--
Nilesh Govindrajan
http://nileshgr.com

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Stroller]

On 29 March 2013, at 03:36, Nilesh Govindrajan wrote:
>> ...
>>> I can only imagine he was pointing out that you have a single CPU with four cores in it.
>> 
>> You're right, of course. I should have said /cores/.
> 
> Cores or CPUs.. in this context it's *almost*, __NOT EXACTLY__ same.

Which is exactly what was so twitch inducing! 

Stroller.

[gentoo-user] Re: Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[»Q«]

On Fri, 29 Mar 2013 16:54:37 +0000
Stroller <stroller@stellar.eclipse.co.uk> wrote:

> 
> On 29 March 2013, at 03:36, Nilesh Govindrajan wrote:
> >> ...
> >>> I can only imagine he was pointing out that you have a single CPU
> >>> with four cores in it.
> >> 
> >> You're right, of course. I should have said /cores/.
> > 
> > Cores or CPUs.. in this context it's *almost*, __NOT EXACTLY__ same.
> 
> Which is exactly what was so twitch inducing! 

Whatever you do, don't read the first sentence at
<https://en.wikipedia.org/wiki/Multi-core_processor>.

Re: [gentoo-user] Re: Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Dale]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

»Q« wrote:
> On Fri, 29 Mar 2013 16:54:37 +0000
> Stroller <stroller@stellar.eclipse.co.uk> wrote:
>
>> On 29 March 2013, at 03:36, Nilesh Govindrajan wrote:
>>>> ...
>>>>> I can only imagine he was pointing out that you have a single CPU
>>>>> with four cores in it.
>>>> You're right, of course. I should have said /cores/.
>>> Cores or CPUs.. in this context it's *almost*, __NOT EXACTLY__ same.
>> Which is exactly what was so twitch inducing! 
> Whatever you do, don't read the first sentence at
> <https://en.wikipedia.org/wiki/Multi-core_processor>.
>
>
>

Especially this FIRST part:

"A *multi-core processor* is a single computing
<https://en.wikipedia.org/wiki/Computing> component . . ."

So, it is a SINGLE component.  To me, CPUs means having more than one
CPU component, such as dual CPUs or even quad CPUs which used to be
fairly common. 

I have a single CPU computer.  It has 4 cores but a single CPU.  I hope
to upgrade one day to a 8 core CPU.  I'll still have a single CPU
component installed tho. 

This is getting really funny.  ROFL  You can tell when the list is
getting slow when we start parsing each word and each words meaning.  ;-) 

Dale 

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!


[-- Attachment #2: Type: text/html, Size: 2547 bytes --]

Re: [gentoo-user] Re: Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Michael Mol]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

On 03/29/2013 01:46 PM, Dale wrote:
> »Q« wrote:
>> On Fri, 29 Mar 2013 16:54:37 +0000
>> Stroller <stroller@stellar.eclipse.co.uk> wrote:
>>
>>> On 29 March 2013, at 03:36, Nilesh Govindrajan wrote:
>>>>> ...
>>>>>> I can only imagine he was pointing out that you have a single CPU
>>>>>> with four cores in it.
>>>>> You're right, of course. I should have said /cores/.
>>>> Cores or CPUs.. in this context it's *almost*, __NOT EXACTLY__ same.
>>> Which is exactly what was so twitch inducing! 
>> Whatever you do, don't read the first sentence at
>> <https://en.wikipedia.org/wiki/Multi-core_processor>.
>>
>>
>>
> 
> Especially this FIRST part:
> 
> "A *multi-core processor* is a single computing
> <https://en.wikipedia.org/wiki/Computing> component . . ."
> 
> So, it is a SINGLE component.  To me, CPUs means having more than one
> CPU component, such as dual CPUs or even quad CPUs which used to be
> fairly common. 
> 
> I have a single CPU computer.  It has 4 cores but a single CPU.  I hope
> to upgrade one day to a 8 core CPU.  I'll still have a single CPU
> component installed tho. 
> 
> This is getting really funny.  ROFL  You can tell when the list is
> getting slow when we start parsing each word and each words meaning.  ;-) 

The list hasn't been slow all week. ^^


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

Re: [gentoo-user] Re: Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Nilesh Govindrajan]

[-- Attachment #1: Type: text/plain, Size: 1749 bytes --]

Another interesting point about this load control thing is that if the
package uses a build system which doesn't support load control, load will
surge high.

It is currently happening with me while installing Mongo, because the build
system scons doesn't have load control feature.
On Mar 29, 2013 11:36 PM, "Michael Mol" <mikemol@gmail.com> wrote:

> On 03/29/2013 01:46 PM, Dale wrote:
> > »Q« wrote:
> >> On Fri, 29 Mar 2013 16:54:37 +0000
> >> Stroller <stroller@stellar.eclipse.co.uk> wrote:
> >>
> >>> On 29 March 2013, at 03:36, Nilesh Govindrajan wrote:
> >>>>> ...
> >>>>>> I can only imagine he was pointing out that you have a single CPU
> >>>>>> with four cores in it.
> >>>>> You're right, of course. I should have said /cores/.
> >>>> Cores or CPUs.. in this context it's *almost*, __NOT EXACTLY__ same.
> >>> Which is exactly what was so twitch inducing!
> >> Whatever you do, don't read the first sentence at
> >> <https://en.wikipedia.org/wiki/Multi-core_processor>.
> >>
> >>
> >>
> >
> > Especially this FIRST part:
> >
> > "A *multi-core processor* is a single computing
> > <https://en.wikipedia.org/wiki/Computing> component . . ."
> >
> > So, it is a SINGLE component.  To me, CPUs means having more than one
> > CPU component, such as dual CPUs or even quad CPUs which used to be
> > fairly common.
> >
> > I have a single CPU computer.  It has 4 cores but a single CPU.  I hope
> > to upgrade one day to a 8 core CPU.  I'll still have a single CPU
> > component installed tho.
> >
> > This is getting really funny.  ROFL  You can tell when the list is
> > getting slow when we start parsing each word and each words meaning.  ;-)
>
> The list hasn't been slow all week. ^^
>
>

[-- Attachment #2: Type: text/html, Size: 2536 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1151 bytes --]

On Thursday 28 Mar 2013 14:03:27 Peter Humphrey wrote:
> On Wednesday 27 March 2013 18:16:22 Walter Dnes wrote:
> >   OK, I'll go with...
> > 
> > MAKEOPTS="-j2 --load-average=3"
> 
> This box is an i5 with four single-threaded CPUs and I limit the average
> load to 8. Since emerge is running at niceness=3 the desktop remains
> responsive throughout. I used not to limit the load at all and KDE was
> still fine to work with. I sometimes think that with modern systems
> there's no need to impose limits of my own since the kernel can cope well
> by itself.
> 
> In fact I'm going to remove the load limit and see how I get on.

I've got a first generation i7 and this is what I have set up in my make.conf:

  MAKEOPTS="-j5 -l12.8"
  EMERGE_DEFAULT_OPTS="--quiet-build=n"

Why is -l set at 12.8 ... ?  At some distant point in the past this made sense 
to me, but I have no idea how I arrived at it.  Other than the cooling fan 
speeding up I have not noticed a problem with any ebuilds.  Very rarely I 
might have used -j1 to complete a failing ebuild, but it was so long ago I 
can't even recall it.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

On Fri, 29 Mar 2013 12:36:56 +0000, Mick wrote:

> I've got a first generation i7 and this is what I have set up in my
> make.conf:
> 
>   MAKEOPTS="-j5 -l12.8"
>   EMERGE_DEFAULT_OPTS="--quiet-build=n"

n is the default for quiet-build if --jobs is set to 1, or unspecified.
But using a higher value will give you faster updates. The MAKEOPTS
setting has no effect during the preparation and installation stages of
an ebuild, and with --jobs=1 that means your CPU spends a lot of time
idling.


-- 
Neil Bothwick

This is as bad as it can get - but don't bet on it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Walter Dnes]

On Mon, Mar 25, 2013 at 10:25:58PM +0000, Neil Bothwick wrote

> Running MAKEOPTS="-j1" as default on a multi-core processor seems an
> awful waste of resources, unless it is needed for something else, in
> which case I don't run emerge at all.

  Running around for a few hours trying to replicate an unreplicatable
build failure is even more of a waste.  After the first couple of builds
that ran into problems, and were cured by -j1, I made it the default on
my mcahines.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

On Mon, 25 Mar 2013 23:01:30 -0400, Walter Dnes wrote:

> > Running MAKEOPTS="-j1" as default on a multi-core processor seems an
> > awful waste of resources, unless it is needed for something else, in
> > which case I don't run emerge at all.  
> 
>   Running around for a few hours trying to replicate an unreplicatable
> build failure is even more of a waste.

That depends on the number of cores you have. On a dual core system this
may well be true, although I'd dispute even that.

% cat /etc/portage/package.env/* | grep -c -e j4 -e j1
4

Four packages out of the 1730 installed on this computer require special
treatment, and all of them build with -j4. 


--  
Neil Bothwick

Isn't it a bit unnerving that doctors call what they do "practice?"

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Walter Dnes]

On Tue, Mar 26, 2013 at 08:34:29AM +0000, Neil Bothwick wrote
> On Mon, 25 Mar 2013 23:01:30 -0400, Walter Dnes wrote:
> 
> > > Running MAKEOPTS="-j1" as default on a multi-core processor seems an
> > > awful waste of resources, unless it is needed for something else, in
> > > which case I don't run emerge at all.  
> > 
> >   Running around for a few hours trying to replicate an unreplicatable
> > build failure is even more of a waste.
> 
> That depends on the number of cores you have. On a dual core system this
> may well be true, although I'd dispute even that.
> 
> % cat /etc/portage/package.env/* | grep -c -e j4 -e j1
> 4
> 
> Four packages out of the 1730 installed on this computer require special
> treatment, and all of them build with -j4. 

  Seeing as how even my older machines are dual core, I'll bump it -j2.
I'm still not totally clear on "load-average".  Should I set...

MAKEOPTS="-j2 --load-average=2"

...on a dual core machine?

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Michael Hampicke]

Am 25.03.2013 21:57, schrieb gottlieb@nyu.edu:
> For a long time I have had in make.conf
> 
> EMERGE_DEFAULT_OPTS="--ask --deep --tree --verbose --jobs --load-average=5"
> MAKEOPTS="--jobs --load-average=5"
> 
> (for previous processors the 5 was 3).
> 
> It seems that this configuration fails for several packages (or tickles
> bugs in their ebuilds/Makefiles).
> 
> Lately whenever a build fails I change to
> 
> MAKEOPTS="--jobs=1"
> 
> and this very often "fixes" the problem.
> 
> It is not clear that any time saved by having jobs=5 compensates for
> having to redo builds.  So my question is do people
> 
> 1. keep jobs=1 in MAKEOPTS
> 2. have jobs=n in MAKEOPTS but degrade on error as I do
> 3. have jobs=n and file bugs when it fails.
> 
> thanks,
> allan
> 
> PS I do not change EMERGE_DEFAULT_OPTS
> 

This is what I use:
EMERGE_DEFAULT_OPTS="--jobs=2 --load-average=6"

I havent't had any failed builds that were related to the --jobs option.
The only exception is when rebuilding my kernel modules. I have to build
spl first, then zfs-kmod. But that's  because zfs-kmod requires a
complete built spl.

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

On Mon, 25 Mar 2013 23:27:04 +0100, Michael Hampicke wrote:

> This is what I use:
> EMERGE_DEFAULT_OPTS="--jobs=2 --load-average=6"
> 
> I havent't had any failed builds that were related to the --jobs option.
> The only exception is when rebuilding my kernel modules. I have to build
> spl first, then zfs-kmod. But that's  because zfs-kmod requires a
> complete built spl.

Interesting, I have --jobs and portage doesn't try t build spl
and zfs-kmod in parallel, it always completes spl first.


-- 
Neil Bothwick

If the cops arrest a mime, do they tell her she has the right to remain
silent?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Michael Hampicke]

Am 25.03.2013 23:32, schrieb Neil Bothwick:
> On Mon, 25 Mar 2013 23:27:04 +0100, Michael Hampicke wrote:
> 
>> This is what I use:
>> EMERGE_DEFAULT_OPTS="--jobs=2 --load-average=6"
>>
>> I havent't had any failed builds that were related to the --jobs option.
>> The only exception is when rebuilding my kernel modules. I have to build
>> spl first, then zfs-kmod. But that's  because zfs-kmod requires a
>> complete built spl.
> 
> Interesting, I have --jobs and portage doesn't try t build spl
> and zfs-kmod in parallel, it always completes spl first.
> 

Interesting. Maybe a problem with module-rebuild? I will look into this
when I have some spare time.

Freshly merged kernel sources, kernel configured, compiled and
installed. Then I ran module-rebuild:


# module-rebuild rebuild
** Preparing to merge modules:
** Packages which I will emerge are:
	=net-misc/r8168-8.035.00-r1
	=app-emulation/virtualbox-modules-4.2.10
	=x11-drivers/ati-drivers-13.3_beta2
	=sys-kernel/spl-0.6.0_rc14-r3
	=sys-fs/zfs-kmod-0.6.0_rc14-r5
5 4 3 2 1
>>> Verifying ebuild manifests
>>> Running pre-merge checks for x11-drivers/ati-drivers-13.3_beta2
 * Determining the location of the kernel source code
 * Found kernel source directory:
 *     /usr/src/linux
 * Found kernel object directory:
 *     /lib/modules/3.8.4-gentoo/build
 * Found sources for kernel version:
 *     3.8.4-gentoo
 * Checking for suitable kernel configuration options...
                                                                    [ ok ]
>>> Emerging (1 of 5) net-misc/r8168-8.035.00-r1
>>> Emerging (2 of 5) app-emulation/virtualbox-modules-4.2.10
>>> Installing (1 of 5) net-misc/r8168-8.035.00-r1
>>> Emerging (3 of 5) x11-drivers/ati-drivers-13.3_beta2
>>> Emerging (4 of 5) sys-kernel/spl-0.6.0_rc14-r3
>>> Installing (2 of 5) app-emulation/virtualbox-modules-4.2.10
>>> Installing (3 of 5) x11-drivers/ati-drivers-13.3_beta2
>>> Emerging (5 of 5) sys-fs/zfs-kmod-0.6.0_rc14-r5
>>> Failed to emerge sys-fs/zfs-kmod-0.6.0_rc14-r5, Log file:
>>>  '/var/tmp/portage/sys-fs/zfs-kmod-0.6.0_rc14-r5/temp/build.log'
>>> Installing (4 of 5) sys-kernel/spl-0.6.0_rc14-r3
>>> Jobs: 4 of 5 complete, 1 failed                 Load avg: 1.53,
0.90, 1.05


And the error:

checking kernel file name for module symbols... Module.symvers
checking spl source directory... Not found
configure: error:
	*** Please make sure the spl devel package for your distribution
	*** is installed then try again.  If that fails you can specify the
	*** location of the spl source with the '--with-spl=PATH' option.

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

On Tue, 26 Mar 2013 21:58:29 +0100, Michael Hampicke wrote:

> >> I havent't had any failed builds that were related to the --jobs
> >> option. The only exception is when rebuilding my kernel modules. I
> >> have to build spl first, then zfs-kmod. But that's  because zfs-kmod
> >> requires a complete built spl.  
> > 
> > Interesting, I have --jobs and portage doesn't try t build spl
> > and zfs-kmod in parallel, it always completes spl first.
> >   
> 
> Interesting. Maybe a problem with module-rebuild? I will look into this
> when I have some spare time.
> 
> Freshly merged kernel sources, kernel configured, compiled and
> installed. Then I ran module-rebuild:
> 
> 
> # module-rebuild rebuild

The only difference I can see is that I use

emerge @module-rebuild


-- 
Neil Bothwick

Stop tagline theft! Copyright your tagline (c)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Michael Hampicke]

Am 26.03.2013 22:40, schrieb Neil Bothwick:
> On Tue, 26 Mar 2013 21:58:29 +0100, Michael Hampicke wrote:
> 
>>>> I havent't had any failed builds that were related to the --jobs
>>>> option. The only exception is when rebuilding my kernel modules. I
>>>> have to build spl first, then zfs-kmod. But that's  because zfs-kmod
>>>> requires a complete built spl.  
>>>
>>> Interesting, I have --jobs and portage doesn't try t build spl
>>> and zfs-kmod in parallel, it always completes spl first.
>>>   
>>
>> Interesting. Maybe a problem with module-rebuild? I will look into this
>> when I have some spare time.
>>
>> Freshly merged kernel sources, kernel configured, compiled and
>> installed. Then I ran module-rebuild:
>>
>>
>> # module-rebuild rebuild
> 
> The only difference I can see is that I use
> 
> emerge @module-rebuild
> 

emerge @module-rebuild seems to work just fine on my machine. Thx for
the tip, I did not know of this @set

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

On Wed, 27 Mar 2013 19:54:39 +0100, Michael Hampicke wrote:

> emerge @module-rebuild seems to work just fine on my machine. Thx for
> the tip, I did not know of this @set

@x11-module-rebuild is worth knowing about too :)


-- 
Neil Bothwick

Fine day for a good workout. Steal something heavy.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Is 'MAKEOPTS="--jobs --load-average=5"' silly?

[gottlieb]

On Mon, Mar 25 2013, Michael Hampicke wrote:

> Am 25.03.2013 21:57, schrieb gottlieb@nyu.edu:
>> For a long time I have had in make.conf
>> 
>> EMERGE_DEFAULT_OPTS="--ask --deep --tree --verbose --jobs --load-average=5"
>> MAKEOPTS="--jobs --load-average=5"
>> 
>> (for previous processors the 5 was 3).
>> 
>> It seems that this configuration fails for several packages (or tickles
>> bugs in their ebuilds/Makefiles).
>> 
>> Lately whenever a build fails I change to
>> 
>> MAKEOPTS="--jobs=1"
>> 
>> and this very often "fixes" the problem.
>> 
>> It is not clear that any time saved by having jobs=5 compensates for
>> having to redo builds.  So my question is do people
>> 
>> 1. keep jobs=1 in MAKEOPTS
>> 2. have jobs=n in MAKEOPTS but degrade on error as I do
>> 3. have jobs=n and file bugs when it fails.
>> 
>> thanks,
>> allan
>> 
>> PS I do not change EMERGE_DEFAULT_OPTS
>> 
>
> This is what I use:
> EMERGE_DEFAULT_OPTS="--jobs=2 --load-average=6"
>
> I havent't had any failed builds that were related to the --jobs option.
> The only exception is when rebuilding my kernel modules. I have to build
> spl first, then zfs-kmod. But that's  because zfs-kmod requires a
> complete built spl.

I don't change EMERGE_DEFAULT_OPTS either.  What I change is
MAKEOPTS.

That is, I always permit portage to run two emerges in parallel.
I am considering telling make to not run different parts of one emerge
in parallel.

allan