[gentoo-dev] Be Careful of Your MAKEOPTS!!

[gentoo-dev] Be Careful of Your MAKEOPTS!!

[Steven Elling]

If your using distcc, have multiple processors, etc. be very careful
with using the '-j' option without a number in the MAKEOPTS variable.

I recently did an update of glibc to glibc-2.3.2-r3 and after a while of
compiling my system was freezing.  I would start the emerge process and
walk away because I knew it would take a while.  When I came back to the
system later on, the drive light was lit solid and would not shut off. 
The system would also not respond to user input even though I could
switch from VT to VT.  Everything I saw happening pointed to a drive
failing only part of the time.

It took me a while to figure out what was going on but I started the
emerge process on one VT and sat there a watched it.  On another VT, I
used watch to periodically display the disk usage, as well as, the
output of vmstat.  Pretty soon I found out what was going on.

The system started to slow to a crawl and I happened to look at the
process table.  There were upward of 870 process running on my box
during the compile.  Ouch!

Essentially, I was slowly fork bombing my box because I was using the
'-j' option without a number in MAKEOPTS.  The man page for make states,
"If the -j option is given without an argument, make will not limit the
number of jobs that can run simultaneously."  With most packages this is
not a problem but, in the case of glibc, the source directory has
several components to compile and there were enough components that it
was bringing my system down.

Therefore, if you find your system freezing during a emerge like mine
did, check your MAKEOPTS.


Steven Elling


P.S. Developers: You might want to look into adding a number to the '-j'
option in MAKEOPTS during emerges of certain packages to account for
this situation.


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Mike Frysinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 04 December 2003 20:26, Steven Elling wrote:
> P.S. Developers: You might want to look into adding a number to the '-j'
> option in MAKEOPTS during emerges of certain packages to account for
> this situation.

i doubt that'll happen
after all, some machines have 64 cpu's in them ... i think they might be able 
to handle the make fork bombing of glibc
- -mike
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iQIVAwUBP8/hFEFjO5/oN/WBAQKnJhAA3+OowTpxVPFbyVbzNS9wnaJ2siydrhkF
PHMsvzcldtoICm9l2xwmqxhz9JOVxsjGQ22oOyqblWYru7Phpx7k+8J8ib/Umne2
XZmmVP4cuI60CnolqrYhhD77ghGr52NvHMLy3EiVn9x3/vDudZ4X5tcnvT1pWk1D
ZePaorP6HotmQDJAk16snlw7Vy1V+9MfRISWXIbX2XSAdZoCHAcAzJwNIQDIpPXe
vcibWfUK/Kozii+9IV6YCXXZjHzqBoh+y07nUJrCsMjBimYPJsg5UIfo+MgpGTwe
pRfcfI7yhn+zhyr3pkZIH3Sq+FgjztfNIBeLWKMjnJmHFsskFu2HdVeApeQaijJ9
bv7CIl7dJ2F86ekThMZDV+nNogDn+XN3JnP6Fa0R4oQxFSlNV7cen1uMoCuybP7z
4BpYzL3cINgKCBsB2Qp1cS8UQXkWLWxRJOjx2nqT+2misrxMFyZ3Uw+EwA+MeJbZ
7MDIMHb6dlNAt5y1i45iA9lBrVesMZpjA2mg1rqv/8ZhhXjraGZlgZ6Xde2NKww4
6dag2XdpFRpAqrmnZyc+UHHdsX+NI2ve6u82G46CcImcHG/SX4IcXXLrndt/7ov2
7TWsYL+PcRIM/o1Gfbw5RWWM76JUyxmDYjWIYmkr9nzRSZUq40mzZWewd3idvXJa
ESTVGVKQW2c=
=v1Od
-----END PGP SIGNATURE-----



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 411 bytes --]

On Thu, 04 Dec 2003 19:26:51 -0600 Steven Elling <ellings@kcnet.com>
wrote:
| P.S. Developers: You might want to look into adding a number to the
| '-j' option in MAKEOPTS during emerges of certain packages to account
| for this situation.

Wouldn't that be rather unfair upon users who have a 100 CPU distcc
cluster?

-- 
Ciaran McCreesh
Mail:    ciaranm at gentoo.org
Web:     http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Spider]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

begin  quote
On Thu, 4 Dec 2003 20:36:12 -0500
Mike Frysinger <vapier@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Thursday 04 December 2003 20:26, Steven Elling wrote:
> > P.S. Developers: You might want to look into adding a number to the
> > '-j'
> > option in MAKEOPTS during emerges of certain packages to account for
> > this situation.
> 
> i doubt that'll happen
> after all, some machines have 64 cpu's in them ... i think they might
> be able  to handle the make fork bombing of glibc


quite true, and its just extra overhead to protect somone who should
know better. And, if you have more RAM it will work ;) (slowly, but
work)

//Spider

-- 
begin  .signature
This is a .signature virus! Please copy me into your .signature!
See Microsoft KB Article Q265230 for more information.
end

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Lisa Seelye]

[-- Attachment #1: Type: text/plain, Size: 416 bytes --]

On Thu, 2003-12-04 at 20:26, Steven Elling wrote:
> P.S. Developers: You might want to look into adding a number to the '-j'
> option in MAKEOPTS during emerges of certain packages to account for
> this situation.

Modifying MAKEOPTS is only indicated for ebuilds that fail to build with
parallel makes (and/or distcc).

-- 
Regards,
-Lisa
<Vix ulla tam iniqua pax, quin bello vel aequissimo sit potior>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Sven Blumenstein]

On Thu, Dec 04, 2003 at 07:26:51PM -0600, Steven Elling wrote:
> 
> The system started to slow to a crawl and I happened to look at the
> process table.  There were upward of 870 process running on my box
> during the compile.  Ouch!
> 

Hehe, I compiled glibc-2.3.2-r9 on a quad Netra 1405 a few days ago:

http://dev.gentoo.org/~bazik/misc/200312012332.png

Not 870 processes but still good ;)
And it didnt freeze for me... maybe you just need more CPU's :P

regards,
-- 
Sven Blumenstein                http://0x1337.net/
Gentoo Linux/Sparc Developer    http://dev.gentoo.org/~bazik/


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Bob Miller]

Ciaran McCreesh wrote:

> On Thu, 04 Dec 2003 19:26:51 -0600 Steven Elling <ellings@kcnet.com>
> wrote:
> | P.S. Developers: You might want to look into adding a number to the
> | '-j' option in MAKEOPTS during emerges of certain packages to account
> | for this situation.
> 
> Wouldn't that be rather unfair upon users who have a 100 CPU distcc
> cluster?

My testing showed that distcc doesn't seem to scale beyond about 8-10
hosts.  (Hosts, not CPUs.)  Since the originator has to do all the cpp
processing and all the linking, the other hosts start standing around
idle when the originator reaches 100% CPU.

Am I wrong?  Am I doing something wrong?  I can get more CPUs... (-:

-- 
Bob Miller                              K<bob>
kbobsoft software consulting
http://kbobsoft.com                     kbob@jogger-egg.com

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Luke-Jr]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Friday 05 December 2003 05:43 pm, Bob Miller wrote:
> My testing showed that distcc doesn't seem to scale beyond about 8-10
> hosts.  (Hosts, not CPUs.)  Since the originator has to do all the cpp
> processing and all the linking, the other hosts start standing around
> idle when the originator reaches 100% CPU.
When you have 100 systems, you can probably have the main one do nothing but 
processing to send it out to the others... Or is that what you did?
- -- 
Luke-Jr
Developer, Gentoo Linux
http://www.gentoo.org/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/0MatZl/BHdU+lYMRAnu8AJ0QTYR3FNlyNHC/ENCtCJgcaLmYzgCfaBAk
payLdnIW0kaEiv5oFWLBbHA=
=Q0fJ
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Bob Miller]

Luke-Jr wrote:

> When you have 100 systems, you can probably have the main one do nothing but 
> processing to send it out to the others... Or is that what you did?

Yes, that's what I did.

Let me describe it exactly.

I have two fast boxes.  Each is a 2.4 GHz Pentium 4 w/ hyperthreading,
2 GB RAM.  I have one slow box.  It's a Celeron 466 MHz w/ 512 MB RAM.
All run Gentoo, of course.  All connected via 100 Mbit ethernet.  I
built on the Celeron, with DISTCC_HOSTS set to "fast1/5 fast2/5".  I
watched performance on all three boxes using gkrellm2.  gkrellm showed
the Celeron 100% CPU bound for most of the build, and showed that the
fast boxes had very "jaggy" CPU utilization that averaged around
30-40% during the compilation phase.  (By "jaggy" I mean that one
second would be 90% CPU, the next is 15%, the next is 100%, etc.  It
appears that when the fast box gets a job, it finishes it in a second
or so and goes back to waiting.)

I guesstimated that the two fast boxes might have roughly 8-10X the
performance of the slow box.  I could be off by a bit.  I don't have a
large fleet of identical boxes to try this on.

-- 
Bob Miller                              K<bob>
kbobsoft software consulting
http://kbobsoft.com                     kbob@jogger-egg.com

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1398 bytes --]

On Fri, 5 Dec 2003 09:43:13 -0800 Bob Miller <kbob@jogger-egg.com>
wrote:
| > On Thu, 04 Dec 2003 19:26:51 -0600 Steven Elling <ellings@kcnet.com>
| > wrote:
| > | P.S. Developers: You might want to look into adding a number to
| > | the'-j' option in MAKEOPTS during emerges of certain packages to
| > | account for this situation.
| > 
| > Wouldn't that be rather unfair upon users who have a 100 CPU distcc
| > cluster?
| 
| My testing showed that distcc doesn't seem to scale beyond about 8-10
| hosts.  (Hosts, not CPUs.)  Since the originator has to do all the cpp
| processing and all the linking, the other hosts start standing around
| idle when the originator reaches 100% CPU.

It depends a lot upon what the central host is. If the central host is
faster and has several CPUs (for example, a server using workgroup
machines as slaves) then distcc can potentially scale very well. If the
central host is a really old sparcstation5 then adding more than about
two slaves doesn't help.

It also depends upon the makefile in question. Very few Makefiles allow
more than maybe 20 things to be compiled simultaneously anyway.

So maybe I should rephrase...

Wouldn't that be rather unfair upon users who have a dozen 8-way amd64
boxes fully interconnected by switched gigE who do glibc development?

-- 
Ciaran McCreesh
Mail:    ciaranm at gentoo.org
Web:     http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 535 bytes --]

On Friday 05 December 2003 19:38, Ciaran McCreesh wrote:
>
> Wouldn't that be rather unfair upon users who have a dozen 8-way amd64
> boxes fully interconnected by switched gigE who do glibc development?

I believe that MAKEOPTS is a part of make.conf. I think that in this case it 
makes sense to have that few users with very powerful set-ups change their 
make.conf instead of having many people bork out on glibc.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 719 bytes --]

On Fri, 5 Dec 2003 21:45:10 +0100 Paul de Vrieze <pauldv@gentoo.org>
wrote:
| On Friday 05 December 2003 19:38, Ciaran McCreesh wrote:
| >
| > Wouldn't that be rather unfair upon users who have a dozen 8-way
| > amd64 boxes fully interconnected by switched gigE who do glibc
| > development?
| 
| I believe that MAKEOPTS is a part of make.conf. I think that in this
| case it makes sense to have that few users with very powerful set-ups
| change their make.conf instead of having many people bork out on
| glibc.

The default make.conf settings are sane. It's only if you change them to
something stupid that problems occur.

-- 
Ciaran McCreesh
Mail:    ciaranm at gentoo.org
Web:     http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Luke-Jr]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Friday 05 December 2003 09:02 pm, Ciaran McCreesh wrote:
> The default make.conf settings are sane. It's only if you change them to
> something stupid that problems occur.
Of course, you could set CFLAGS to something stupid too and problems would 
occur...
- -- 
Luke-Jr
Developer, Gentoo Linux
http://www.gentoo.org/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/0Q6AZl/BHdU+lYMRAj6jAJoCHooD6k3waHQ3b7SdacfMCrIpeACgk8NG
tATMovze8MkJ5uAZXxYgolY=
=geRa
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Steven Elling]

On Thu, 2003-12-04 at 19:37, Ciaran McCreesh wrote:
> On Thu, 04 Dec 2003 19:26:51 -0600 Steven Elling <ellings@kcnet.com>
> wrote:
> | P.S. Developers: You might want to look into adding a number to the
> | '-j' option in MAKEOPTS during emerges of certain packages to account
> | for this situation.
> 
> Wouldn't that be rather unfair upon users who have a 100 CPU distcc
> cluster?

Not if you make the routine smart.


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Be Careful of Your MAKEOPTS!!

[Steven Elling]

On Thu, 2003-12-04 at 19:54, Spider wrote:
> begin  quote
> On Thu, 4 Dec 2003 20:36:12 -0500
> Mike Frysinger <vapier@gentoo.org> wrote:
> 
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> > 
> > On Thursday 04 December 2003 20:26, Steven Elling wrote:
> > > P.S. Developers: You might want to look into adding a number to the
> > > '-j'
> > > option in MAKEOPTS during emerges of certain packages to account for
> > > this situation.
> > 
> > i doubt that'll happen
> > after all, some machines have 64 cpu's in them ... i think they might
> > be able  to handle the make fork bombing of glibc
> 
> 
> quite true, and its just extra overhead to protect somone who should
> know better. And, if you have more RAM it will work ;) (slowly, but
> work)
> 
> //Spider

I did not mention it but I was using ccache and distcc with 3 machines
(problem box included).

I used to have 512MB of memory in this system but I had to rob 256MB for
my server because one stick fried in it.  I would buy more memory but my
current job situation will not allow it.

Oh, and I know better.  It's just one of those problems I don't deal
with that often and, when things go awry, I don't recognize it off the
bat.  However, I do know what to look at when problems occur.

The only reason it took me a while to figure out what was going wrong is
I didn't really care to look into the problem and left it for later.


--
gentoo-dev@gentoo.org mailing list