From mboxrd@z Thu Jan 1 00:00:00 1970
Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org)
by finch.gentoo.org with esmtp (Exim 4.60)
(envelope-from <gentoo-user+bounces-131670-garchives=archives.gentoo.org@lists.gentoo.org>)
id 1RUtmN-0008Ap-Ah
for garchives@archives.gentoo.org; Mon, 28 Nov 2011 05:29:23 +0000
Received: from pigeon.gentoo.org (localhost [127.0.0.1])
by pigeon.gentoo.org (Postfix) with SMTP id 632C321C144;
Mon, 28 Nov 2011 05:29:00 +0000 (UTC)
Received: from mail-bw0-f53.google.com (mail-bw0-f53.google.com [209.85.214.53])
by pigeon.gentoo.org (Postfix) with ESMTP id 3E5FD21C105
for <gentoo-user@lists.gentoo.org>; Mon, 28 Nov 2011 05:27:28 +0000 (UTC)
Received: by bkaq10 with SMTP id q10so9472571bka.40
for <gentoo-user@lists.gentoo.org>; Sun, 27 Nov 2011 21:27:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=gamma;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:content-type;
bh=PcmsSSJ8+jHawrOg73hJObJHxeNI4su98umZLPf/ccI=;
b=rWWejBBJdV9dE4+6WYqav3qQVuejedBoJIVVUaAEXZ21x581k6Grg8CSpGhXzvb2ZQ
3fyLkmSR11GzclnXYxvHkrYqElJf/oLj1RoimBzIhwdLdvbB4IqdNGedxc3eXytkAV+S
Iii/mEsDQLGZnwKLiz9eHTIssWSiYUHtUZ9Mo=
Precedence: bulk
List-Post: <mailto:gentoo-user@lists.gentoo.org>
List-Help: <mailto:gentoo-user+help@lists.gentoo.org>
List-Unsubscribe: <mailto:gentoo-user+unsubscribe@lists.gentoo.org>
List-Subscribe: <mailto:gentoo-user+subscribe@lists.gentoo.org>
List-Id: Gentoo Linux mail <gentoo-user.gentoo.org>
X-BeenThere: gentoo-user@lists.gentoo.org
Reply-to: gentoo-user@lists.gentoo.org
MIME-Version: 1.0
Received: by 10.204.128.208 with SMTP id l16mr24600816bks.14.1322458048355;
Sun, 27 Nov 2011 21:27:28 -0800 (PST)
Received: by 10.204.14.7 with HTTP; Sun, 27 Nov 2011 21:27:28 -0800 (PST)
In-Reply-To: <CAA2qdGVy5=Va6vz54GVi-BatH2wQ9b5sDoZt+s35U77Xi2B14g@mail.gmail.com>
References: <CA+czFiB4pSTcVTgAbRnL6AUgGoE9E7gsg-1O0LgfU4FVF_DB+Q@mail.gmail.com>
<201111270927.57294.michaelkintzios@gmail.com>
<CA+czFiBAR2_j1vETXGZk6SL9XK9w=doe4orU+bLeZbXHXBUqPg@mail.gmail.com>
<CAA2qdGWwuHUUd7ZkbBOwJxx8Ax88xfAptASgvbFtNa0YMqe6bQ@mail.gmail.com>
<20111127232108.08cbacfd@digimed.co.uk>
<CAA2qdGU-qnp7Agv3Ha5L=YLotHD2MxfzBoXPPDTLYuxaQgBU+A@mail.gmail.com>
<CA+czFiDNeBusc4izgsvdBFXC6e+JQ-Ax6K+2xdhrMiJ20A34jg@mail.gmail.com>
<CAA2qdGVy5=Va6vz54GVi-BatH2wQ9b5sDoZt+s35U77Xi2B14g@mail.gmail.com>
Date: Mon, 28 Nov 2011 00:27:28 -0500
Message-ID: <CA+czFiDnpbvAgcxpNg34DVCjxFTat9Ow+7CUOp4Ztxb9CpvQ+Q@mail.gmail.com>
Subject: Re: [gentoo-user] emerge -j, make -j and make -l
From: Michael Mol <mikemol@gmail.com>
To: gentoo-user@lists.gentoo.org
Content-Type: text/plain; charset=UTF-8
X-Archives-Salt: b12192c4-4414-49e2-95aa-0ec96f55d66a
X-Archives-Hash: 7e5c95b8876e7e2819520faced10ad21
On Mon, Nov 28, 2011 at 12:13 AM, Pandu Poluan <pandu@poluan.info> wrote:
> On Nov 28, 2011 11:38 AM, "Michael Mol" <mikemol@gmail.com> wrote:
>> On Sun, Nov 27, 2011 at 7:54 PM, Pandu Poluan <pandu@poluan.info> wrote:
>> > On Nov 28, 2011 6:24 AM, "Neil Bothwick" <neil@digimed.co.uk> wrote:
>> >> On Mon, 28 Nov 2011 00:56:17 +0700, Pandu Poluan wrote:
>> >>
>> >> > I don't know where the 'blame' lies, but I've found myself
>> >> > standardizing on MAKEOPTS=-j3, and PORTAGE_DEFAULT_OPTS="--jobs
>> >> > --load-average=<1.6*num_of_vCPU>"
>> >> >
>> >> > (Yes, no explicit number of jobs. The newer portages are smart enough
>> >> > to
>> >> > keep starting new jobs until the load number is reached)
>> >>
>> >> The problem I found with that is the ebuilds load the system lightly to
>> >> start with, before they enter the compile phase, to portage starts
>> >> dozens
>> >> of parallel ebuilds, then the system gets completely bogged down when
>> >> they start compiling.
>> >>
>> >
>> > Yes, sometimes that would happen if at the beginning there are
>> > network-bound
>> > ebuilds all downloading their respective distfiles. The load stays low
>> > until
>> > they all start ./configure-ing roughly at the same time. Then all hell
>> > breaks loose.
>> >
>> > I successfully mitigate such "load-explosion" by doing a --fetchonly
>> > step
>> > first, and keeping MAKEOPTS at low -j (which, in my case, is actually
>> > required).
>> >
>> > Just to add more info: I use USE=graphite (with some CFLAGS, uh,
>> > 'enhancements') with gcc-4.5.3. IIRC, I could push MAKEOPTS up to -j5
>> > (and
>> > even more, but I ran out of cores) when I was still using gcc-4.4.x and
>> > no
>> > USE=graphite.
>> >
>> > Won't file a bug report, though. I have a feeling that my bug report re:
>> > emerge failure will be marked WONTFIX thanks to the 'ricer special'
>> > CFLAGS
>>
>> As I noted, "-l" in MAKEOPTS takes care of the load explosion very nicely.
>
> Most likely so. I am not aware of -l in MAKEOPTS before, so what I posted
> was my workaround to prevent load explosion. Thanks to your very useful tip,
> I now no longer have to worry about load explosion :-)
>
> (I still like doing pre-fetchonly-ing, though. But now for a different main
> reason :-)
The explosion of information in this thread is going to make for a
*great* followup blog post. :)
Now I just wish there were a way to get Portage and Make to watch CPU
usage and raise or lower the load-average threshold depending on how
much CPU was going to 'sys', 'user' and 'wait'; Lower -l if a great
deal of time is spent in 'sys'; you're likely burning cycles in
context switches. Raise -l if a great deal of time is spent waiting on
I/O.
It'd also be helpful to be able to give keystone[1] packages and Make
recipes a more favorable NICE value than those less important, to
induce the scheduler to favor the important packages over
less-important packages when we've got more ready work than cores. I
don't think Make *can* have the smarts for that, but Portage could
conceivably do it for its own parallelization.
[1] Where many things depend on them, either directly or indirectly.
Getting these out of the way means more parallel-buildable packages
being available at the same time.
--
:wq