From mboxrd@z Thu Jan 1 00:00:00 1970
Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org)
by finch.gentoo.org with esmtp (Exim 4.60)
(envelope-from <gentoo-dev+bounces-49072-garchives=archives.gentoo.org@lists.gentoo.org>)
id 1Rdme1-0004Zb-PY
for garchives@archives.gentoo.org; Thu, 22 Dec 2011 17:41:30 +0000
Received: from pigeon.gentoo.org (localhost [127.0.0.1])
by pigeon.gentoo.org (Postfix) with SMTP id 5F6BF21C102;
Thu, 22 Dec 2011 17:41:17 +0000 (UTC)
Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183])
by pigeon.gentoo.org (Postfix) with ESMTP id 16C0321C066
for <gentoo-dev@lists.gentoo.org>; Thu, 22 Dec 2011 17:40:39 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1])
by smtp.gentoo.org (Postfix) with ESMTP id 938751B4029
for <gentoo-dev@lists.gentoo.org>; Thu, 22 Dec 2011 17:40:39 +0000 (UTC)
X-Virus-Scanned: by amavisd-new using ClamAV at gentoo.org
X-Spam-Flag: NO
X-Spam-Score: -4.869
X-Spam-Level:
X-Spam-Status: No, score=-4.869 tagged_above=-999 required=5.5
tests=[AWL=1.933, BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3,
RP_MATCHES_RCVD=-2.6, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001]
autolearn=ham
Received: from smtp.gentoo.org ([127.0.0.1])
by localhost (smtp.gentoo.org [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id Ogf0vUBLZYkg for <gentoo-dev@lists.gentoo.org>;
Thu, 22 Dec 2011 17:40:31 +0000 (UTC)
Received: from lo.gmane.org (lo.gmane.org [80.91.229.12])
by smtp.gentoo.org (Postfix) with ESMTP id 1DD781B4004
for <gentoo-dev@gentoo.org>; Thu, 22 Dec 2011 17:40:30 +0000 (UTC)
Received: from list by lo.gmane.org with local (Exim 4.69)
(envelope-from <lnx-gentoo-dev@m.gmane.org>)
id 1Rdmcz-00060w-4W
for gentoo-dev@gentoo.org; Thu, 22 Dec 2011 18:40:25 +0100
Received: from ip68-231-22-224.ph.ph.cox.net ([68.231.22.224])
by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
id 1AlnuQ-0007hv-00
for <gentoo-dev@gentoo.org>; Thu, 22 Dec 2011 18:40:25 +0100
Received: from 1i5t5.duncan by ip68-231-22-224.ph.ph.cox.net with local (Gmexim 0.1 (Debian))
id 1AlnuQ-0007hv-00
for <gentoo-dev@gentoo.org>; Thu, 22 Dec 2011 18:40:25 +0100
X-Injected-Via-Gmane: http://gmane.org/
To: gentoo-dev@lists.gentoo.org
From: Duncan <1i5t5.duncan@cox.net>
Subject: [gentoo-dev] Re: Six month major project on Gentoo
Date: Thu, 22 Dec 2011 17:40:13 +0000 (UTC)
Message-ID: <pan.2011.12.22.17.40.12@cox.net>
References: <CAOh0hwNZ07zVeFmSMO5ES5hbg=frM4g+gi8gmagkf-u+6Xue3g@mail.gmail.com>
<CAAr7Pr8cPELqcxVOPe8x2ERo0r26Zc-K_pDxUCu0S5yz3OwDdQ@mail.gmail.com>
<20111222044323.GA18095@comet.roc.mn.charter.com>
<CAGfcS_==iegE4yhtsG4Rn=ZefHeTQduwtt6pSnb5HbaRZTgU9g@mail.gmail.com>
<CAD6zcDwnexHze1S0uAnBCxVV6quh_JvU+pRAvtxefagyUHzKnQ@mail.gmail.com>
<20111222165518.6cd251d9@pomiocik.lan>
<CAGfcS_n3wP3O5gn8cv-BYXi1JwTpS8gPVomOLVG8XxCe6=A=ZA@mail.gmail.com>
Precedence: bulk
List-Post: <mailto:gentoo-dev@lists.gentoo.org>
List-Help: <mailto:gentoo-dev+help@lists.gentoo.org>
List-Unsubscribe: <mailto:gentoo-dev+unsubscribe@lists.gentoo.org>
List-Subscribe: <mailto:gentoo-dev+subscribe@lists.gentoo.org>
List-Id: Gentoo Linux mail <gentoo-dev.gentoo.org>
X-BeenThere: gentoo-dev@lists.gentoo.org
Reply-to: gentoo-dev@lists.gentoo.org
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
X-Complaints-To: usenet@dough.gmane.org
X-Gmane-NNTP-Posting-Host: ip68-231-22-224.ph.ph.cox.net
User-Agent: Pan/0.135 (Tomorrow I'll Wake Up and Scald Myself with Tea; GIT
ef87149 /st/portage/src/egit-src/pan2)
Content-Transfer-Encoding: quoted-printable
X-Archives-Salt: d5dbaffe-c334-4d88-a9cb-814daaa94893
X-Archives-Hash: a84cd3834da25217c07c89e0602964b8
Rich Freeman posted on Thu, 22 Dec 2011 11:09:16 -0500 as excerpted:
> On Thu, Dec 22, 2011 at 10:55 AM, Micha=C5=82 G=C3=B3rny <mgorny@gentoo=
.org>
> wrote:
>>> Just wanted to point out that (if there is enough memory) recent
>>> kernels manage much better parallelism, even excess of it, once
>>> reached the maximum load augmenting threads only bring minimal loss o=
f
>>> "real" time.
>>
>> Does that include handling complete lack of memory and heavy swapping?
>>
>>
> I think the key was the "if there is enough memory" - which I think is =
a
> pretty big issue.
User experience but it matters too...
The above memory limit-facter being the real limiter was true here when I=
=20
used to run unlimited local jobs, too (I moderated a decent bit since=20
portage itself parallelizes now, the fascination phase is over, and the=20
kernel no longer requires double-load-factor to stay 95-100% CPU=20
utilized). If the make is setup in such a way that it can sufficiently=20
parallelize, even back on dual-core, I could run hundreds of parallel=20
jobs (say, with the kernel build) surprisingly well -- as long as they=20
didn't require much memory.
As I've moderated since then, it's mostly to keep memory usage (and I/O,=20
especially when it compounds with swap) sensible, way more than it is any=
=20
real problem scheduling the actual loads. Plus, the kernel has gotten=20
rather better at fully utilizing CPU resources; it used to be that you=20
had to run about double (sometimes more, but generally slightly less)=20
your cores in load average to avoid idle CPUs. That's no longer the case=
=20
as a 25% over-scheduling seems to do what it used to require a 100% over-
scheduling to achieve in terms of keeping 95-100% CPU usage, so by 50%=20
over you're just wasting cycles and bytes.
What I /really/ wish was that there was a make (and portage) switch=20
parallel to -l, that responded based on memory instead of load average! =20
Then those single-job gig-plus links could limit to say 4 jobs on an 8-
gig machine, even if it's an 8-way that would otherwise be -j16 -l10 or
-l12 (regardless of the balance or lack thereof, of an 8-way with only a=20
gig of RAM per core... 2 gigs/core seems to be about the sweet-spot I've=20
found for a general purpose gentooer build-station/desktop).
That'd be an interesting and very useful parallel-domain task, adding a=20
make switch that responded to memory just as -l responds to load. Maybe=20
a make other than gmake already has such an option?
--=20
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman