From: Duncan <1i5t5.duncan@cox.net>
To: gentoo-dev@lists.gentoo.org
Subject: [gentoo-dev] Re: Six month major project on Gentoo
Date: Thu, 22 Dec 2011 17:40:13 +0000 (UTC) [thread overview]
Message-ID: <pan.2011.12.22.17.40.12@cox.net> (raw)
In-Reply-To: CAGfcS_n3wP3O5gn8cv-BYXi1JwTpS8gPVomOLVG8XxCe6=A=ZA@mail.gmail.com
Rich Freeman posted on Thu, 22 Dec 2011 11:09:16 -0500 as excerpted:
> On Thu, Dec 22, 2011 at 10:55 AM, Michał Górny <mgorny@gentoo.org>
> wrote:
>>> Just wanted to point out that (if there is enough memory) recent
>>> kernels manage much better parallelism, even excess of it, once
>>> reached the maximum load augmenting threads only bring minimal loss of
>>> "real" time.
>>
>> Does that include handling complete lack of memory and heavy swapping?
>>
>>
> I think the key was the "if there is enough memory" - which I think is a
> pretty big issue.
User experience but it matters too...
The above memory limit-facter being the real limiter was true here when I
used to run unlimited local jobs, too (I moderated a decent bit since
portage itself parallelizes now, the fascination phase is over, and the
kernel no longer requires double-load-factor to stay 95-100% CPU
utilized). If the make is setup in such a way that it can sufficiently
parallelize, even back on dual-core, I could run hundreds of parallel
jobs (say, with the kernel build) surprisingly well -- as long as they
didn't require much memory.
As I've moderated since then, it's mostly to keep memory usage (and I/O,
especially when it compounds with swap) sensible, way more than it is any
real problem scheduling the actual loads. Plus, the kernel has gotten
rather better at fully utilizing CPU resources; it used to be that you
had to run about double (sometimes more, but generally slightly less)
your cores in load average to avoid idle CPUs. That's no longer the case
as a 25% over-scheduling seems to do what it used to require a 100% over-
scheduling to achieve in terms of keeping 95-100% CPU usage, so by 50%
over you're just wasting cycles and bytes.
What I /really/ wish was that there was a make (and portage) switch
parallel to -l, that responded based on memory instead of load average!
Then those single-job gig-plus links could limit to say 4 jobs on an 8-
gig machine, even if it's an 8-way that would otherwise be -j16 -l10 or
-l12 (regardless of the balance or lack thereof, of an 8-way with only a
gig of RAM per core... 2 gigs/core seems to be about the sweet-spot I've
found for a general purpose gentooer build-station/desktop).
That'd be an interesting and very useful parallel-domain task, adding a
make switch that responded to memory just as -l responds to load. Maybe
a make other than gmake already has such an option?
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
next prev parent reply other threads:[~2011-12-22 17:41 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-12-14 11:06 [gentoo-dev] Six month major project on Gentoo Gaurav Saxena
2011-12-14 18:05 ` Christian Ruppert
2011-12-14 18:14 ` "C. Bergström"
2011-12-15 18:07 ` Gaurav Saxena
2011-12-19 17:46 ` Christian Ruppert
2011-12-14 18:29 ` Alec Warner
2011-12-15 18:09 ` Gaurav Saxena
2011-12-22 4:43 ` Donnie Berkholz
2011-12-22 10:52 ` Rich Freeman
2011-12-22 11:11 ` Francesco Riosa
2011-12-22 15:55 ` Michał Górny
2011-12-22 16:09 ` Rich Freeman
2011-12-22 17:40 ` Duncan [this message]
2011-12-18 17:02 ` Petteri Räty
2011-12-18 17:13 ` "Paweł Hajdan, Jr."
2011-12-18 17:17 ` [gentoo-dev] libbash licensing Petteri Räty
2011-12-18 17:45 ` [gentoo-dev] Six month major project on Gentoo Michał Górny
2011-12-19 18:14 ` Sébastien Fabbro
2012-01-02 14:33 ` "Paweł Hajdan, Jr."
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=pan.2011.12.22.17.40.12@cox.net \
--to=1i5t5.duncan@cox.net \
--cc=gentoo-dev@lists.gentoo.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox