public inbox for gentoo-dev@lists.gentoo.org
 help / color / mirror / Atom feed
From: Duncan <1i5t5.duncan@cox.net>
To: gentoo-dev@lists.gentoo.org
Subject: [gentoo-dev] Re: Six month major project on Gentoo
Date: Thu, 22 Dec 2011 17:40:13 +0000 (UTC)	[thread overview]
Message-ID: <pan.2011.12.22.17.40.12@cox.net> (raw)
In-Reply-To: CAGfcS_n3wP3O5gn8cv-BYXi1JwTpS8gPVomOLVG8XxCe6=A=ZA@mail.gmail.com

Rich Freeman posted on Thu, 22 Dec 2011 11:09:16 -0500 as excerpted:

> On Thu, Dec 22, 2011 at 10:55 AM, Michał Górny <mgorny@gentoo.org>
> wrote:
>>> Just wanted to point out that (if there is enough memory) recent
>>> kernels manage much better parallelism, even excess of it, once
>>> reached the maximum load augmenting threads only bring minimal loss of
>>> "real" time.
>>
>> Does that include handling complete lack of memory and heavy swapping?
>>
>>
> I think the key was the "if there is enough memory" - which I think is a
> pretty big issue.

User experience but it matters too...

The above memory limit-facter being the real limiter was true here when I 
used to run unlimited local jobs, too (I moderated a decent bit since 
portage itself parallelizes now, the fascination phase is over, and the 
kernel no longer requires double-load-factor to stay 95-100% CPU 
utilized).  If the make is setup in such a way that it can sufficiently 
parallelize, even back on dual-core, I could run hundreds of parallel 
jobs (say, with the kernel build) surprisingly well -- as long as they 
didn't require much memory.

As I've moderated since then, it's mostly to keep memory usage (and I/O, 
especially when it compounds with swap) sensible, way more than it is any 
real problem scheduling the actual loads.  Plus, the kernel has gotten 
rather better at fully utilizing CPU resources; it used to be that you 
had to run about double (sometimes more, but generally slightly less) 
your cores in load average to avoid idle CPUs.  That's no longer the case 
as a 25% over-scheduling seems to do what it used to require a 100% over-
scheduling to achieve in terms of keeping 95-100% CPU usage, so by 50% 
over you're just wasting cycles and bytes.

What I /really/ wish was that there was a make (and portage) switch 
parallel to -l, that responded based on memory instead of load average!  
Then those single-job gig-plus links could limit to say 4 jobs on an 8-
gig machine, even if it's an 8-way that would otherwise be -j16 -l10 or
-l12 (regardless of the balance or lack thereof, of an 8-way with only a 
gig of RAM per core... 2 gigs/core seems to be about the sweet-spot I've 
found for a general purpose gentooer build-station/desktop).

That'd be an interesting and very useful parallel-domain task, adding a 
make switch that responded to memory just as -l responds to load.  Maybe 
a make other than gmake already has such an option?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman




  reply	other threads:[~2011-12-22 17:41 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-12-14 11:06 [gentoo-dev] Six month major project on Gentoo Gaurav Saxena
2011-12-14 18:05 ` Christian Ruppert
2011-12-14 18:14   ` "C. Bergström"
2011-12-15 18:07   ` Gaurav Saxena
2011-12-19 17:46     ` Christian Ruppert
2011-12-14 18:29 ` Alec Warner
2011-12-15 18:09   ` Gaurav Saxena
2011-12-22  4:43   ` Donnie Berkholz
2011-12-22 10:52     ` Rich Freeman
2011-12-22 11:11       ` Francesco Riosa
2011-12-22 15:55         ` Michał Górny
2011-12-22 16:09           ` Rich Freeman
2011-12-22 17:40             ` Duncan [this message]
2011-12-18 17:02 ` Petteri Räty
2011-12-18 17:13   ` "Paweł Hajdan, Jr."
2011-12-18 17:17     ` [gentoo-dev] libbash licensing Petteri Räty
2011-12-18 17:45     ` [gentoo-dev] Six month major project on Gentoo Michał Górny
2011-12-19 18:14 ` Sébastien Fabbro
2012-01-02 14:33   ` "Paweł Hajdan, Jr."

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=pan.2011.12.22.17.40.12@cox.net \
    --to=1i5t5.duncan@cox.net \
    --cc=gentoo-dev@lists.gentoo.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox