From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1Rdme1-0004Zb-PY for garchives@archives.gentoo.org; Thu, 22 Dec 2011 17:41:30 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 5F6BF21C102; Thu, 22 Dec 2011 17:41:17 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by pigeon.gentoo.org (Postfix) with ESMTP id 16C0321C066 for ; Thu, 22 Dec 2011 17:40:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.gentoo.org (Postfix) with ESMTP id 938751B4029 for ; Thu, 22 Dec 2011 17:40:39 +0000 (UTC) X-Virus-Scanned: by amavisd-new using ClamAV at gentoo.org X-Spam-Flag: NO X-Spam-Score: -4.869 X-Spam-Level: X-Spam-Status: No, score=-4.869 tagged_above=-999 required=5.5 tests=[AWL=1.933, BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-2.6, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham Received: from smtp.gentoo.org ([127.0.0.1]) by localhost (smtp.gentoo.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ogf0vUBLZYkg for ; Thu, 22 Dec 2011 17:40:31 +0000 (UTC) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by smtp.gentoo.org (Postfix) with ESMTP id 1DD781B4004 for ; Thu, 22 Dec 2011 17:40:30 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Rdmcz-00060w-4W for gentoo-dev@gentoo.org; Thu, 22 Dec 2011 18:40:25 +0100 Received: from ip68-231-22-224.ph.ph.cox.net ([68.231.22.224]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 22 Dec 2011 18:40:25 +0100 Received: from 1i5t5.duncan by ip68-231-22-224.ph.ph.cox.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 22 Dec 2011 18:40:25 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: gentoo-dev@lists.gentoo.org From: Duncan <1i5t5.duncan@cox.net> Subject: [gentoo-dev] Re: Six month major project on Gentoo Date: Thu, 22 Dec 2011 17:40:13 +0000 (UTC) Message-ID: References: <20111222044323.GA18095@comet.roc.mn.charter.com> <20111222165518.6cd251d9@pomiocik.lan> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: ip68-231-22-224.ph.ph.cox.net User-Agent: Pan/0.135 (Tomorrow I'll Wake Up and Scald Myself with Tea; GIT ef87149 /st/portage/src/egit-src/pan2) Content-Transfer-Encoding: quoted-printable X-Archives-Salt: d5dbaffe-c334-4d88-a9cb-814daaa94893 X-Archives-Hash: a84cd3834da25217c07c89e0602964b8 Rich Freeman posted on Thu, 22 Dec 2011 11:09:16 -0500 as excerpted: > On Thu, Dec 22, 2011 at 10:55 AM, Micha=C5=82 G=C3=B3rny > wrote: >>> Just wanted to point out that (if there is enough memory) recent >>> kernels manage much better parallelism, even excess of it, once >>> reached the maximum load augmenting threads only bring minimal loss o= f >>> "real" time. >> >> Does that include handling complete lack of memory and heavy swapping? >> >> > I think the key was the "if there is enough memory" - which I think is = a > pretty big issue. User experience but it matters too... The above memory limit-facter being the real limiter was true here when I= =20 used to run unlimited local jobs, too (I moderated a decent bit since=20 portage itself parallelizes now, the fascination phase is over, and the=20 kernel no longer requires double-load-factor to stay 95-100% CPU=20 utilized). If the make is setup in such a way that it can sufficiently=20 parallelize, even back on dual-core, I could run hundreds of parallel=20 jobs (say, with the kernel build) surprisingly well -- as long as they=20 didn't require much memory. As I've moderated since then, it's mostly to keep memory usage (and I/O,=20 especially when it compounds with swap) sensible, way more than it is any= =20 real problem scheduling the actual loads. Plus, the kernel has gotten=20 rather better at fully utilizing CPU resources; it used to be that you=20 had to run about double (sometimes more, but generally slightly less)=20 your cores in load average to avoid idle CPUs. That's no longer the case= =20 as a 25% over-scheduling seems to do what it used to require a 100% over- scheduling to achieve in terms of keeping 95-100% CPU usage, so by 50%=20 over you're just wasting cycles and bytes. What I /really/ wish was that there was a make (and portage) switch=20 parallel to -l, that responded based on memory instead of load average! =20 Then those single-job gig-plus links could limit to say 4 jobs on an 8- gig machine, even if it's an 8-way that would otherwise be -j16 -l10 or -l12 (regardless of the balance or lack thereof, of an 8-way with only a=20 gig of RAM per core... 2 gigs/core seems to be about the sweet-spot I've=20 found for a general purpose gentooer build-station/desktop). That'd be an interesting and very useful parallel-domain task, adding a=20 make switch that responded to memory just as -l responds to load. Maybe=20 a make other than gmake already has such an option? --=20 Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman