From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1RdgZF-0002QW-VE for garchives@archives.gentoo.org; Thu, 22 Dec 2011 11:12:10 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id DE75A21C0B9; Thu, 22 Dec 2011 11:11:58 +0000 (UTC) Received: from mail-gy0-f181.google.com (mail-gy0-f181.google.com [209.85.160.181]) by pigeon.gentoo.org (Postfix) with ESMTP id 9C9D821C09F for ; Thu, 22 Dec 2011 11:11:33 +0000 (UTC) Received: by ghrr19 with SMTP id r19so2238027ghr.40 for ; Thu, 22 Dec 2011 03:11:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=VnK50dNzIN3bdnQIdQp/4YhfGAPnfxUnb2Ox+T0ynbw=; b=YAJNeWH/O4N4K6UutP22/Ev0ZVxvQ6Fv7LmGWXtyskOeRoOjlFCSZ3E/KUq8SEvcAh A7bgELwb56/8XoCjRc1bkBNNil62WX5zBgzjh9142yIj5+zSArLJBYsEKt6yxebkI/QF qQfWmRSmDfxkzvGJk2NcGWVHHj6kRhJ2GWSh4= Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Received: by 10.236.135.77 with SMTP id t53mr14131310yhi.49.1324552293117; Thu, 22 Dec 2011 03:11:33 -0800 (PST) Received: by 10.236.75.200 with HTTP; Thu, 22 Dec 2011 03:11:32 -0800 (PST) In-Reply-To: References: <20111222044323.GA18095@comet.roc.mn.charter.com> Date: Thu, 22 Dec 2011 12:11:32 +0100 Message-ID: Subject: Re: [gentoo-dev] Six month major project on Gentoo From: Francesco Riosa To: gentoo-dev@lists.gentoo.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 9509e4e3-398c-4fb4-8a8a-e9663bf1850d X-Archives-Hash: 017fe9832fb5120b33c5453ba6b414f3 2011/12/22 Rich Freeman : > On Wed, Dec 21, 2011 at 11:43 PM, Donnie Berkholz = wrote: >> I looked into this 6 or 7 years ago. It wasn't feasible unless you were >> on an extremely high-speed, low-latency network, beyond what was >> typically accessible at the time outside of universities and LANs. Could >> be worth exploring again now that 25-100 mbps connections are becoming >> more common. > > I tried messing around with this with Amazon EC2. =A0The problem was > that due to latency I only really saw the benefit for VERY high levels > of parallelization (think -j25+).. =A0However, make isn't actually > "distcc-aware" so it just runs 25 jobs of anything in parallel. =A0So, > anytime a makefile launched a ton of java or python jobs the host > ground to a halt as it wasn't distributed and it was way more than the > host could handle (especially java - which swapped like there was no > tomorrow). > > If somebody were to do a distcc-ng for a large cluster one of the > problems to solve would be having it not run jobs in parallel if it > couldn't actually distribute them. > > Rich > Just wanted to point out that (if there is enough memory) recent kernels manage much better parallelism, even excess of it, once reached the maximum load augmenting threads only bring minimal loss of "real" time.