From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 4DC03138010 for ; Wed, 12 Sep 2012 16:49:54 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 9E629E0687; Wed, 12 Sep 2012 16:49:35 +0000 (UTC) Received: from mail-bk0-f53.google.com (mail-bk0-f53.google.com [209.85.214.53]) by pigeon.gentoo.org (Postfix) with ESMTP id 114F0E0678 for ; Wed, 12 Sep 2012 16:48:54 +0000 (UTC) Received: by bkwj4 with SMTP id j4so23655bkw.40 for ; Wed, 12 Sep 2012 09:48:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=rYjK56v1vrISd96rE/MMNdPutB+eIrJ3R+dFTyNG15M=; b=XQp6yPH1Rw5PvV6uqJEjugAkjNPMoHK4SpCm6wlknJkKsz22Wx+u6Ga/CrxjHuenqZ Bv8sdNg5Oeqp5GWE9u8rxQVMTyUfYX/6GNxVAadK0H6gXgaktHi6/NGKA0PpYejcbL4b PNaCnEUaC/9WXyQse4nF8OlBuisnvZeSJ48EleEQhnJHEjbIPd4lIAu3+4KEIHQTdANf j1VRnIqp6HsB975oXjTgtMkycNwhzPiWynEu1L6IwIrUF6mcVAfCZJkQhrxGTGHfBa2H tLyUd1uJ4SmoPmQEU5u/xqcREl88v47Mg+nGYOke1yHR36y02/Gwwz9t7CAi6Ziyxb/p JmSQ== Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Received: by 10.204.152.75 with SMTP id f11mr6491428bkw.83.1347468533768; Wed, 12 Sep 2012 09:48:53 -0700 (PDT) Received: by 10.204.112.211 with HTTP; Wed, 12 Sep 2012 09:48:53 -0700 (PDT) In-Reply-To: <1347467612.32530.1.camel@localhost> References: <20544.29691.208130.35494@a1i15.kph.uni-mainz.de> <20120831154521.5258c549@googlemail.com> <20120831111244.0c17b8aa@gentoo.org> <20120902002002.GB25302@localhost> <20120904110041.GA19158@waltdnes.org> <50463738.7000209@gentoo.org> <504F6884.9000201@gmail.com> <504F6A21.4080709@gentoo.org> <504F6CB1.4030500@gentoo.org> <50505C12.4010307@malth.us> <50508700.1030605@gentoo.org> <1347467612.32530.1.camel@localhost> Date: Wed, 12 Sep 2012 12:48:53 -0400 Message-ID: Subject: Re: [gentoo-dev] EJOBS variable for EAPI 5? From: Michael Mol To: gentoo-dev@lists.gentoo.org Content-Type: multipart/alternative; boundary=00151759293c4d227e04c983f712 X-Archives-Salt: aa45c5fb-3a52-4dfa-b9b1-f74cea2a3b72 X-Archives-Hash: 4838a34038a20b87401933a4aa08f768 --00151759293c4d227e04c983f712 Content-Type: text/plain; charset=UTF-8 On Wed, Sep 12, 2012 at 12:33 PM, Hans de Graaff wrote: > On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: > > > So essentially what you're saying here is that it might be worthwhile > > to look into parallelism as a whole and possibly come up with a > > solution that combines 'emerge --jobs' and build-system parallelism > > together to maximum benefit? > > Forget about jobs and load average, and just keep starting jobs all > around until there is only 20% (or whatever tuneable amount) free memory > left. As far as I can tell this is always the real bottleneck in the > end. Once you hit swap overall throughput has to go down quite a bit. > > I've been thinking about this, but that only works until you get to the huge link step of, e.g. chromium, firefox, libreoffice. I've had programs with memory leaks in the past, but I've never seen a program validly consume as much memory as ld during those builds. To cover something like that, you would need to be able to freeze and swap out an entire process (such as ld) to allow something else to complete quickly...but there's no good way I can think of to prioritize sanely between the one big process and the few dozen smaller ones which might be allowed to spawn and complete first. -- :wq --00151759293c4d227e04c983f712 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On Wed, Sep 12, 2012 at 12:33 PM, Hans de Graaff= <graaff@gentoo.org> wrote:
On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote= :

> So essentially what you're saying here is that it might be worthwh= ile
> to look into parallelism as a whole and possibly come up with a
> solution that combines 'emerge --jobs' and build-system parall= elism
> together to maximum benefit?

Forget about jobs and load average, and just keep starting jobs all around until there is only 20% (or whatever tuneable amount) free memory left. As far as I can tell this is always the real bottleneck in the
end. Once you hit swap overall throughput has to go down quite a bit.

I've been thinking about this, but that only works until you ge= t to the huge link step of, e.g. chromium, firefox, libreoffice.

I've had programs with memory leaks in the past, bu= t I've never seen a program validly consume as much memory as ld during= those builds.

To cover something like that, you w= ould need to be able to freeze and swap out an entire process (such as ld) = to allow something else to complete quickly...but there's no good way I= can think of to prioritize sanely between the one big process and the few = dozen smaller ones which might be allowed to spawn and complete first.

--
:wq
--00151759293c4d227e04c983f712--