From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id A0F97138010 for ; Wed, 12 Sep 2012 09:57:02 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 71884E07B5; Wed, 12 Sep 2012 09:56:38 +0000 (UTC) Received: from malth.us (malth.us [75.147.143.249]) by pigeon.gentoo.org (Postfix) with ESMTP id 73BF1E07AA for ; Wed, 12 Sep 2012 09:55:50 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by malth.us (Postfix) with ESMTP id 71BCA40CA4E8 for ; Wed, 12 Sep 2012 02:55:49 -0700 (PDT) X-Virus-Scanned: amavisd-new at malth.us Received: from malth.us ([127.0.0.1]) by localhost (malth.us [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D2kfm8nGniAY for ; Wed, 12 Sep 2012 02:55:33 -0700 (PDT) Received: from [192.168.0.175] (moneypit.hq [192.168.0.175]) by malth.us (Postfix) with ESMTPSA id DFEAF40CB671 for ; Wed, 12 Sep 2012 02:55:32 -0700 (PDT) Message-ID: <50505C12.4010307@malth.us> Date: Wed, 12 Sep 2012 02:55:30 -0700 From: "Gregory M. Turner" User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20120907 Thunderbird/15.0.1 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] EJOBS variable for EAPI 5? References: <20544.29691.208130.35494@a1i15.kph.uni-mainz.de> <20120831154521.5258c549@googlemail.com> <20120831111244.0c17b8aa@gentoo.org> <20120902002002.GB25302@localhost> <20120904110041.GA19158@waltdnes.org> <50463738.7000209@gentoo.org> <504F6884.9000201@gmail.com> <504F6A21.4080709@gentoo.org> <504F6CB1.4030500@gentoo.org> In-Reply-To: <504F6CB1.4030500@gentoo.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Archives-Salt: d2649541-cdf7-4c67-9fbf-28d71f0f5e00 X-Archives-Hash: 55387566816d25e5bd413430250daf19 On 9/11/2012 9:54 AM, Ian Stakenvicius wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > On 11/09/12 12:43 PM, Zac Medico wrote: >> On 09/11/2012 09:36 AM, vivo75@gmail.com wrote: >>> Dunno where to place this request, but if we go for something >>> like EJOBS could we also make it phase specific? So compile, >>> install and test could have a different number of jobs running. >>> Possibly three different variables that override a predefined >>> EJOBS. >> >> Per-phase sounds a little to fine-grained. Instead, I'd suggest to >> add an ELOADAVG variable that's analogous to make's --load-average >> option. That should be enough to compensate for any differences >> between phases. > > I personally wonder about why this would be necessary from the > perspective of the user; if the user's system at emerge time can > handle X concurrent processes per emerge-job , i don't see why it > would matter what phase these jobs would be launched from. > > At the ebuild level, certainly, but that's one of the reasons for > EJOBS in the first place, so that it can be overridden consistently > within a phase, if necessary for the ebuild (regardless of build > system type), right? I would hope so. There are definitely real-world reasons to want to restrict jobs -- usually to just one -- in a particular phase... several ebuilds, notably several in @system, do this already by injecting -j1 somewhere. To say it's all about performance seems to be forgetting a major reason somebody might want to do this as a user or an ebuild author. Hmm... I should preface this by saying that it really pains me to say this, because now this just starts to seem so fucking complicated -- but now that I think about it, this really seems to highlight a kind of semantic discrepancy that this thread has mostly dusted under the rug. There are really TWO things we seem to care about -- one is some kind of global build-parallelism frob-set, and the other is the ability to turn off all parallelism for certain parts of an ebuild and those are actually kind-of orthogonal. In other words, imagine if we had some kind of "global parallellism arbiter" thingy that looked at EJOBS or ELOADAVG or maybe some other things we haven't thought of yet, and decided things like: o should portage start a new parallel emerge process? o What is the appropriate "MAKEOPTS" for the "emake" that foo-1.0.ebuild just issued? or even, one could imagine, such things as: o should portage decide that parallelism has gotten out of control somehow and suspend or kill a running ebuild for subsequent resuming or restarting? o should portage re-nice some of its processes? Note that, effectively, we have this already, and it's called "portage". But one could certainly make a case for modularizing it better, since, in truth, we are talking about a very common, very abstract problem here which portage shares with any number of batch-build systems. Such an engine could very well do exactly the right thing if it were faced with a constraint that a certain part of a certain build needed to proceed without parallelism due to limitations coming from the build. Also, there are very large parts of most builds -- configure comes to mind -- that don't parallelize even if, perhaps, they should. In such cases, a really smart global parallelism arbiter could easily respond by spawning more jobs from other builds. Not sure what I'm suggesting we do about it, exactly, but just pointing out that maybe a completely "correct" solution requires a much more elaborate implementation than just a bunch of syntactic sugar around what we currently call MAKEOPTS. Whether or not Gentoo wants to take that all on, right now, as part of the next EAPI, is certainly debatable -- in fact, I'm inclined to say maybe it's not the best idea. Perhaps we should really be asking: is the status-quo really so problematic/inelegant that it needs fixing? -- before we decide how best to fix it. -gmt