From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 5BD38138010 for ; Wed, 12 Sep 2012 17:04:32 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 4B7E221C01C; Wed, 12 Sep 2012 17:04:12 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by pigeon.gentoo.org (Postfix) with ESMTP id B41AE21C00D for ; Wed, 12 Sep 2012 17:03:26 +0000 (UTC) Received: from [192.168.1.145] (CPE002401f30b73-CM001cea3ddad8.cpe.net.cable.rogers.com [99.240.69.152]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: axs) by smtp.gentoo.org (Postfix) with ESMTPSA id 1B18C33C99E for ; Wed, 12 Sep 2012 17:03:26 +0000 (UTC) Message-ID: <5050C05A.1030107@gentoo.org> Date: Wed, 12 Sep 2012 13:03:22 -0400 From: Ian Stakenvicius User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.6esrpre) Gecko/20120731 Thunderbird/10.0.6 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] EJOBS variable for EAPI 5? References: <20544.29691.208130.35494@a1i15.kph.uni-mainz.de> <20120831154521.5258c549@googlemail.com> <20120831111244.0c17b8aa@gentoo.org> <20120902002002.GB25302@localhost> <20120904110041.GA19158@waltdnes.org> <50463738.7000209@gentoo.org> <504F6884.9000201@gmail.com> <504F6A21.4080709@gentoo.org> <504F6CB1.4030500@gentoo.org> <50505C12.4010307@malth.us> <50508700.1030605@gentoo.org> <1347467612.32530.1.camel@localhost> <5050BF1E.3020501@gentoo.org> In-Reply-To: <5050BF1E.3020501@gentoo.org> X-Enigmail-Version: 1.3.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Archives-Salt: b88c9a10-983e-4fe4-8ba6-d7fe8ce909a4 X-Archives-Hash: 406037b8abf8b77d5f8a94930b6bb6a7 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/09/12 12:58 PM, Zac Medico wrote: > On 09/12/2012 09:33 AM, Hans de Graaff wrote: >> On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote: >> >>> So essentially what you're saying here is that it might be >>> worthwhile to look into parallelism as a whole and possibly >>> come up with a solution that combines 'emerge --jobs' and >>> build-system parallelism together to maximum benefit? >> >> Forget about jobs and load average, and just keep starting jobs >> all around until there is only 20% (or whatever tuneable amount) >> free memory left. As far as I can tell this is always the real >> bottleneck in the end. Once you hit swap overall throughput has >> to go down quite a bit. > > Well, I think it's still good to limit the number of jobs at > least, since otherwise you could become overloaded with processes > that don't consume a lot of memory at first but by the time they > complete they have consumed much more memory than desired (using > swap). I think this would need to be dealt with by having the parent emerge process monitor all children and specifically block individual processes (ie, 'make' , 'ld' , etc) once resources are unavailable until they become so. Swap may be hit by the big processes but they wouldn't continue to be processed while in swap, at least. I don't have a solution to the potential 'thrashing' issue, though, which i expect would be a problem even if there's enough memory. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iF4EAREIAAYFAlBQwFoACgkQ2ugaI38ACPAiAwD/foU8Xw1BQM3jeO6IiVfCGOnw xHtIufwVmMpsGVdJQRIA/3W7Utg92foSc6ZtKMzBP5Fj0qB2BXMt/RS2I4pLsCQm =gy9K -----END PGP SIGNATURE-----