From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1Sb31X-0005Ai-9r for garchives@archives.gentoo.org; Sun, 03 Jun 2012 05:06:43 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 5C1C8E0738; Sun, 3 Jun 2012 05:06:28 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by pigeon.gentoo.org (Postfix) with ESMTP id CFA5EE0700 for ; Sun, 3 Jun 2012 05:05:47 +0000 (UTC) Received: from vapier.localnet (localhost [127.0.0.1]) by smtp.gentoo.org (Postfix) with ESMTP id 4F9541B4008 for ; Sun, 3 Jun 2012 05:05:47 +0000 (UTC) From: Mike Frysinger Organization: wh0rd.org To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash Date: Sun, 3 Jun 2012 01:05:49 -0400 User-Agent: KMail/1.13.7 (Linux/3.4.0; KDE/4.6.5; x86_64; ; ) References: <20120602235902.GC9296@localhost> In-Reply-To: <20120602235902.GC9296@localhost> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart3134076.hz9m9Imj9j"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <201206030105.50520.vapier@gentoo.org> X-Archives-Salt: dc699d0d-945b-477a-b14c-624fbecd3ee1 X-Archives-Hash: a00021935f1d189a4a7e97907b5c8e21 --nextPart3134076.hz9m9Imj9j Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable On Saturday 02 June 2012 19:59:02 Brian Harring wrote: > On Fri, Jun 01, 2012 at 06:41:22PM -0400, Mike Frysinger wrote: > > # @FUNCTION: multijob_post_fork > > # @DESCRIPTION: > > # You must call this in the parent process after forking a child proces= s. > > # If the parallel limit has been hit, it will wait for one to finish and > > # return the child's exit status. > > multijob_post_fork() { > >=20 > > [[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments" > > =09 > > : $(( ++mj_num_jobs )) > > =09 > > if [[ ${mj_num_jobs} -ge ${mj_max_jobs} ]] ; then > > =09 > > multijob_finish_one > > =09 > > fi > > return $? > >=20 > > } >=20 > Minor note; the design of this (fork then check), means when a job > finishes, we'll not be ready with more work. This implicitly means > that given a fast job identification step (main thread), and a slower > job execution (what's backgrounded), we'll not breach #core of > parallelism, nor will we achieve that level either (meaning > potentially some idle cycles left on the floor). >=20 > Realistically, the main thread (what invokes post_fork) is *likely*, > (if the consumer isn't fricking retarded) to be doing minor work- > mostly just poking about figuring out what the next task/arguments > are to submit to the pool. That work isn't likely to be a full core > worth of work, else as I said, the consumer is being a retard. >=20 > The original form of this was designed around the assumption that the > main thread was light, and the backgrounded jobs weren't, thus it > basically did the equivalent of make -j+1, allowing #cores > background jobs running, while allowing the main thread to continue on > and get the next job ready, once it had that ready, it would block > waiting for a slot to open, then immediately submit the job once it > had done a reclaim. the original code i designed this around had a heavier main thread because = it=20 had series of parallel sections followed by serial followed by parallel whe= re=20 the serial regions didn't depend on the parallel finishing right away. tha= t=20 and doing things post meant it was easier to pass up return values because = i=20 didn't have to save $? anywhere ;). thinking a bit more, i don't think the two methods are mutually exclusive. = =20 it's easy to have the code support both, but i'm not sure the extended=20 documentation helps. =2Dmike --nextPart3134076.hz9m9Imj9j Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAABAgAGBQJPyvCuAAoJEEFjO5/oN/WBCoMQAK6xSghwsHU4D9ZZbNWCGS9D 1XqPiBjJShP0L4Su2f+2z0tDOK7MHX7g6u/ZFoNP7AJUBX2gIjCuezVIULZeIyqQ v3RYG2YXuzyj/a/yfQRc0C0j1xrGZmCxq8eN1oE9J4yKazp/vhRcQ+iZ0JsO9amW 9npS8of2pLIrZXP8yfsB/qXRA4IEQFIGbmIDaNojriJW5i5HhGSzJGq+6Xw5Q/2Q +ilHblsneMRy2wTyB8IT/5v0NsJHORGQC1f6dtNdyiRTwahtRzSX336DrYjorarM HGETsObgbG2RIoO8fiJOBskx/E2/3N7hrnE1B6lDDjLuufKAUzmZFS/VK284Hr/6 QIeqeKmfuc105AC2C6rgaSjdbgweDZY+WI60qasNJrNQEHVcAFwKEN20qotivDHP 1mAvL1474CBtWBKlZgEyqWHrbupcj7xO7YjS8lZeubI3smRupXIZiH+T5RK/CovE RWmTnZ7F/n0bPPb9zGMQr1oB0gnI3o9KKwPHFJs3HajEbfGfoIIzq+lAppggONFc 8Hk+9l5zfc119f/fRCjFXse2/QxZwYaaq1mrQUyB32iwyrv/RgUe0FOImXdqv3He TzFYb7HEikz2eoxSpvPV7AU0gA40XQx0OOyVO1+z5Wk/hxkC7TyFdwXoAQeJ8vJy QynkBSv/7Mkj2x38x59E =TQql -----END PGP SIGNATURE----- --nextPart3134076.hz9m9Imj9j--