From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1SagQN-00077Y-Bk for garchives@archives.gentoo.org; Sat, 02 Jun 2012 04:58:52 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id E7B8EE0795; Sat, 2 Jun 2012 04:58:32 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by pigeon.gentoo.org (Postfix) with ESMTP id 76097E0793 for ; Sat, 2 Jun 2012 04:57:37 +0000 (UTC) Received: from vapier.localnet (localhost [127.0.0.1]) by smtp.gentoo.org (Postfix) with ESMTP id E5FE21B4011 for ; Sat, 2 Jun 2012 04:57:36 +0000 (UTC) From: Mike Frysinger Organization: wh0rd.org To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] multiprocessing.eclass: doing parallel work in bash Date: Sat, 2 Jun 2012 00:57:40 -0400 User-Agent: KMail/1.13.7 (Linux/3.4.0; KDE/4.6.5; x86_64; ; ) References: <201206011841.23302.vapier@gentoo.org> <20120602041119.GA9296@localhost> In-Reply-To: <20120602041119.GA9296@localhost> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1494330.ylB7XLeD7Y"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <201206020057.40625.vapier@gentoo.org> X-Archives-Salt: b68eef24-4ef9-4c76-a9df-8ced120405b3 X-Archives-Hash: a2b1c4039234dfd5cd9f1757453786ba --nextPart1494330.ylB7XLeD7Y Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable On Saturday 02 June 2012 00:11:19 Brian Harring wrote: > On Fri, Jun 01, 2012 at 06:41:22PM -0400, Mike Frysinger wrote: > > and put it into a new multiprocessing.eclass. this way people can > > generically utilize this in their own eclasses/ebuilds. > >=20 > > it doesn't currently support nesting. not sure if i should fix that. > >=20 > > i'll follow up with an example of parallelizing of eautoreconf. for > > mail-filter/maildrop on my 4 core system, it cuts the time needed to run > > from ~2.5 min to ~1 min. >=20 > My main concern here is cleanup during uncontrolled shutdown; if the > backgrounded job has hung itself for some reason, the job *will* just > sit; I'm not aware of any of the PMs doing process tree killing, or > cgroups containment; in my copious free time I'm planning on adding a > 'cjobs' tool for others, and adding cgroups awareness into pkgcore; > that said, none of 'em do this *now*, thus my concern. i'm not sure there's much i can do here beyond adding traps > > makeopts_jobs() { >=20 > This function belongs in eutils, or somewhere similar- pretty sure > we've got variants of this in multiple spots. I'd prefer a single > point to change if/when we add a way to pass parallelism down into the > env via EAPI. it's already in eutils. but i'm moving it out of that and into this since = it=20 makes more sense in this eclass imo, and avoids this eclass from inheriting= =20 eutils. > > multijob_child_init() { > > [[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments" > > trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT > > trap 'exit 1' INT TERM > > } >=20 > Kind of dislike this form since it means consuming code has to be > aware of, and do the () & trick. >=20 > A helper function, something like > multijob_child_job() { > ( > multijob_child_init > "$@" > ) & > multijob_post_fork || die "game over man, game over" > } >=20 > Doing so, would conver your eautoreconf from: > for x in $(autotools_check_macro_val AC_CONFIG_SUBDIRS) ; do > if [[ -d ${x} ]] ; then > pushd "${x}" >/dev/null > ( > multijob_child_init > AT_NOELIBTOOLIZE=3D"yes" eautoreconf > ) & > multijob_post_fork || die > popd >/dev/null > fi > done >=20 > To: > for x in $(autotools_check_macro_val AC_CONFIG_SUBDIRS) ; do > if [[ -d ${x} ]]; then > pushd "${x}" > /dev/null > AT_NOELIBTOOLIZE=3D"yes" multijob_child_job eautoreconf > popd > fi > done it depends on the form of the code. i can see both being useful. should b= e=20 easy to support both though: multijob_child_init() { if [[ $# -eq 0 ]] ; then trap 'echo ${BASHPID} $? >&'${mj_control_fd} EXIT trap 'exit 1' INT TERM else ( multijob_child_init "$@" ) & multijob_post_fork || die fi } > Note, if we used an eval in multijob_child_job, the pushd/popd could > be folded in. Debatable. i'd lean towards not. keeps things simple and people don't have to get int= o=20 quoting hell. > > # @FUNCTION: multijob_finish_one > > # @DESCRIPTION: > > # Wait for a single process to exit and return its exit code. > > multijob_finish_one() { > >=20 > > [[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments" > > =09 > > local pid ret > > read -r -u ${mj_control_fd} pid ret >=20 > Mildly concerned about the failure case here- specifically if the read > fails (fd was closed, take your pick). read || die ? not sure what else could be done really. > > multijob_finish() { > > [[ $# -eq 0 ]] || die "${FUNCNAME} takes no arguments" >=20 > Tend to think this should do cleanup, then die if someone invoked the > api incorrectly; I'd rather see the children reaped before this blows > up. sounds good. along those lines, i could add multijob_finish to=20 EBUILD_DEATH_HOOKS so other `die` points also wait by default ... =2Dmike --nextPart1494330.ylB7XLeD7Y Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAABAgAGBQJPyZ1EAAoJEEFjO5/oN/WB5wcQAKJKONLSodDzFDNBflLLEQNE IUkK9Y/w2OLJv0aovEeiUow2iqLHR14H1MDSZLCpDSPM8lFpS1nxHhUZWAmhL42a qb1EhfHjXqldulc0/OCbwSMqQYZejuYMn2sSgAIRg5kF2BR4pZA7IvZsSAA6kgwr 2KKgjB23zWyMkkQ3oo6J+kykiWU8LNixHLik0dsN+YcmwAmonE2Y24/cdesVSp+Y dHqlUij/o53AnQaX4Dyijadmy/+89oo+5AlG6fpxoTXIvk0wYiIQ9Y8CZCbnj3cD MIhknJ6mi/oXCwefpuMWPW96Iy5Lgo+sLe0Y7x19FWqGioHP1yHE0YQnvfrzMUkJ 6xo2CUYJAVoLAhRpehtOnLm2fvue80q8E2rl92KcCDGDCVYkZGS4YfCQ7ukCpTlW z8h4NvxUWCX+E7NElIuBuRMy2eTXMKItIoStTq75bMRAi3za5D517A0nR/z7D3Qm VnjdfXVWfJl03qlYVgSPD2aQILVPUo3psVkfCAtvZei+a8ykH0min4eD8mesBtY3 faxUaM9eYjNsSqq44nievDgk0rbTFf1qOxPSt+OVVJr/PUiA9ObD7tQmS6vz8oVk Ro3AwQR83KhHuxwJScAQQB4EIl7Q+UOqaU+fzNv64YUHR34IBQdtcyU6/n70sj4+ i7DxzkpBLaSTcdiZdIuR =dqnB -----END PGP SIGNATURE----- --nextPart1494330.ylB7XLeD7Y--