[gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization

[gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization

[Greg Turner]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: 005-MULTILIB_PARALLEL_PHASES.patch --]
[-- Type: text/x-patch, Size: 4866 bytes --]

This patch adds a new frob, MULTILIB_PARALLEL_PHASES, to
multlib-minimal.eclass, which implements eclass-consumer-selectable
parallelization of src_configure, src_compile, and src_test.

By default, all parallelization is deactivated, which represents
a change from the previous gentoo-x86 behavior (which was
parallelizing src_configure automatically).

--- 004-multilib-phase-all/multilib-minimal.eclass	2013-12-03 02:54:40.045335905 -0800
+++ 005-MULTILIB_PARALLEL_PHASES/multilib-minimal.eclass	2013-12-03 02:59:33.687448429 -0800
@@ -45,8 +45,8 @@
 # for more information.
 #
 # Another snag you may encounter is that, in many circumstances, it is possible
-# for variables to flow between multilib_<phase> functions.  This is because
-# no subshell is utilized.
+# for variables to flow between multilib_<phase> functions.  This is because,
+# unless you parallelize the phase in question, no subshell is utilized.
 # Extreme care must sometimes be taken to ensure that different ABI's don't
 # trample each other's variables.
 #
@@ -122,6 +122,58 @@ case ${EAPI:-0} in
 	*) die "EAPI=${EAPI} is not supported" ;;
 esac
 
+# @ECLASS-VARIABLE: MULTILIB_PARALLEL_PHASES
+# @DEFAULT-UNSET
+# @DESCRIPTION:
+# multilib-minimal.eclass consumers may set this to a string containing
+# a space-separated list of phase-function names, like so:
+#
+# @CODE@
+# MULTILIB_PARALLEL_PHASES="src_configure src_test"
+# @CODE@
+#
+# For each (supported) phase-function name in the list, the corresponding
+# multilib-minimal_<phase> phase function will execute in parallel, in
+# the sense that, within limits (set by multiprocessing.eclass) the
+# multilib_<phase> function invocations (or, if none is present, the
+# built-in default implementation) for each ABI will run simultaneously.
+#
+# Any phase-function-names not specified are executed serially, with the
+# native ABI coming last.
+#
+# It is OK to add extra "bonus" whitespace or duplicated values.
+# Changes take effect immediately, so it may be set before or after
+# inheriting multilib-minimal, or even during execution of a phase function
+# (but not in parallelized multilib_<phase> functions; see below).
+#
+# By default, MULTILIB_PARALLEL_PHASES is empty.  Consuming eclasses could
+# override this default by setting a new default before the inherits clause.
+# For example,
+#
+# @CODE
+# : {MULTILIB_PARALLEL_PHASES:="src_configure"}
+# inherit multilib-minimal
+# @CODE
+#
+# would create a default behavior of parallel src_configure.  The question
+# of whether, in practice, eclasses ever should do so is potentially
+# complicated; you'll have to figure that out for yourself.
+#
+# Supported phase-function-names are: src_configure, src_compile, and src_test.
+# All other values are silently ignored.
+#
+# Note that parallel execution can lead to ugly -- and sometimes very, very ugly --
+# console output at build-time.  Consumers of multilib-minimal.eclass
+# may wish to consider activating less verbose build-time output modes, if they are
+# readily available, to mitigate this effect, although doing so goes against a
+# long-standing Gentoo tradition and might make debugging a hassle for bugzilla
+# and IRC participants, so use your best judgement.
+#
+# A handy per-ABI log is kept in ${T}/build-${ABI}.log by multibuild.eclass.
+# Debugging and bug-reporting may also benefit (unless your bug specifically
+# pertains to parallelization) from setting MAKEOPTS=-j1, which will prevent
+# parallelization entirely.
+
 # @ECLASS-VARIABLE: MULTILIB_INSECURE_INSTALL
 # @DEFAULT-UNSET
 # @DESCRIPTION:
@@ -161,7 +213,12 @@ multilib-minimal_src_configure() {
 	if declare -f multilib_src_configure_all > /dev/null ; then
 		multilib_src_configure_all
 	fi
-	multilib_foreach_abi multilib-minimal_abi_src_configure
+
+	if has src_configure ${MULTILIB_PARALLEL_PHASES} ; then
+		multilib_parallel_foreach_abi multilib-minimal_abi_src_configure
+	else
+		multilib_foreach_abi multilib-minimal_abi_src_configure
+	fi
 }
 
 multilib-minimal_src_compile() {
@@ -179,7 +236,12 @@ multilib-minimal_src_compile() {
 		popd >/dev/null || die
 	}
 
-	multilib_foreach_abi multilib-minimal_abi_src_compile
+	if has src_compile ${MULTILIB_PARALLEL_PHASES} ; then
+		multilib_parallel_foreach_abi multilib-minimal_abi_src_compile
+	else
+		multilib_foreach_abi multilib-minimal_abi_src_compile
+	fi
+
 	if declare -f multilib_src_compile_all > /dev/null ; then
 		multilib_src_compile_all
 	fi
@@ -200,7 +262,12 @@ multilib-minimal_src_test() {
 		popd >/dev/null || die
 	}
 
-	multilib_foreach_abi multilib-minimal_abi_src_test
+	if has src_test ${MULTILIB_PARALLEL_PHASES} ; then
+		multilib_parallel_foreach_abi multilib-minimal_abi_src_test
+	else
+		multilib_foreach_abi multilib-minimal_abi_src_test
+	fi
+
 	if declare -f multilib_src_test_all >/dev/null ; then
 		multilib_src_test_all
 	fi

Re: [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 221 bytes --]

Dnia 2013-12-11, o godz. 13:19:04
Greg Turner <gmt@malth.us> napisał(a):

> 

Very limited usefulness, a lot of extra complexity. We'd rather work on
making eclasses simpler.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization

[Greg Turner]

On Wed, Dec 11, 2013 at 2:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Very limited usefulness, a lot of extra complexity

Can't say I entirely agree on either point, but it wouldn't
meaningfully jam up my patch queues, which makes me much less inclined
to be attached to it.

Another reason, upon consideration, not to rush to merge this into
gx86, is that this could perhaps be implemented in multibuild, rather
than multilib-anything (i.e.: by allowing folks to set a variable that
would cause (only) the next multibuild_foreach_variant invocation to
route to multibuild_parallel_foreach_variant... but maybe that's not a
great idea since it would provide third parties with too easy a way to
accidentally break stuff like the header wrapping in
multilib-minimal_src_install).

-gmt

Re: [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization

[Greg Turner]

On Wed, Dec 11, 2013 at 2:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Very limited usefulness, a lot of extra complexity. We'd rather work on
> making eclasses simpler.

Sorry to beat a potentially dead horse, but many days later, I still
can't bring myself to agree with this, at least, not with respect to
src_compile.

The patch to the eclass is trivial and easily understood.  The
MULTILIB_PARALLEL_PHASES interface seems intuitive, just name the
phases you want parallel-ized in the string...  Where's the
complexity?

If end-users would rather not risk it, they can set MAKEOPTS=-j1.  If
ebuild authors suspect that their particular ebuilds will have
parallel build problems, they can simply do nothing, and their
multilib-minimal-based ebuild would be non-abi-parallel.  It all seems
pretty reasonable to me.

As for usefulness -- if someone wants to sit around and wait for three
dev-qt/qtwebkit compiles in a row to complete, one-at-a-time, then I
guess it's not useful to them.  Personally, it'd be useful to me.  I
can't imagine I'm the only person who feels that way.

-gmt