* [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization
@ 2013-12-11 21:19 Greg Turner
2013-12-11 22:41 ` Michał Górny
0 siblings, 1 reply; 4+ messages in thread
From: Greg Turner @ 2013-12-11 21:19 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1 bytes --]
[-- Attachment #2: 005-MULTILIB_PARALLEL_PHASES.patch --]
[-- Type: text/x-patch, Size: 4866 bytes --]
This patch adds a new frob, MULTILIB_PARALLEL_PHASES, to
multlib-minimal.eclass, which implements eclass-consumer-selectable
parallelization of src_configure, src_compile, and src_test.
By default, all parallelization is deactivated, which represents
a change from the previous gentoo-x86 behavior (which was
parallelizing src_configure automatically).
--- 004-multilib-phase-all/multilib-minimal.eclass 2013-12-03 02:54:40.045335905 -0800
+++ 005-MULTILIB_PARALLEL_PHASES/multilib-minimal.eclass 2013-12-03 02:59:33.687448429 -0800
@@ -45,8 +45,8 @@
# for more information.
#
# Another snag you may encounter is that, in many circumstances, it is possible
-# for variables to flow between multilib_<phase> functions. This is because
-# no subshell is utilized.
+# for variables to flow between multilib_<phase> functions. This is because,
+# unless you parallelize the phase in question, no subshell is utilized.
# Extreme care must sometimes be taken to ensure that different ABI's don't
# trample each other's variables.
#
@@ -122,6 +122,58 @@ case ${EAPI:-0} in
*) die "EAPI=${EAPI} is not supported" ;;
esac
+# @ECLASS-VARIABLE: MULTILIB_PARALLEL_PHASES
+# @DEFAULT-UNSET
+# @DESCRIPTION:
+# multilib-minimal.eclass consumers may set this to a string containing
+# a space-separated list of phase-function names, like so:
+#
+# @CODE@
+# MULTILIB_PARALLEL_PHASES="src_configure src_test"
+# @CODE@
+#
+# For each (supported) phase-function name in the list, the corresponding
+# multilib-minimal_<phase> phase function will execute in parallel, in
+# the sense that, within limits (set by multiprocessing.eclass) the
+# multilib_<phase> function invocations (or, if none is present, the
+# built-in default implementation) for each ABI will run simultaneously.
+#
+# Any phase-function-names not specified are executed serially, with the
+# native ABI coming last.
+#
+# It is OK to add extra "bonus" whitespace or duplicated values.
+# Changes take effect immediately, so it may be set before or after
+# inheriting multilib-minimal, or even during execution of a phase function
+# (but not in parallelized multilib_<phase> functions; see below).
+#
+# By default, MULTILIB_PARALLEL_PHASES is empty. Consuming eclasses could
+# override this default by setting a new default before the inherits clause.
+# For example,
+#
+# @CODE
+# : {MULTILIB_PARALLEL_PHASES:="src_configure"}
+# inherit multilib-minimal
+# @CODE
+#
+# would create a default behavior of parallel src_configure. The question
+# of whether, in practice, eclasses ever should do so is potentially
+# complicated; you'll have to figure that out for yourself.
+#
+# Supported phase-function-names are: src_configure, src_compile, and src_test.
+# All other values are silently ignored.
+#
+# Note that parallel execution can lead to ugly -- and sometimes very, very ugly --
+# console output at build-time. Consumers of multilib-minimal.eclass
+# may wish to consider activating less verbose build-time output modes, if they are
+# readily available, to mitigate this effect, although doing so goes against a
+# long-standing Gentoo tradition and might make debugging a hassle for bugzilla
+# and IRC participants, so use your best judgement.
+#
+# A handy per-ABI log is kept in ${T}/build-${ABI}.log by multibuild.eclass.
+# Debugging and bug-reporting may also benefit (unless your bug specifically
+# pertains to parallelization) from setting MAKEOPTS=-j1, which will prevent
+# parallelization entirely.
+
# @ECLASS-VARIABLE: MULTILIB_INSECURE_INSTALL
# @DEFAULT-UNSET
# @DESCRIPTION:
@@ -161,7 +213,12 @@ multilib-minimal_src_configure() {
if declare -f multilib_src_configure_all > /dev/null ; then
multilib_src_configure_all
fi
- multilib_foreach_abi multilib-minimal_abi_src_configure
+
+ if has src_configure ${MULTILIB_PARALLEL_PHASES} ; then
+ multilib_parallel_foreach_abi multilib-minimal_abi_src_configure
+ else
+ multilib_foreach_abi multilib-minimal_abi_src_configure
+ fi
}
multilib-minimal_src_compile() {
@@ -179,7 +236,12 @@ multilib-minimal_src_compile() {
popd >/dev/null || die
}
- multilib_foreach_abi multilib-minimal_abi_src_compile
+ if has src_compile ${MULTILIB_PARALLEL_PHASES} ; then
+ multilib_parallel_foreach_abi multilib-minimal_abi_src_compile
+ else
+ multilib_foreach_abi multilib-minimal_abi_src_compile
+ fi
+
if declare -f multilib_src_compile_all > /dev/null ; then
multilib_src_compile_all
fi
@@ -200,7 +262,12 @@ multilib-minimal_src_test() {
popd >/dev/null || die
}
- multilib_foreach_abi multilib-minimal_abi_src_test
+ if has src_test ${MULTILIB_PARALLEL_PHASES} ; then
+ multilib_parallel_foreach_abi multilib-minimal_abi_src_test
+ else
+ multilib_foreach_abi multilib-minimal_abi_src_test
+ fi
+
if declare -f multilib_src_test_all >/dev/null ; then
multilib_src_test_all
fi
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization
2013-12-11 21:19 [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization Greg Turner
@ 2013-12-11 22:41 ` Michał Górny
2013-12-12 0:24 ` Greg Turner
2013-12-31 2:55 ` Greg Turner
0 siblings, 2 replies; 4+ messages in thread
From: Michał Górny @ 2013-12-11 22:41 UTC (permalink / raw
To: gentoo-dev; +Cc: gmt
[-- Attachment #1: Type: text/plain, Size: 221 bytes --]
Dnia 2013-12-11, o godz. 13:19:04
Greg Turner <gmt@malth.us> napisał(a):
>
Very limited usefulness, a lot of extra complexity. We'd rather work on
making eclasses simpler.
--
Best regards,
Michał Górny
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization
2013-12-11 22:41 ` Michał Górny
@ 2013-12-12 0:24 ` Greg Turner
2013-12-31 2:55 ` Greg Turner
1 sibling, 0 replies; 4+ messages in thread
From: Greg Turner @ 2013-12-12 0:24 UTC (permalink / raw
To: Michał Górny, gentoo-dev
On Wed, Dec 11, 2013 at 2:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Very limited usefulness, a lot of extra complexity
Can't say I entirely agree on either point, but it wouldn't
meaningfully jam up my patch queues, which makes me much less inclined
to be attached to it.
Another reason, upon consideration, not to rush to merge this into
gx86, is that this could perhaps be implemented in multibuild, rather
than multilib-anything (i.e.: by allowing folks to set a variable that
would cause (only) the next multibuild_foreach_variant invocation to
route to multibuild_parallel_foreach_variant... but maybe that's not a
great idea since it would provide third parties with too easy a way to
accidentally break stuff like the header wrapping in
multilib-minimal_src_install).
-gmt
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization
2013-12-11 22:41 ` Michał Górny
2013-12-12 0:24 ` Greg Turner
@ 2013-12-31 2:55 ` Greg Turner
1 sibling, 0 replies; 4+ messages in thread
From: Greg Turner @ 2013-12-31 2:55 UTC (permalink / raw
To: multilib, gentoo-dev
On Wed, Dec 11, 2013 at 2:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Very limited usefulness, a lot of extra complexity. We'd rather work on
> making eclasses simpler.
Sorry to beat a potentially dead horse, but many days later, I still
can't bring myself to agree with this, at least, not with respect to
src_compile.
The patch to the eclass is trivial and easily understood. The
MULTILIB_PARALLEL_PHASES interface seems intuitive, just name the
phases you want parallel-ized in the string... Where's the
complexity?
If end-users would rather not risk it, they can set MAKEOPTS=-j1. If
ebuild authors suspect that their particular ebuilds will have
parallel build problems, they can simply do nothing, and their
multilib-minimal-based ebuild would be non-abi-parallel. It all seems
pretty reasonable to me.
As for usefulness -- if someone wants to sit around and wait for three
dev-qt/qtwebkit compiles in a row to complete, one-at-a-time, then I
guess it's not useful to them. Personally, it'd be useful to me. I
can't imagine I'm the only person who feels that way.
-gmt
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2013-12-31 2:55 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-11 21:19 [gentoo-dev] some multilib-minimal enhancements [5/6]: add frob to control phase parallelization Greg Turner
2013-12-11 22:41 ` Michał Górny
2013-12-12 0:24 ` Greg Turner
2013-12-31 2:55 ` Greg Turner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox