[gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib & multibuild

[gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib & multibuild

[Michał Górny]

Hello, everyone.

Following a similar change in distutils-r1, I would like to remove
the parallel run support in multilib eclasses, and effectively from
multibuild completely.

The goal is to make things simpler, and follow PMS properly. Right now
the parallel runs imply running commands in a subshell. This has three
important implications:

1. variable exports and environment changes inside the implicit subshell
don't affect the outer scope -- developers are sometimes confused by
that,

2. parallel runs can collide if temporary files aren't named uniquely --
think of ffmpeg, waf,

3. 'die' called in a subshell is unsupported by the PMS, and didn't make
it into EAPI 6.

The side advantages are that we can get rid of multiprocessing inherit
and locking in multibuild_merge_root().

Possible issues: if people were relying on commands inside parallel not
to affect the outer environment (i.e. doing 'cd' and expecting the next
call to start in the initial location, or exporting variables and
expecting them to disappear), the ebuilds will fail randomly. However,
that's quite unlikely.

Your thoughts?

--
Best regards,
Michał Górny

[gentoo-dev] [PATCH 1/3] multilib-minimal: do not run src_configure() in parallel

[Michał Górny]

---
 eclass/multilib-minimal.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/multilib-minimal.eclass b/eclass/multilib-minimal.eclass
index 4e1843e..3f6cfbd 100644
--- a/eclass/multilib-minimal.eclass
+++ b/eclass/multilib-minimal.eclass
@@ -51,7 +51,7 @@ multilib-minimal_src_configure() {
 		popd >/dev/null || die
 	}
 
-	multilib_parallel_foreach_abi multilib-minimal_abi_src_configure
+	multilib_foreach_abi multilib-minimal_abi_src_configure
 }
 
 multilib-minimal_src_compile() {
-- 
2.2.0

[gentoo-dev] [PATCH 2/3] multilib-build: unparallelize multilib_parallel_foreach_abi

[Michał Górny]

---
 eclass/multilib-build.eclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/eclass/multilib-build.eclass b/eclass/multilib-build.eclass
index 2e35d47..b0a4ea1 100644
--- a/eclass/multilib-build.eclass
+++ b/eclass/multilib-build.eclass
@@ -223,18 +223,18 @@ multilib_foreach_abi() {
 # @DESCRIPTION:
 # If multilib support is enabled, sets the toolchain up for each
 # supported ABI along with the ABI variable and correct BUILD_DIR,
-# and runs the given commands with them. The commands are run
-# in parallel with number of jobs being determined from MAKEOPTS.
+# and runs the given commands with them.
 #
 # If multilib support is disabled, it just runs the commands. No setup
 # is done.
 #
-# Useful for running configure scripts.
+# This function used to run multiple commands in parallel. Now it's just
+# a deprecated alias to multilib_foreach_abi.
 multilib_parallel_foreach_abi() {
 	debug-print-function ${FUNCNAME} "${@}"
 
 	local MULTIBUILD_VARIANTS=( $(multilib_get_enabled_abi_pairs) )
-	multibuild_parallel_foreach_variant _multilib_multibuild_wrapper "${@}"
+	multibuild_foreach_variant _multilib_multibuild_wrapper "${@}"
 }
 
 # @FUNCTION: multilib_for_best_abi
-- 
2.2.0

[gentoo-dev] [PATCH 3/3] multibuild: remove parallel run support

[Michał Górny]

---
 eclass/multibuild.eclass | 48 +++---------------------------------------------
 1 file changed, 3 insertions(+), 45 deletions(-)

diff --git a/eclass/multibuild.eclass b/eclass/multibuild.eclass
index 03e6280..bb17421 100644
--- a/eclass/multibuild.eclass
+++ b/eclass/multibuild.eclass
@@ -26,8 +26,6 @@ esac
 
 if [[ ! ${_MULTIBUILD} ]]; then
 
-inherit multiprocessing
-
 # @ECLASS-VARIABLE: MULTIBUILD_VARIANTS
 # @DESCRIPTION:
 # An array specifying all enabled variants which multibuild_foreach*
@@ -138,8 +136,8 @@ multibuild_foreach_variant() {
 # @USAGE: [<argv>...]
 # @DESCRIPTION:
 # Run the passed command repeatedly for each of the enabled package
-# variants alike multibuild_foreach_variant. Multiple invocations of the command
-# will be performed in parallel, up to MULTIBUILD_JOBS tasks.
+# variants. This used to run the commands in parallel but now it's
+# just a deprecated alias to multibuild_foreach_variant.
 #
 # The function returns 0 if all commands return 0, or the first non-zero
 # exit status otherwise. However, it performs all the invocations
@@ -148,31 +146,7 @@ multibuild_foreach_variant() {
 multibuild_parallel_foreach_variant() {
 	debug-print-function ${FUNCNAME} "${@}"
 
-	local ret lret
-
-	_multibuild_parallel() {
-		(
-			multijob_child_init
-			"${@}"
-		) &
-		multijob_post_fork
-	}
-
-	local opts
-	if [[ ${MULTIBUILD_JOBS} ]]; then
-		opts=-j${MULTIBUILD_JOBS}
-	else
-		opts=${MAKEOPTS}
-	fi
-
-	multijob_init "${opts}"
-	multibuild_foreach_variant _multibuild_parallel "${@}"
-	ret=${?}
-	multijob_finish
-	lret=${?}
-
-	[[ ${ret} -eq 0 ]] && ret=${lret}
-	return ${ret}
+	multibuild_foreach_variant "${@}"
 }
 
 # @FUNCTION: multibuild_for_best_variant
@@ -252,25 +226,12 @@ run_in_build_dir() {
 # Merge the directory tree (fake root) from <src-root> to <dest-root>
 # (the real root). Both directories have to be real, absolute paths
 # (i.e. including ${D}). Source root will be removed.
-#
-# This functions uses locking to support merging during parallel
-# installs.
 multibuild_merge_root() {
 	local src=${1}
 	local dest=${2}
 
-	local lockfile=${T}/.multibuild_merge_lock
-	local lockfile_l=${lockfile}.${BASHPID}
 	local ret
 
-	# Lock the install tree for merge. The touch+ln method ensures race
-	# condition-free locking with maximum portability.
-	touch "${lockfile_l}" || die
-	until ln "${lockfile_l}" "${lockfile}" &>/dev/null; do
-		sleep 1
-	done
-	rm "${lockfile_l}" || die
-
 	if use userland_BSD; then
 		# Most of BSD variants fail to copy broken symlinks, #447370
 		# also, they do not support --version
@@ -297,9 +258,6 @@ multibuild_merge_root() {
 		ret=${?}
 	fi
 
-	# Remove the lock.
-	rm "${lockfile}" || die
-
 	if [[ ${ret} -ne 0 ]]; then
 		die "${MULTIBUILD_VARIANT:-(unknown)}: merging image failed."
 	fi
-- 
2.2.0

Re: [gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib & multibuild

[Alexis Ballier]

On Thu, 11 Dec 2014 11:36:29 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Hello, everyone.
> 
> Following a similar change in distutils-r1, I would like to remove
> the parallel run support in multilib eclasses, and effectively from
> multibuild completely.
[...]
> Your thoughts?

+1

I hate parallel builds as I can't follow build execution properly; and,
on a different level, there's emerge --jobs already.

Alexis.

Re: [gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib & multibuild

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1528 bytes --]

Dnia 2014-12-11, o godz. 11:36:29
Michał Górny <mgorny@gentoo.org> napisał(a):

> Hello, everyone.
> 
> Following a similar change in distutils-r1, I would like to remove
> the parallel run support in multilib eclasses, and effectively from
> multibuild completely.
> 
> The goal is to make things simpler, and follow PMS properly. Right now
> the parallel runs imply running commands in a subshell. This has three
> important implications:
> 
> 1. variable exports and environment changes inside the implicit subshell
> don't affect the outer scope -- developers are sometimes confused by
> that,
> 
> 2. parallel runs can collide if temporary files aren't named uniquely --
> think of ffmpeg, waf,
> 
> 3. 'die' called in a subshell is unsupported by the PMS, and didn't make
> it into EAPI 6.
> 
> The side advantages are that we can get rid of multiprocessing inherit
> and locking in multibuild_merge_root().
> 
> Possible issues: if people were relying on commands inside parallel not
> to affect the outer environment (i.e. doing 'cd' and expecting the next
> call to start in the initial location, or exporting variables and
> expecting them to disappear), the ebuilds will fail randomly. However,
> that's quite unlikely.
> 
> Your thoughts?

And committed.

Not sure if we want to keep split build logs now. Their main purpose
was making around parallel runs readable. However, they still allow you
to easily trace build for one variant.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib & multibuild

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 320 bytes --]

>>>>> On Sat, 13 Dec 2014, Michał Górny wrote:

> Not sure if we want to keep split build logs now. Their main purpose
> was making around parallel runs readable. However, they still allow
> you to easily trace build for one variant.

Kill them. Splitting build.log is trivial if things are sequential.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib & multibuild

[Diamond]

On Sat, 13 Dec 2014 09:44:21 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 2014-12-11, o godz. 11:36:29
> Michał Górny <mgorny@gentoo.org> napisał(a):
> 
> > Hello, everyone.
> > 
> > Following a similar change in distutils-r1, I would like to remove
> > the parallel run support in multilib eclasses, and effectively from
> > multibuild completely.
> > 

What is parallel run support? Will it affect (the speed of) parallel
emerging of packages? 

Re: [gentoo-dev] [PATCH 0/3] Remove parallel run support from multilib & multibuild

[Zac Medico]

On 12/13/2014 10:10 AM, Diamond wrote:
> On Sat, 13 Dec 2014 09:44:21 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
>> Dnia 2014-12-11, o godz. 11:36:29
>> Michał Górny <mgorny@gentoo.org> napisał(a):
>>
>>> Hello, everyone.
>>>
>>> Following a similar change in distutils-r1, I would like to remove
>>> the parallel run support in multilib eclasses, and effectively from
>>> multibuild completely.
>>>
> 
> What is parallel run support?

It uses multiprocessing.eclass, which parses the make -j setting from
the MAKEOPTS variable, and uses that setting to internally parallelize
the build (as make would if we were using make for that build).

> Will it affect (the speed of) parallel emerging of packages? 

The effect will be to eliminate the internal parallelization of some
builds that were internally parallelized via multiprocessing.eclass
usage. This will reduce the load incurred by those builds, which reduces
the global load average, allowing emerge to spawn more concurrent builds
when using emerge --jobs with the --load-average option.
-- 
Thanks,
Zac