[gentoo-python] distutils-r1: a bit of clean up + parallel builds

[gentoo-python] distutils-r1: a bit of clean up + parallel builds

[Michał Górny]

1) setup.py installs files to intermediate root (alike python.eclass).

This way anything we do on the installed files doesn't collide with
other merges potentially running in parallel. This also means that we
don't have to delay installing the wrapper till all setup.py invocations
have completed.

This is done directly in distutils-r1_python_install. The setup.py is
given a different --root, the renaming is done on intermediate image
and the image is quickly merged to the destination.

In order to perform the merge efficiently, I used:

	cp --archive --link --no-clobber

so that copy should preserve everything and use hard-links whenever
possible. --no-clobber is necessary to avoid error on colliding files
(cp doesn't want to overwrite when hardlinking).


2) the wrapper is installed in distutils-r1_python_install.

Previously, distutils-r1_python_install only renamed the installed
executables (because of distutils no-clobber behavior),
and distutils-r1_python_install_all installed the wrapper.

Now we can install both in the same function, since distutils installs
into intermediate images. Therefore, the wrapper being installed
in another intermediate image or even the real image won't collide.


3) the sub-phases are run in parallel.

Since distutils itself is unable to do parallel builds, building Python
packages with C extensions for multiple Python implementations can get
very slow. In order to circumvent that, we're using the multiprocessing
eclass to run sub-phases in parallel.

This means that with 4 implementations enabled and -j4, all four
implementations will be built at the same time. And if they have C
extensions, 4 source files will be built at the same time. This also
makes it possible to use distcc.

As stated in the last patch:

dev-python/lxml-3.0.1 for py2.6+2.7+3.2+3.3:

- non-parallel: 11 min 23 sec
- parallel: 7 min 49 sec (with a bit of swapping)
- parallel w/ distcc: 3 min 40 sec

main machine: Core2 2x1.6 GHz and almost 2 GiBs of RAM
distcc host: Athlon64 2x2 GHz and 3 GiBs of RAM

[gentoo-python] [PATCH distutils-r1 1/6] Make distutils-r1_rename_scripts private.

[Michał Górny]

---
 gx86/eclass/distutils-r1.eclass | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index fc9416b..ee7dc83 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -243,13 +243,14 @@ distutils-r1_python_test() {
 	:
 }
 
-# @FUNCTION: distutils-r1_rename_scripts
+# @FUNCTION: _distutils-r1_rename_scripts
+# @INTERNAL
 # @DESCRIPTION:
 # Renames installed Python scripts to be implementation-suffixed.
 # ${PYTHON} has to be set to the expected Python executable (which
 # hashbang will be grepped for), and ${EPYTHON} to the implementation
 # name (for new name).
-distutils-r1_rename_scripts() {
+_distutils-r1_rename_scripts() {
 	debug-print-function ${FUNCNAME} "${@}"
 
 	local f
@@ -274,7 +275,7 @@ distutils-r1_rename_scripts() {
 # @USAGE: [additional-args...]
 # @DESCRIPTION:
 # The default python_install(). Runs 'esetup.py install', appending
-# the optimization flags. Then calls distutils-r1_rename_scripts.
+# the optimization flags. Then calls _distutils-r1_rename_scripts.
 # Any parameters passed to this function will be passed to setup.py.
 distutils-r1_python_install() {
 	debug-print-function ${FUNCNAME} "${@}"
@@ -295,7 +296,7 @@ distutils-r1_python_install() {
 
 	esetup.py install "${flags[@]}" --root="${D}" "${@}"
 
-	distutils-r1_rename_scripts
+	_distutils-r1_rename_scripts
 }
 
 # @FUNCTION: distutils-r1_python_install_all
-- 
1.8.0

[gentoo-python] [PATCH distutils-r1 2/6] Rename all Python executables in distutils-r1_rename_scripts.

[Michał Górny]

Do not use hardcoded paths.
---
 gx86/eclass/distutils-r1.eclass | 20 +++++++++-----------
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index ee7dc83..0d8df87 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -255,20 +255,18 @@ _distutils-r1_rename_scripts() {
 
 	local f
 	# XXX: change this if we ever allow directories in bin/sbin
-	for f in "${D}"/{bin,sbin,usr/bin,usr/sbin}/*; do
-		if [[ -x ${f} ]]; then
-			debug-print "${FUNCNAME}: found executable at ${f#${D}/}"
+	while IFS= read -r -d '' f; do
+		debug-print "${FUNCNAME}: found executable at ${f#${D}/}"
 
-			if [[ "$(head -n 1 "${f}")" == '#!'*${EPYTHON}* ]]
-			then
-				debug-print "${FUNCNAME}: matching shebang: $(head -n 1 "${f}")"
+		if [[ "$(head -n 1 "${f}")" == '#!'*${EPYTHON}* ]]
+		then
+			debug-print "${FUNCNAME}: matching shebang: $(head -n 1 "${f}")"
 
-				local newf=${f}-${EPYTHON}
-				debug-print "${FUNCNAME}: renamed to ${newf#${D}/}"
-				mv "${f}" "${newf}" || die
-			fi
+			local newf=${f}-${EPYTHON}
+			debug-print "${FUNCNAME}: renamed to ${newf#${D}/}"
+			mv "${f}" "${newf}" || die
 		fi
-	done
+	done < <(find "${D}" -type f -executable -print0)
 }
 
 # @FUNCTION: distutils-r1_python_install
-- 
1.8.0

[gentoo-python] [PATCH distutils-r1 3/6] _distutils-r1_rename_scripts: require explicit path.

[Michał Górny]

---
 gx86/eclass/distutils-r1.eclass | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index 0d8df87..e1e89c6 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -244,15 +244,20 @@ distutils-r1_python_test() {
 }
 
 # @FUNCTION: _distutils-r1_rename_scripts
+# @USAGE: <path>
 # @INTERNAL
 # @DESCRIPTION:
 # Renames installed Python scripts to be implementation-suffixed.
-# ${PYTHON} has to be set to the expected Python executable (which
-# hashbang will be grepped for), and ${EPYTHON} to the implementation
-# name (for new name).
+# ${EPYTHON} needs to be set to the implementation name.
+#
+# All executable scripts having shebang referencing ${EPYTHON}
+# in given path will be renamed.
 _distutils-r1_rename_scripts() {
 	debug-print-function ${FUNCNAME} "${@}"
 
+	local path=${1}
+	[[ ${path} ]] || die "${FUNCNAME}: no path given"
+
 	local f
 	# XXX: change this if we ever allow directories in bin/sbin
 	while IFS= read -r -d '' f; do
@@ -266,14 +271,14 @@ _distutils-r1_rename_scripts() {
 			debug-print "${FUNCNAME}: renamed to ${newf#${D}/}"
 			mv "${f}" "${newf}" || die
 		fi
-	done < <(find "${D}" -type f -executable -print0)
+	done < <(find "${path}" -type f -executable -print0)
 }
 
 # @FUNCTION: distutils-r1_python_install
 # @USAGE: [additional-args...]
 # @DESCRIPTION:
 # The default python_install(). Runs 'esetup.py install', appending
-# the optimization flags. Then calls _distutils-r1_rename_scripts.
+# the optimization flags. Then renames the installed scripts.
 # Any parameters passed to this function will be passed to setup.py.
 distutils-r1_python_install() {
 	debug-print-function ${FUNCNAME} "${@}"
@@ -294,7 +299,7 @@ distutils-r1_python_install() {
 
 	esetup.py install "${flags[@]}" --root="${D}" "${@}"
 
-	_distutils-r1_rename_scripts
+	_distutils-r1_rename_scripts "${D}"
 }
 
 # @FUNCTION: distutils-r1_python_install_all
-- 
1.8.0

[gentoo-python] [PATCH distutils-r1 4/6] Install files to intermediate root and merge afterwards.

[Michał Górny]

This will allow us to install wrapper in distutils-r1_rename_scripts
without interfering with distutils' no-clobber behavior. In future it
will also make it possible to run install in parallel.
---
 gx86/eclass/distutils-r1.eclass | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index e1e89c6..fffa800 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -297,9 +297,14 @@ distutils-r1_python_install() {
 	local PYTHONDONTWRITEBYTECODE
 	export PYTHONDONTWRITEBYTECODE
 
-	esetup.py install "${flags[@]}" --root="${D}" "${@}"
+	local root=${D}/_${EPYTHON}
 
-	_distutils-r1_rename_scripts "${D}"
+	esetup.py install "${flags[@]}" --root="${root}" "${@}"
+	_distutils-r1_rename_scripts "${root}"
+
+	# merge
+	cp -a -l -n "${root}"/* "${D}"/ || die "Merging ${EPYTHON} image failed."
+	rm -rf "${root}"
 }
 
 # @FUNCTION: distutils-r1_python_install_all
-- 
1.8.0

[gentoo-python] [PATCH distutils-r1 5/6] Install the wrapper symlink in _distutils-r1_rename_scripts.

[Michał Górny]

This is cleaner than delaying it to install_all().
---
 gx86/eclass/distutils-r1.eclass | 23 +++++------------------
 1 file changed, 5 insertions(+), 18 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index fffa800..c21ab30 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -268,8 +268,11 @@ _distutils-r1_rename_scripts() {
 			debug-print "${FUNCNAME}: matching shebang: $(head -n 1 "${f}")"
 
 			local newf=${f}-${EPYTHON}
-			debug-print "${FUNCNAME}: renamed to ${newf#${D}/}"
+			debug-print "${FUNCNAME}: renaming to ${newf#${D}/}"
 			mv "${f}" "${newf}" || die
+
+			debug-print "${FUNCNAME}: installing wrapper at ${f#${D}/}"
+			_python_ln_rel "${path}"/usr/bin/python-exec "${f}" || die
 		fi
 	done < <(find "${path}" -type f -executable -print0)
 }
@@ -309,9 +312,7 @@ distutils-r1_python_install() {
 
 # @FUNCTION: distutils-r1_python_install_all
 # @DESCRIPTION:
-# The default python_install_all(). It symlinks wrappers
-# for the implementation-suffixed executables and installs
-# documentation.
+# The default python_install_all(). It installs the documentation.
 distutils-r1_python_install_all() {
 	debug-print-function ${FUNCNAME} "${@}"
 
@@ -331,20 +332,6 @@ distutils-r1_python_install_all() {
 	if declare -p HTML_DOCS &>/dev/null; then
 		dohtml -r "${HTML_DOCS[@]}" || die "dohtml failed"
 	fi
-
-	# note: keep in sync with ...rename_scripts()
-	# also, we assume that each script is installed for all impls
-	local EPYTHON
-	python_export_best EPYTHON
-
-	local f
-	while IFS= read -r -d '' f; do
-		debug-print "${FUNCNAME}: found executable at ${f#${D}/}"
-
-		local wrapf=${f%-${EPYTHON}}
-
-		_python_ln_rel "${ED}"/usr/bin/python-exec "${wrapf}" || die
-	done < <(find "${D}" -type f -executable -name "*-${EPYTHON}" -print0)
 }
 
 # @FUNCTION: distutils-r1_run_phase
-- 
1.8.0

[gentoo-python] [PATCH distutils-r1 6/6] Run sub-phases in parallel.

[Michał Górny]

This circumvents the distutils inability to build sources in parallel.

On 2-core CPU, dev-python/lxml-3.0.1 for py2.6+2.7+3.2+3.3:

- non-parallel: 11 min 23 sec

- parallel: 7 min 49 sec (with a bit of swapping)

- parallel w/ distcc: 3 min 40 sec
---
 gx86/eclass/distutils-r1.eclass | 41 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index c21ab30..5e962a4 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -57,7 +57,7 @@ esac
 
 if [[ ! ${_DISTUTILS_R1} ]]; then
 
-inherit eutils python-r1
+inherit eutils multiprocessing python-r1
 
 fi
 
@@ -130,6 +130,29 @@ DEPEND=${PYTHON_DEPS}
 # 'build --build-base ${BUILD_DIR}' to enforce keeping & using built
 # files in the specific root.
 
+# @ECLASS-VARIABLE: DISTUTILS_NO_PARALLEL_BUILD
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# If set to a non-null value, the parallel build feature will
+# be disabled.
+#
+# When parallel builds are used, the implementation-specific sub-phases
+# for selected Python implementation will be run in parallel. This will
+# increase build efficiency with distutils which does not do parallel
+# builds.
+#
+# This variable can be used to disable the afore-mentioned feature
+# in case it causes issues with the package.
+
+#
+# If in-source builds are used, the eclass will create a copy of package
+# sources for each Python implementation in python_prepare_all(),
+# and work on that copy afterwards.
+#
+# If out-of-source builds are used, the eclass will instead work
+# on the sources directly, prepending setup.py arguments with
+# 'build --build-base ${BUILD_DIR}' to enforce keeping & using built
+# files in the specific root.
 # @ECLASS-VARIABLE: mydistutilsargs
 # @DEFAULT_UNSET
 # @DESCRIPTION:
@@ -356,7 +379,11 @@ distutils-r1_run_phase() {
 		export PYTHONPATH
 	fi
 
-	"${@}" || die "${1} failed."
+	if [[ ${DISTUTILS_NO_PARALLEL_BUILD} ]]; then
+		"${@}" || die "${1} failed."
+	else
+		multijob_child_init "${@}" || die "${1} failed."
+	fi
 
 	if [[ ${DISTUTILS_IN_SOURCE_BUILD} ]]; then
 		popd &>/dev/null || die
@@ -373,19 +400,23 @@ distutils-r1_src_prepare() {
 		distutils-r1_python_prepare_all
 	fi
 
+	multijob_init
 	if declare -f python_prepare >/dev/null; then
 		python_foreach_impl distutils-r1_run_phase python_prepare
 	else
 		python_foreach_impl distutils-r1_run_phase distutils-r1_python_prepare
 	fi
+	multijob_finish
 }
 
 distutils-r1_src_configure() {
+	multijob_init
 	if declare -f python_configure >/dev/null; then
 		python_foreach_impl distutils-r1_run_phase python_configure
 	else
 		python_foreach_impl distutils-r1_run_phase distutils-r1_python_configure
 	fi
+	multijob_finish
 
 	if declare -f python_configure_all >/dev/null; then
 		python_configure_all
@@ -395,11 +426,13 @@ distutils-r1_src_configure() {
 distutils-r1_src_compile() {
 	debug-print-function ${FUNCNAME} "${@}"
 
+	multijob_init
 	if declare -f python_compile >/dev/null; then
 		python_foreach_impl distutils-r1_run_phase python_compile
 	else
 		python_foreach_impl distutils-r1_run_phase distutils-r1_python_compile
 	fi
+	multijob_finish
 
 	if declare -f python_compile_all >/dev/null; then
 		python_compile_all
@@ -409,11 +442,13 @@ distutils-r1_src_compile() {
 distutils-r1_src_test() {
 	debug-print-function ${FUNCNAME} "${@}"
 
+	multijob_init
 	if declare -f python_test >/dev/null; then
 		python_foreach_impl distutils-r1_run_phase python_test
 	else
 		python_foreach_impl distutils-r1_run_phase distutils-r1_python_test
 	fi
+	multijob_finish
 
 	if declare -f python_test_all >/dev/null; then
 		python_test_all
@@ -423,11 +458,13 @@ distutils-r1_src_test() {
 distutils-r1_src_install() {
 	debug-print-function ${FUNCNAME} "${@}"
 
+	multijob_init
 	if declare -f python_install >/dev/null; then
 		python_foreach_impl distutils-r1_run_phase python_install
 	else
 		python_foreach_impl distutils-r1_run_phase distutils-r1_python_install
 	fi
+	multijob_finish
 
 	if declare -f python_install_all >/dev/null; then
 		python_install_all
-- 
1.8.0

[gentoo-python] Re: distutils-r1: a bit of clean up + parallel builds

[Mike Gilbert]

On Thu, Nov 29, 2012 at 6:31 AM, Michał Górny <mgorny@gentoo.org> wrote:
> 1) setup.py installs files to intermediate root (alike python.eclass).
>
> This way anything we do on the installed files doesn't collide with
> other merges potentially running in parallel. This also means that we
> don't have to delay installing the wrapper till all setup.py invocations
> have completed.
>
> This is done directly in distutils-r1_python_install. The setup.py is
> given a different --root, the renaming is done on intermediate image
> and the image is quickly merged to the destination.
>
> In order to perform the merge efficiently, I used:
>
>         cp --archive --link --no-clobber
>
> so that copy should preserve everything and use hard-links whenever
> possible. --no-clobber is necessary to avoid error on colliding files
> (cp doesn't want to overwrite when hardlinking).
>
>
> 2) the wrapper is installed in distutils-r1_python_install.
>
> Previously, distutils-r1_python_install only renamed the installed
> executables (because of distutils no-clobber behavior),
> and distutils-r1_python_install_all installed the wrapper.
>
> Now we can install both in the same function, since distutils installs
> into intermediate images. Therefore, the wrapper being installed
> in another intermediate image or even the real image won't collide.
>
>
> 3) the sub-phases are run in parallel.
>
> Since distutils itself is unable to do parallel builds, building Python
> packages with C extensions for multiple Python implementations can get
> very slow. In order to circumvent that, we're using the multiprocessing
> eclass to run sub-phases in parallel.
>
> This means that with 4 implementations enabled and -j4, all four
> implementations will be built at the same time. And if they have C
> extensions, 4 source files will be built at the same time. This also
> makes it possible to use distcc.
>
> As stated in the last patch:
>
> dev-python/lxml-3.0.1 for py2.6+2.7+3.2+3.3:
>
> - non-parallel: 11 min 23 sec
> - parallel: 7 min 49 sec (with a bit of swapping)
> - parallel w/ distcc: 3 min 40 sec
>
> main machine: Core2 2x1.6 GHz and almost 2 GiBs of RAM
> distcc host: Athlon64 2x2 GHz and 3 GiBs of RAM
>

I was just thinking to myself last night that parallel builds/installs
would be nice. You must be psychic.

My thought was to add a flag/variable for python_foreach_impl, but
your method should work just as well.

Building the temporary image in sub-directories of ${D} feels a bit
strange, but I guess it works. I think distutils.eclass used ${T}?

Anyway, +1 from me.

Re: [gentoo-python] Re: distutils-r1: a bit of clean up + parallel builds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

On Thu, 29 Nov 2012 11:29:35 -0500
Mike Gilbert <floppym@gentoo.org> wrote:

> I was just thinking to myself last night that parallel builds/installs
> would be nice. You must be psychic.

Well, to be honest I was attempting of doing this earlier. But hit
the problem that parallel install had a race condition between setup.py
and renaming scripts. And I decided once then that this is not
important to make scary intermediate image magic.

Today I have thought it over again and noticed that the intermediate
images can be done simpler than python.eclass does it.

> My thought was to add a flag/variable for python_foreach_impl, but
> your method should work just as well.

I was thinking a bit about that one too. However, I don't like adding
option parsing to random commands, and I don't see much use of parallel
runs outside of distutils. I believe that those calls mostly do simple
things (like installing a single file) or call a build system which
supports parallel builds itself.

> Building the temporary image in sub-directories of ${D} feels a bit
> strange, but I guess it works. I think distutils.eclass used ${T}?

Yes, it used ${T}. I assumed that ${D} is closer to the final
directory, therefore it is less likely for any issues to occur.
Theoretically a PM can put ${T} on some kind of tmpfs while ${D}
on the root filesystem to make merging faster.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]