[gentoo-python] [distutils-r1] Improvements to parallel builds

[gentoo-python] [distutils-r1] Improvements to parallel builds

[Michał Górny]

A two quick improvements to parallel builds using distutils-r1.


1) use post-fork mode to avoid early output

In other words, you no longer see 'running X for Y' before it is
actually being run.


2) create split logs for each implementation

Aside to the usual soup on stdout, the eclass writes separate build logs
for each implementation to ${T}/build-${EPYTHON}.log.

TODO: replace tee with an awesome use of multitail to display logs
of currently running builds on split-screen.

[gentoo-python] [PATCH distutils-r1 1/4] Don't rely on phase function return code.

[Michał Górny]

It prohibits constructs like 'use x && y', and is not really necessary
when '|| die' should be used there anyway.
---
 gx86/eclass/distutils-r1.eclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index b1b3f90..9e37a56 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -380,9 +380,9 @@ distutils-r1_run_phase() {
 	fi
 
 	if [[ ${DISTUTILS_NO_PARALLEL_BUILD} ]]; then
-		"${@}" || die "${1} failed."
+		"${@}"
 	else
-		multijob_child_init "${@}" || die "${1} failed."
+		multijob_child_init "${@}"
 	fi
 
 	if [[ ${DISTUTILS_IN_SOURCE_BUILD} ]]; then
-- 
1.8.0

[gentoo-python] [PATCH distutils-r1 2/4] Use post-fork mode to avoid early output.

[Michał Górny]

In pre-fork mode, we first output 'running phase', then check for jobs.
Therefore, the output for (n+1)th phase goes too early. Using post-fork
mode fixes that.
---
 gx86/eclass/distutils-r1.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index 9e37a56..1006813 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -382,7 +382,7 @@ distutils-r1_run_phase() {
 	if [[ ${DISTUTILS_NO_PARALLEL_BUILD} ]]; then
 		"${@}"
 	else
-		multijob_child_init "${@}"
+		multijob_child_init --post "${@}"
 	fi
 
 	if [[ ${DISTUTILS_IN_SOURCE_BUILD} ]]; then
-- 
1.8.0

[gentoo-python] [PATCH distutils-r1 3/4] multiprocessing: use the expanded call form.

[Michał Górny]

The short one doesn't want to work with pipes, for some reason.
---
 gx86/eclass/distutils-r1.eclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index 1006813..8ba2aa8 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -382,7 +382,11 @@ distutils-r1_run_phase() {
 	if [[ ${DISTUTILS_NO_PARALLEL_BUILD} ]]; then
 		"${@}"
 	else
-		multijob_child_init --post "${@}"
+		(
+			multijob_child_init
+			"${@}"
+		) &
+		multijob_post_fork
 	fi
 
 	if [[ ${DISTUTILS_IN_SOURCE_BUILD} ]]; then
-- 
1.8.0

[gentoo-python] [PATCH distutils-r1 4/4] Create split logs when doing parallel builds.

[Michał Górny]

Therefore, making it possible to get a non-ugly build log for each
implementation used.
---
 gx86/eclass/distutils-r1.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index 8ba2aa8..a0c6f6b 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -384,7 +384,7 @@ distutils-r1_run_phase() {
 	else
 		(
 			multijob_child_init
-			"${@}"
+			"${@}" 2>&1 | tee -a "${T}/build-${EPYTHON}.log"
 		) &
 		multijob_post_fork
 	fi
-- 
1.8.0

[gentoo-python] Re: [distutils-r1] Improvements to parallel builds

[Mike Gilbert]

On Thu, Dec 6, 2012 at 5:54 AM, Michał Górny <mgorny@gentoo.org> wrote:
> A two quick improvements to parallel builds using distutils-r1.
>
>
> 1) use post-fork mode to avoid early output
>
> In other words, you no longer see 'running X for Y' before it is
> actually being run.
>

I assume you know what you are doing here; I'm not that familiar with
the multiprocessing eclass.

>
> 2) create split logs for each implementation
>
> Aside to the usual soup on stdout, the eclass writes separate build logs
> for each implementation to ${T}/build-${EPYTHON}.log.
>
> TODO: replace tee with an awesome use of multitail to display logs
> of currently running builds on split-screen.
>

The split logs could be handy for developers.

I had not considered the ugly bug reports that the multi-job build
will generate. If that becomes a major problem, we might need to
change the default to single-job, or produce some output to point
users to the split logs.

Re: [gentoo-python] Re: [distutils-r1] Improvements to parallel builds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

On Thu, 6 Dec 2012 12:28:37 -0500
Mike Gilbert <floppym@gentoo.org> wrote:

> On Thu, Dec 6, 2012 at 5:54 AM, Michał Górny <mgorny@gentoo.org> wrote:
> >
> > 2) create split logs for each implementation
> >
> > Aside to the usual soup on stdout, the eclass writes separate build logs
> > for each implementation to ${T}/build-${EPYTHON}.log.
> >
> > TODO: replace tee with an awesome use of multitail to display logs
> > of currently running builds on split-screen.
> >
> 
> The split logs could be handy for developers.
> 
> I had not considered the ugly bug reports that the multi-job build
> will generate. If that becomes a major problem, we might need to
> change the default to single-job, or produce some output to point
> users to the split logs.

It would be great if we had a more general solution here. I will ask
Zac if portage could point users to all *.log files in ${T}.

On the other hand, telling user to attach up to 8 logs feels a bit bad.
But I guess they'll just put them in a tarball.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]