[gentoo-dev] [PATCH] distutils-r1.eclass: Enable parallel builds in py3.5+

[gentoo-dev] [PATCH] distutils-r1.eclass: Enable parallel builds in py3.5+

[Michał Górny]

Python 3.5+ introduces parallel build support in distutils.  Take
advantage of that by passing appropriate -j option.  Since distutils
does not support an equivalent of --load-average, default to the number
of CPUs+1 when unspecified.
---
 eclass/distutils-r1.eclass | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
index 975383acc09b..4d8aa3ca6677 100644
--- a/eclass/distutils-r1.eclass
+++ b/eclass/distutils-r1.eclass
@@ -80,10 +80,10 @@ if [[ ! ${_DISTUTILS_R1} ]]; then
 
 [[ ${EAPI} == [45] ]] && inherit eutils
 [[ ${EAPI} == [56] ]] && inherit xdg-utils
-inherit toolchain-funcs
+inherit multiprocessing toolchain-funcs
 
 if [[ ! ${DISTUTILS_SINGLE_IMPL} ]]; then
-	inherit multiprocessing python-r1
+	inherit python-r1
 else
 	inherit python-single-r1
 fi
@@ -454,7 +454,18 @@ distutils-r1_python_compile() {
 
 	_distutils-r1_copy_egg_info
 
-	esetup.py build "${@}"
+	local build_args=()
+	# distutils is parallel-capable since py3.5
+	if python_is_python3 && [[ ${EPYTHON} != python3.4 ]]; then
+		local jobs=$(makeopts_jobs "${MAKEOPTS}" INF)
+		if [[ ${jobs} == INF ]]; then
+			local nproc=$(get_nproc)
+			jobs=$(( nproc + 1 ))
+		fi
+		build_args+=( -j "${jobs}" )
+	fi
+
+	esetup.py build "${build_args[@]}" "${@}"
 }
 
 # @FUNCTION: _distutils-r1_wrap_scripts
-- 
2.18.0

Re: [gentoo-dev] [PATCH] distutils-r1.eclass: Enable parallel builds in py3.5+

[Mike Gilbert]

On Tue, Jul 17, 2018 at 4:41 AM, Michał Górny <mgorny@gentoo.org> wrote:
> Python 3.5+ introduces parallel build support in distutils.  Take
> advantage of that by passing appropriate -j option.  Since distutils
> does not support an equivalent of --load-average, default to the number
> of CPUs+1 when unspecified.

How do we disable this in individual ebuilds when we inevitably run
into problematic packages?

Re: [gentoo-dev] [PATCH] distutils-r1.eclass: Enable parallel builds in py3.5+

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 670 bytes --]

W dniu wto, 17.07.2018 o godzinie 10∶40 -0400, użytkownik Mike Gilbert
napisał:
> On Tue, Jul 17, 2018 at 4:41 AM, Michał Górny <mgorny@gentoo.org> wrote:
> > Python 3.5+ introduces parallel build support in distutils.  Take
> > advantage of that by passing appropriate -j option.  Since distutils
> > does not support an equivalent of --load-average, default to the number
> > of CPUs+1 when unspecified.
> 
> How do we disable this in individual ebuilds when we inevitably run
> into problematic packages?

My guess would be, same as emake:

  distutils-r1_python_compile -j1

(I haven't tested it though)

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [PATCH] distutils-r1.eclass: Enable parallel builds in py3.5+

[Mike Gilbert]

On Tue, Jul 17, 2018 at 10:48 AM, Michał Górny <mgorny@gentoo.org> wrote:
> W dniu wto, 17.07.2018 o godzinie 10∶40 -0400, użytkownik Mike Gilbert
> napisał:
>> On Tue, Jul 17, 2018 at 4:41 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> > Python 3.5+ introduces parallel build support in distutils.  Take
>> > advantage of that by passing appropriate -j option.  Since distutils
>> > does not support an equivalent of --load-average, default to the number
>> > of CPUs+1 when unspecified.
>>
>> How do we disable this in individual ebuilds when we inevitably run
>> into problematic packages?
>
> My guess would be, same as emake:
>
>   distutils-r1_python_compile -j1
>
> (I haven't tested it though)

This will affect stable ebuilds that have python3_5 or python3_6
enabled. It would be a good idea to test it on a reasonable sample of
ebuilds first.

Alternatively, we could enable it for python3_7 first and see what
kind of fallout that produces.

Re: [gentoo-dev] [PATCH] distutils-r1.eclass: Enable parallel builds in py3.5+

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1312 bytes --]

W dniu czw, 19.07.2018 o godzinie 10∶06 -0400, użytkownik Mike Gilbert
napisał:
> On Tue, Jul 17, 2018 at 10:48 AM, Michał Górny <mgorny@gentoo.org> wrote:
> > W dniu wto, 17.07.2018 o godzinie 10∶40 -0400, użytkownik Mike Gilbert
> > napisał:
> > > On Tue, Jul 17, 2018 at 4:41 AM, Michał Górny <mgorny@gentoo.org> wrote:
> > > > Python 3.5+ introduces parallel build support in distutils.  Take
> > > > advantage of that by passing appropriate -j option.  Since distutils
> > > > does not support an equivalent of --load-average, default to the number
> > > > of CPUs+1 when unspecified.
> > > 
> > > How do we disable this in individual ebuilds when we inevitably run
> > > into problematic packages?
> > 
> > My guess would be, same as emake:
> > 
> >   distutils-r1_python_compile -j1
> > 
> > (I haven't tested it though)
> 
> This will affect stable ebuilds that have python3_5 or python3_6
> enabled. It would be a good idea to test it on a reasonable sample of
> ebuilds first.
> 
> Alternatively, we could enable it for python3_7 first and see what
> kind of fallout that produces.
> 

Given that we still have a reasonably small set of EAPI 7 ebuilds, how
about EAPI 7 for all Pythons + python3_7 for all EAPIs?  ;-)

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [PATCH] distutils-r1.eclass: Enable parallel builds in py3.5+

[Mike Gilbert]

On Thu, Jul 19, 2018 at 10:14 AM, Michał Górny <mgorny@gentoo.org> wrote:
> W dniu czw, 19.07.2018 o godzinie 10∶06 -0400, użytkownik Mike Gilbert
> napisał:
>> On Tue, Jul 17, 2018 at 10:48 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> > W dniu wto, 17.07.2018 o godzinie 10∶40 -0400, użytkownik Mike Gilbert
>> > napisał:
>> > > On Tue, Jul 17, 2018 at 4:41 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> > > > Python 3.5+ introduces parallel build support in distutils.  Take
>> > > > advantage of that by passing appropriate -j option.  Since distutils
>> > > > does not support an equivalent of --load-average, default to the number
>> > > > of CPUs+1 when unspecified.
>> > >
>> > > How do we disable this in individual ebuilds when we inevitably run
>> > > into problematic packages?
>> >
>> > My guess would be, same as emake:
>> >
>> >   distutils-r1_python_compile -j1
>> >
>> > (I haven't tested it though)
>>
>> This will affect stable ebuilds that have python3_5 or python3_6
>> enabled. It would be a good idea to test it on a reasonable sample of
>> ebuilds first.
>>
>> Alternatively, we could enable it for python3_7 first and see what
>> kind of fallout that produces.
>>
>
> Given that we still have a reasonably small set of EAPI 7 ebuilds, how
> about EAPI 7 for all Pythons + python3_7 for all EAPIs?  ;-)

That sounds good to me.