[gentoo-dev] [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation

[gentoo-dev] [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation

[Mart Raudsepp]

Python 3.5 added support for compileall to run parallel workers for
performing bytecode compilation. Make use of it to the extent
possible without refactoring the code too much to get different
paths into the same call for best possible parallelization.
---
 eclass/python-utils-r1.eclass | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass
index e2f05606993..f7a38f8c4e0 100644
--- a/eclass/python-utils-r1.eclass
+++ b/eclass/python-utils-r1.eclass
@@ -34,7 +34,7 @@ fi
 
 if [[ ! ${_PYTHON_UTILS_R1} ]]; then
 
-inherit toolchain-funcs
+inherit multiprocessing toolchain-funcs
 
 # @ECLASS-VARIABLE: _PYTHON_ALL_IMPLS
 # @INTERNAL
@@ -615,6 +615,12 @@ python_optimize() {
                debug-print "${FUNCNAME}: using sys.path: ${*/%/;}"
        fi
 
+       local jobs=$(makeopts_jobs "${MAKEOPTS}" INF)
+       if [[ ${jobs} == INF ]]; then
+               local nproc=$(get_nproc)
+               jobs=$(( nproc + 1 ))
+       fi
+
        local d
        for d; do
                # make sure to get a nice path without //
@@ -628,12 +634,12 @@ python_optimize() {
                                ;;
                        python3.[5678]|pypy3)
                                # both levels of optimization are separate since 3.5
-                               "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"
-                               "${PYTHON}" -O -m compileall -q -f -d "${instpath}" "${d}"
-                               "${PYTHON}" -OO -m compileall -q -f -d "${instpath}" "${d}"
+                               "${PYTHON}" -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}"
+                               "${PYTHON}" -O -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}"
+                               "${PYTHON}" -OO -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}"
                                ;;
                        python*)
-                               "${PYTHON}" -m compileall -o 0 -o 1 -o 2 --hardlink-dupes -q -f -d "${instpath}" "${d}"
+                               "${PYTHON}" -m compileall -j "${jobs}" -o 0 -o 1 -o 2 --hardlink-dupes -q -f -d "${instpath}" "${d}"
                                ;;
                        *)
                                "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"

Re: [gentoo-dev] [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation

[Agostino Sarubbo]

On mercoledì 23 giugno 2021 11:36:33 CEST Mart Raudsepp wrote:
> +               jobs=$(( nproc + 1 ))

I don't know who historically started the nproc+1 story (it looks to be in the 
handbook too), but it has been demonstrated that nproc+1 is slower than nproc.

Agostino

Re: [gentoo-dev] [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation

[Mart Raudsepp]

[-- Attachment #1: Type: text/plain, Size: 440 bytes --]

Ühel kenal päeval, K, 23.06.2021 kell 11:43, kirjutas Agostino Sarubbo:
> On mercoledì 23 giugno 2021 11:36:33 CEST Mart Raudsepp wrote:
> > +               jobs=$(( nproc + 1 ))
> 
> I don't know who historically started the nproc+1 story (it looks to
> be in the 
> handbook too), but it has been demonstrated that nproc+1 is slower
> than nproc.

This is a copy of what distutils-r1.eclass is doing.


Mart

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation

[Michał Górny]

On Wed, 2021-06-23 at 11:43 +0200, Agostino Sarubbo wrote:
> On mercoledì 23 giugno 2021 11:36:33 CEST Mart Raudsepp wrote:
> > +               jobs=$(( nproc + 1 ))
> 
> I don't know who historically started the nproc+1 story (it looks to be in the 
> handbook too), but it has been demonstrated that nproc+1 is slower than nproc.
> 

Demonstrated on what?  This largely depends on the task in question.
In this case, I guess nproc makes more sense and makes things simpler
as it's got little I/O overhead.

-- 
Best regards,
Michał Górny