From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 3BEA11382C5 for ; Wed, 23 Jun 2021 09:36:44 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 2CD89E089C; Wed, 23 Jun 2021 09:36:40 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id C91ABE0895 for ; Wed, 23 Jun 2021 09:36:39 +0000 (UTC) Message-ID: <6e2af38b108ca1dbc166b7638c041bf57994e0a8.camel@gentoo.org> Subject: [gentoo-dev] [PATCH 2/2] python-utils-r1.eclass: Enable parallel bytecompile compilation From: Mart Raudsepp To: gentoo-dev@lists.gentoo.org Date: Wed, 23 Jun 2021 12:36:33 +0300 Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.4 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Archives-Salt: 83e21554-1e64-41d2-a4d9-a892f71b41b5 X-Archives-Hash: 782bce353ee41b655b594804416957f3 Python 3.5 added support for compileall to run parallel workers for performing bytecode compilation. Make use of it to the extent possible without refactoring the code too much to get different paths into the same call for best possible parallelization. ---  eclass/python-utils-r1.eclass | 16 +++++++++++-----  1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass index e2f05606993..f7a38f8c4e0 100644 --- a/eclass/python-utils-r1.eclass +++ b/eclass/python-utils-r1.eclass @@ -34,7 +34,7 @@ fi    if [[ ! ${_PYTHON_UTILS_R1} ]]; then   -inherit toolchain-funcs +inherit multiprocessing toolchain-funcs    # @ECLASS-VARIABLE: _PYTHON_ALL_IMPLS  # @INTERNAL @@ -615,6 +615,12 @@ python_optimize() {                 debug-print "${FUNCNAME}: using sys.path: ${*/%/;}"         fi   +       local jobs=$(makeopts_jobs "${MAKEOPTS}" INF) +       if [[ ${jobs} == INF ]]; then +               local nproc=$(get_nproc) +               jobs=$(( nproc + 1 )) +       fi +         local d         for d; do                 # make sure to get a nice path without // @@ -628,12 +634,12 @@ python_optimize() {                                 ;;                         python3.[5678]|pypy3)                                 # both levels of optimization are separate since 3.5 -                               "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}" -                               "${PYTHON}" -O -m compileall -q -f -d "${instpath}" "${d}" -                               "${PYTHON}" -OO -m compileall -q -f -d "${instpath}" "${d}" +                               "${PYTHON}" -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}" +                               "${PYTHON}" -O -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}" +                               "${PYTHON}" -OO -m compileall -j "${jobs}" -q -f -d "${instpath}" "${d}"                                 ;;                         python*) -                               "${PYTHON}" -m compileall -o 0 -o 1 -o 2 --hardlink-dupes -q -f -d "${instpath}" "${d}" +                               "${PYTHON}" -m compileall -j "${jobs}" -o 0 -o 1 -o 2 --hardlink-dupes -q -f -d "${instpath}" "${d}"                                 ;;                         *)                                 "${PYTHON}" -m compileall -q -f -d "${instpath}" "${d}"