[gentoo-user] Suboptimal emerging of python libraries with native code

[gentoo-user] Suboptimal emerging of python libraries with native code

[Ian Zimmerman]

I just emerged sci-libs/scipy.  I was unpleasantly surprised that the
job only loaded one of my cpus most of the time, resulting in a longish
wait (15-20 minutes).

I investigated a bit.  Originally, I had this in make.conf:

MAKEOPTS="-j5"

Then I had a look at the ebuild.  I saw that it inherits
multiprocessing.eclass and uses the makeopts_jobs function from it.
Here is the function:

makeopts_jobs() {
	[[ $# -eq 0 ]] && set -- "${MAKEOPTS}"
	# This assumes the first .* will be more greedy than the second .*
	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
	local jobs=$(echo " $* " | sed -r -n \
		-e 's:.*[[:space:]](-[a-z]*j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
		-e "s:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:${2:-999}:p")
	echo ${jobs:-1}
}

Looking at this it seems to require at least one space between the -j
and the number!  Isn't that an unwarranted assumption?

So, I changed my make.conf to

MAKEOPTS="-j 5"

and rerun the build, but no joy - still no parallelization!  Another
closer look at the ebuild and I see this:

python_compile() {
	${EPYTHON} tools/cythonize.py || die
	distutils-r1_python_compile \
		$(usex python_targets_python3_5 "" "-j $(makeopts_jobs)") \
		${SCIPY_FCONFIG}
}

and sure enough, I'm building for python3.5, so I guess makeopts_jobs is
not even called and my MAKEOPTS are just ignored.  Why - what is so
special with python3.5 in the respect?  Would it help to switch to python3.6?

-- 
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.

Re: [gentoo-user] Suboptimal emerging of python libraries with native code

[Andrés Becerra Sandoval]

[-- Attachment #1: Type: text/plain, Size: 348 bytes --]

2018-05-03 18:13 GMT-05:00 Ian Zimmerman <itz@very.loosely.org>:

> I just emerged sci-libs/scipy.  I was unpleasantly surprised that the
> job only loaded one of my cpus most of the time, resulting in a longish
> wait (15-20 minutes).
>

​Hello Ian,

​​Maybe it is because this bug:
​
https://bugs.gentoo.org/614464

​ ​

[-- Attachment #2: Type: text/html, Size: 1046 bytes --]

[gentoo-user] Re: Suboptimal emerging of python libraries with native code

[Ian Zimmerman]

On 2018-05-04 08:33, Andrés Becerra Sandoval wrote:
 
> ​​Maybe it is because this bug:
> ​
> https://bugs.gentoo.org/614464

It almost certainly is, although the bug seems to say that
parallelization wouldn't work for python3.6 either, and the current
official ebuild seems to enable it (or least try to, I don't know if it
would because it goes through makeopts_jobs which could be buggy too).

By the way, if you're going to use thread pools in your python code, why
import them via _multiprocessing_ ??  There is an official and well
documented threading package?

And of course the code at the link in the bug [1] is horrible, complete
misuse of OOP so I wouldn't draw too many conclusions from it - other
than: the programmer is not very used to python.

[1]
http://lucasb.eyer.be/snips/python-thread-pool.html
-- 
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.