[gentoo-dev] [PATCH] multiprocessing.eclass: Improvements for wider use

[gentoo-dev] [PATCH] multiprocessing.eclass: Improvements for wider use

[Michał Górny]

Hello,

Here's a short batch of improvements to multiprocessing.eclass.

Patch 1 fixes handling multiple short args, e.g. '-kj'.

Patch 2 adds get_nproc() (copied from scons-utils.eclass) that tries
to portably determine the number of available CPUs.

Patch 3 adds the option to explicitly specify the replacement for
unlimited jobs/loadvg in makeopts_*() functions.

2+3 combined makes it possible to provide a safe replacement for
unlimited '--jobs' for build systems that do not support limiting
'--loadavg'.

--
Best regards,
Michał Górny

[gentoo-dev] [PATCH 1/3] multiprocessing.eclass: Fix handling multiple short options (e.g. -kj)

[Michał Górny]

Improve the regular expressions to handle parameters consisting of
multiple short options (such as -kj). It should be noted that the code
is not perfect but should handle all common (valid) cases; it could e.g.
incorrectly process a short option followed by string arg such as
'-Wfoo.j' although having this in MAKEOPTS is extremely unlikely.
---
 eclass/multiprocessing.eclass                    | 8 ++++----
 eclass/tests/multiprocessing_makeopts_jobs.sh    | 3 +++
 eclass/tests/multiprocessing_makeopts_loadavg.sh | 3 +++
 3 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index 06e004aa1669..5a5fe9acb56a 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -67,8 +67,8 @@ makeopts_jobs() {
 	# This assumes the first .* will be more greedy than the second .*
 	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
 	local jobs=$(echo " $* " | sed -r -n \
-		-e 's:.*[[:space:]](-j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
-		-e 's:.*[[:space:]](-j|--jobs)[[:space:]].*:999:p')
+		-e 's:.*[[:space:]](-[a-z]*j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
+		-e 's:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:999:p')
 	echo ${jobs:-1}
 }
 
@@ -86,8 +86,8 @@ makeopts_loadavg() {
 	# This assumes the first .* will be more greedy than the second .*
 	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
 	local lavg=$(echo " $* " | sed -r -n \
-		-e 's:.*[[:space:]](-l|--(load-average|max-load)[=[:space:]])[[:space:]]*([0-9]+|[0-9]+\.[0-9]+).*:\3:p' \
-		-e 's:.*[[:space:]](-l|--(load-average|max-load))[[:space:]].*:999:p')
+		-e 's:.*[[:space:]](-[a-z]*l|--(load-average|max-load)[=[:space:]])[[:space:]]*([0-9]+|[0-9]+\.[0-9]+).*:\3:p' \
+		-e 's:.*[[:space:]](-[a-z]*l|--(load-average|max-load))[[:space:]].*:999:p')
 	# Default to 999 since the default is to not use a load limit.
 	echo ${lavg:-999}
 }
diff --git a/eclass/tests/multiprocessing_makeopts_jobs.sh b/eclass/tests/multiprocessing_makeopts_jobs.sh
index 017d491156a0..a1e43c8b91d7 100755
--- a/eclass/tests/multiprocessing_makeopts_jobs.sh
+++ b/eclass/tests/multiprocessing_makeopts_jobs.sh
@@ -31,6 +31,9 @@ tests=(
 	7 "-l3 --jobs 7 -w"
 	4 "-j1 -j 2 --jobs 3 --jobs=4"
 	8 "     -j        			8     "
+	999 "-kj"
+	4 "-kj4"
+	5 "-kj 5"
 )
 for (( i = 0; i < ${#tests[@]}; i += 2 )) ; do
 	test-makeopts_jobs "${tests[i]}" "${tests[i+1]}"
diff --git a/eclass/tests/multiprocessing_makeopts_loadavg.sh b/eclass/tests/multiprocessing_makeopts_loadavg.sh
index 12f9d01f9fcd..276b7e70d393 100755
--- a/eclass/tests/multiprocessing_makeopts_loadavg.sh
+++ b/eclass/tests/multiprocessing_makeopts_loadavg.sh
@@ -28,6 +28,9 @@ tests=(
 	4 "-j1 -j 2 --load-average 3 --load-average=4"
 	3 " --max-load=3 -x"
 	8 "     -l        			8     "
+	999 "-kl"
+	4 "-kl4"
+	5 "-kl 5"
 )
 for (( i = 0; i < ${#tests[@]}; i += 2 )) ; do
 	test-makeopts_loadavg "${tests[i]}" "${tests[i+1]}"
-- 
2.11.0

[gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Michał Górny]

Introduce get_nproc(), a portable 'nproc' wrapper. It uses either
'nproc' or a fallback Python multiprocessing module call to attempt to
determine the number of available processing units.

This can be used e.g. to determine a safe number of jobs to run when
MAKEOPTS specifies unlimited --jobs and the build system in question
does not support --load-average.
---
 eclass/multiprocessing.eclass | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index 5a5fe9acb56a..0d241cdc15b6 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -53,6 +53,31 @@ bashpid() {
 	sh -c 'echo ${PPID}'
 }
 
+# @FUNCTION: get_nproc
+# @USAGE: [${fallback:-1}]
+# @DESCRIPTION:
+# Attempt to figure out the number of processing units available.
+# If the value can not be determined, prints the provided fallback
+# instead. If no fallback is provided, defaults to 1.
+get_nproc() {
+	local nproc
+
+	if type -P nproc &>/dev/null; then
+		# GNU
+		nproc=$(nproc)
+	elif type -P python &>/dev/null; then
+		# fallback to python2.6+
+		# note: this may fail (raise NotImplementedError)
+		nproc=$(python -c 'import multiprocessing; print(multiprocessing.cpu_count());' 2>/dev/null)
+	fi
+
+	if [[ -n ${nproc} ]]; then
+		echo "${nproc}"
+	else
+		echo "${1:-1}"
+	fi
+}
+
 # @FUNCTION: makeopts_jobs
 # @USAGE: [${MAKEOPTS}]
 # @DESCRIPTION:
-- 
2.11.0

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2044 bytes --]

On Tue, 13 Dec 2016 10:36:15 +0100 Michał Górny wrote:
> Introduce get_nproc(), a portable 'nproc' wrapper. It uses either
> 'nproc' or a fallback Python multiprocessing module call to attempt to
> determine the number of available processing units.
> 
> This can be used e.g. to determine a safe number of jobs to run when
> MAKEOPTS specifies unlimited --jobs and the build system in question
> does not support --load-average.
> ---
>  eclass/multiprocessing.eclass | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
> index 5a5fe9acb56a..0d241cdc15b6 100644
> --- a/eclass/multiprocessing.eclass
> +++ b/eclass/multiprocessing.eclass
> @@ -53,6 +53,31 @@ bashpid() {
>  	sh -c 'echo ${PPID}'
>  }
>  
> +# @FUNCTION: get_nproc
> +# @USAGE: [${fallback:-1}]
> +# @DESCRIPTION:
> +# Attempt to figure out the number of processing units available.
> +# If the value can not be determined, prints the provided fallback
> +# instead. If no fallback is provided, defaults to 1.
> +get_nproc() {
> +	local nproc
> +
> +	if type -P nproc &>/dev/null; then
> +		# GNU
> +		nproc=$(nproc)
> +	elif type -P python &>/dev/null; then
> +		# fallback to python2.6+
> +		# note: this may fail (raise NotImplementedError)
> +		nproc=$(python -c 'import multiprocessing; print(multiprocessing.cpu_count());' 2>/dev/null)

This is not portable. E.g. paludis users can have python-less
system. Adding dev-lang/python to DEPEND will be also a bad idea,
since this is quite heavy dependency.

Since on Linux boxes nproc is from coreutils, which is in @system,
so looks like only *bsd setups are the problem. On FreeBSD
  sysctl -a | egrep -i 'hw.ncpu'
should be sufficient solution.

> +	fi
> +
> +	if [[ -n ${nproc} ]]; then
> +		echo "${nproc}"
> +	else
> +		echo "${1:-1}"
> +	fi
> +}
> +
>  # @FUNCTION: makeopts_jobs
>  # @USAGE: [${MAKEOPTS}]
>  # @DESCRIPTION:


Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2581 bytes --]

On Wed, 14 Dec 2016 15:27:25 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:

> On Tue, 13 Dec 2016 10:36:15 +0100 Michał Górny wrote:
> > Introduce get_nproc(), a portable 'nproc' wrapper. It uses either
> > 'nproc' or a fallback Python multiprocessing module call to attempt to
> > determine the number of available processing units.
> > 
> > This can be used e.g. to determine a safe number of jobs to run when
> > MAKEOPTS specifies unlimited --jobs and the build system in question
> > does not support --load-average.
> > ---
> >  eclass/multiprocessing.eclass | 25 +++++++++++++++++++++++++
> >  1 file changed, 25 insertions(+)
> > 
> > diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
> > index 5a5fe9acb56a..0d241cdc15b6 100644
> > --- a/eclass/multiprocessing.eclass
> > +++ b/eclass/multiprocessing.eclass
> > @@ -53,6 +53,31 @@ bashpid() {
> >  	sh -c 'echo ${PPID}'
> >  }
> >  
> > +# @FUNCTION: get_nproc
> > +# @USAGE: [${fallback:-1}]
> > +# @DESCRIPTION:
> > +# Attempt to figure out the number of processing units available.
> > +# If the value can not be determined, prints the provided fallback
> > +# instead. If no fallback is provided, defaults to 1.
> > +get_nproc() {
> > +	local nproc
> > +
> > +	if type -P nproc &>/dev/null; then
> > +		# GNU
> > +		nproc=$(nproc)
> > +	elif type -P python &>/dev/null; then
> > +		# fallback to python2.6+
> > +		# note: this may fail (raise NotImplementedError)
> > +		nproc=$(python -c 'import multiprocessing; print(multiprocessing.cpu_count());' 2>/dev/null)  
> 
> This is not portable. E.g. paludis users can have python-less
> system. Adding dev-lang/python to DEPEND will be also a bad idea,
> since this is quite heavy dependency.

You can bikeshed potential circumstances where it wouldn't work for
the next year. Which doesn't change that it would work quite reliably
for the most of Gentoo users.

> Since on Linux boxes nproc is from coreutils, which is in @system,
> so looks like only *bsd setups are the problem. On FreeBSD

...and all other operating systems (see: Prefix).

>   sysctl -a | egrep -i 'hw.ncpu'

I somehow doubt that would give me the expected number only, and I lack
a BSD install handy to test it.

> > +	fi
> > +
> > +	if [[ -n ${nproc} ]]; then
> > +		echo "${nproc}"
> > +	else
> > +		echo "${1:-1}"
> > +	fi
> > +}
> > +
> >  # @FUNCTION: makeopts_jobs
> >  # @USAGE: [${MAKEOPTS}]
> >  # @DESCRIPTION:  

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Doug Freed]

On Wed, Dec 14, 2016 at 11:04 AM, Michał Górny <mgorny@gentoo.org> wrote:
> On Wed, 14 Dec 2016 15:27:25 +0300
> Andrew Savchenko <bircoph@gentoo.org> wrote:
>
>> On Tue, 13 Dec 2016 10:36:15 +0100 Michał Górny wrote:
>> > +           nproc=$(python -c 'import multiprocessing; print(multiprocessing.cpu_count());' 2>/dev/null)
>>
>> This is not portable. E.g. paludis users can have python-less
>> system. Adding dev-lang/python to DEPEND will be also a bad idea,
>> since this is quite heavy dependency.
>
> You can bikeshed potential circumstances where it wouldn't work for
> the next year. Which doesn't change that it would work quite reliably
> for the most of Gentoo users.
>
>> Since on Linux boxes nproc is from coreutils, which is in @system,
>> so looks like only *bsd setups are the problem. On FreeBSD
>
> ...and all other operating systems (see: Prefix).
>
>>   sysctl -a | egrep -i 'hw.ncpu'
>
> I somehow doubt that would give me the expected number only, and I lack
> a BSD install handy to test it.

$(sysctl -n hw.ncpu)

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Nathan Zachary]

[-- Attachment #1.1: Type: text/plain, Size: 1266 bytes --]

On 14/12/16 10:11, Doug Freed wrote:
> On Wed, Dec 14, 2016 at 11:04 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> On Wed, 14 Dec 2016 15:27:25 +0300
>> Andrew Savchenko <bircoph@gentoo.org> wrote:
>>
>>> On Tue, 13 Dec 2016 10:36:15 +0100 Michał Górny wrote:
>>>> +           nproc=$(python -c 'import multiprocessing; print(multiprocessing.cpu_count());' 2>/dev/null)
>>> This is not portable. E.g. paludis users can have python-less
>>> system. Adding dev-lang/python to DEPEND will be also a bad idea,
>>> since this is quite heavy dependency.
>> You can bikeshed potential circumstances where it wouldn't work for
>> the next year. Which doesn't change that it would work quite reliably
>> for the most of Gentoo users.
>>
>>> Since on Linux boxes nproc is from coreutils, which is in @system,
>>> so looks like only *bsd setups are the problem. On FreeBSD
>> ...and all other operating systems (see: Prefix).
>>
>>>   sysctl -a | egrep -i 'hw.ncpu'
>> I somehow doubt that would give me the expected number only, and I lack
>> a BSD install handy to test it.
> $(sysctl -n hw.ncpu)
>
I don't know that the sysctl command works universally:

# sysctl -n hw.ncpu
sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 870 bytes --]

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Doug Freed]

On Wed, Dec 14, 2016 at 12:48 PM, Nathan Zachary
<nathanzachary@gentoo.org> wrote:
> On 14/12/16 10:11, Doug Freed wrote:
>> On Wed, Dec 14, 2016 at 11:04 AM, Michał Górny <mgorny@gentoo.org> wrote:
>>> On Wed, 14 Dec 2016 15:27:25 +0300
>>> Andrew Savchenko <bircoph@gentoo.org> wrote:
>>>
>>>> On Tue, 13 Dec 2016 10:36:15 +0100 Michał Górny wrote:
>>>>> +           nproc=$(python -c 'import multiprocessing; print(multiprocessing.cpu_count());' 2>/dev/null)
>>>> This is not portable. E.g. paludis users can have python-less
>>>> system. Adding dev-lang/python to DEPEND will be also a bad idea,
>>>> since this is quite heavy dependency.
>>> You can bikeshed potential circumstances where it wouldn't work for
>>> the next year. Which doesn't change that it would work quite reliably
>>> for the most of Gentoo users.
>>>
>>>> Since on Linux boxes nproc is from coreutils, which is in @system,
>>>> so looks like only *bsd setups are the problem. On FreeBSD
>>> ...and all other operating systems (see: Prefix).
>>>
>>>>   sysctl -a | egrep -i 'hw.ncpu'
>>> I somehow doubt that would give me the expected number only, and I lack
>>> a BSD install handy to test it.
>> $(sysctl -n hw.ncpu)
>>
> I don't know that the sysctl command works universally:
>
> # sysctl -n hw.ncpu
> sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
>

It's BSD-specific (Darwin may have it too, but I'm not in OS X at the
moment, so I can't check), which pretty much describes this branch in
the codepath as well.  Linux users will have the nproc command from
coreutils.

-Doug

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

On 14-12-2016 13:01:16 -0500, Doug Freed wrote:
> On Wed, Dec 14, 2016 at 12:48 PM, Nathan Zachary
> <nathanzachary@gentoo.org> wrote:
> > On 14/12/16 10:11, Doug Freed wrote:
> >>> I somehow doubt that would give me the expected number only, and I lack
> >>> a BSD install handy to test it.
> >> $(sysctl -n hw.ncpu)
> >>
> > I don't know that the sysctl command works universally:
> >
> > # sysctl -n hw.ncpu
> > sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
> >
> 
> It's BSD-specific (Darwin may have it too, but I'm not in OS X at the
> moment, so I can't check), which pretty much describes this branch in
> the codepath as well.  Linux users will have the nproc command from
> coreutils.

Yes, Darwin has it too, but the tool lives in /usr/sbin instead:

https://gitweb.gentoo.org/repo/proj/prefix.git/tree/scripts/bootstrap-prefix.sh#n1987

Fabian


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Nathan Zachary]

[-- Attachment #1.1: Type: text/plain, Size: 1138 bytes --]

On 14/12/16 12:29, Fabian Groffen wrote:
> On 14-12-2016 13:01:16 -0500, Doug Freed wrote:
>> On Wed, Dec 14, 2016 at 12:48 PM, Nathan Zachary
>> <nathanzachary@gentoo.org> wrote:
>>> On 14/12/16 10:11, Doug Freed wrote:
>>>>> I somehow doubt that would give me the expected number only, and I lack
>>>>> a BSD install handy to test it.
>>>> $(sysctl -n hw.ncpu)
>>>>
>>> I don't know that the sysctl command works universally:
>>>
>>> # sysctl -n hw.ncpu
>>> sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
>>>
>> It's BSD-specific (Darwin may have it too, but I'm not in OS X at the
>> moment, so I can't check), which pretty much describes this branch in
>> the codepath as well.  Linux users will have the nproc command from
>> coreutils.
> Yes, Darwin has it too, but the tool lives in /usr/sbin instead:
>
> https://gitweb.gentoo.org/repo/proj/prefix.git/tree/scripts/bootstrap-prefix.sh#n1987
>
> Fabian
>
>
Ah, yes, I missed the part about it being BSD-specific.

This one-liner certainly isn't as graceful or elegant, but it works:

# cat /proc/cpuinfo | grep 'processor' | wc -l


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 870 bytes --]

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 1430 bytes --]

2016-12-14 19:32 GMT+01:00 Nathan Zachary <nathanzachary@gentoo.org>:

> On 14/12/16 12:29, Fabian Groffen wrote:
> > On 14-12-2016 13:01:16 -0500, Doug Freed wrote:
> >> On Wed, Dec 14, 2016 at 12:48 PM, Nathan Zachary
> >> <nathanzachary@gentoo.org> wrote:
> >>> On 14/12/16 10:11, Doug Freed wrote:
> >>>>> I somehow doubt that would give me the expected number only, and I
> lack
> >>>>> a BSD install handy to test it.
> >>>> $(sysctl -n hw.ncpu)
> >>>>
> >>> I don't know that the sysctl command works universally:
> >>>
> >>> # sysctl -n hw.ncpu
> >>> sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
> >>>
> >> It's BSD-specific (Darwin may have it too, but I'm not in OS X at the
> >> moment, so I can't check), which pretty much describes this branch in
> >> the codepath as well.  Linux users will have the nproc command from
> >> coreutils.
> > Yes, Darwin has it too, but the tool lives in /usr/sbin instead:
> >
> > https://gitweb.gentoo.org/repo/proj/prefix.git/tree/
> scripts/bootstrap-prefix.sh#n1987
> >
> > Fabian
> >
> >
> Ah, yes, I missed the part about it being BSD-specific.
>
> This one-liner certainly isn't as graceful or elegant, but it works:
>
> # cat /proc/cpuinfo | grep 'processor' | wc -l
>

# grep -c processor /proc/cpuinfo

however Documentation/filesystems/proc.txt is very vague on it's content,
just stating it has info on the cpu, better leave this as a backup value
IMHO

[-- Attachment #2: Type: text/html, Size: 2361 bytes --]

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

On Wed, 14 Dec 2016 12:32:41 -0600 Nathan Zachary wrote:
> On 14/12/16 12:29, Fabian Groffen wrote:
> > On 14-12-2016 13:01:16 -0500, Doug Freed wrote:
> >> On Wed, Dec 14, 2016 at 12:48 PM, Nathan Zachary
> >> <nathanzachary@gentoo.org> wrote:
> >>> On 14/12/16 10:11, Doug Freed wrote:
> >>>>> I somehow doubt that would give me the expected number only, and I lack
> >>>>> a BSD install handy to test it.
> >>>> $(sysctl -n hw.ncpu)
> >>>>
> >>> I don't know that the sysctl command works universally:
> >>>
> >>> # sysctl -n hw.ncpu
> >>> sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
> >>>
> >> It's BSD-specific (Darwin may have it too, but I'm not in OS X at the
> >> moment, so I can't check), which pretty much describes this branch in
> >> the codepath as well.  Linux users will have the nproc command from
> >> coreutils.
> > Yes, Darwin has it too, but the tool lives in /usr/sbin instead:
> >
> > https://gitweb.gentoo.org/repo/proj/prefix.git/tree/scripts/bootstrap-prefix.sh#n1987
> >
> > Fabian
> >
> >
> Ah, yes, I missed the part about it being BSD-specific.
> 
> This one-liner certainly isn't as graceful or elegant, but it works:
> 
> # cat /proc/cpuinfo | grep 'processor' | wc -l
 
Not all BSD systems have procfs enabled.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [PATCH 2/3] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Mike Gilbert]

On Wed, Dec 14, 2016 at 1:29 PM, Fabian Groffen <grobian@gentoo.org> wrote:
> On 14-12-2016 13:01:16 -0500, Doug Freed wrote:
>> On Wed, Dec 14, 2016 at 12:48 PM, Nathan Zachary
>> <nathanzachary@gentoo.org> wrote:
>> > On 14/12/16 10:11, Doug Freed wrote:
>> >>> I somehow doubt that would give me the expected number only, and I lack
>> >>> a BSD install handy to test it.
>> >> $(sysctl -n hw.ncpu)
>> >>
>> > I don't know that the sysctl command works universally:
>> >
>> > # sysctl -n hw.ncpu
>> > sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
>> >
>>
>> It's BSD-specific (Darwin may have it too, but I'm not in OS X at the
>> moment, so I can't check), which pretty much describes this branch in
>> the codepath as well.  Linux users will have the nproc command from
>> coreutils.
>
> Yes, Darwin has it too, but the tool lives in /usr/sbin instead:
>
> https://gitweb.gentoo.org/repo/proj/prefix.git/tree/scripts/bootstrap-prefix.sh#n1987
>

Seems like we could borrow that code for the eclass, with some minor
modifications. It should probably use ${CBUILD:-${CHOST}} to cover
cross-compiling.

Another alternative would be to check the userland_BSD and
userland_GNU use flags.

[gentoo-dev] [PATCH v2] multiprocessing.eclass: Introduce get_nproc() to get no of CPUs

[Michał Górny]

Introduce get_nproc(), a portable 'nproc' wrapper. It uses either
'nproc' or a fallback Python multiprocessing module call to attempt to
determine the number of available processing units.

This can be used e.g. to determine a safe number of jobs to run when
MAKEOPTS specifies unlimited --jobs and the build system in question
does not support --load-average.

// v2: added sysctl call for BSDs
---
 eclass/multiprocessing.eclass | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index 5a5fe9acb56a..3c5dfff2d11c 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -53,6 +53,38 @@ bashpid() {
 	sh -c 'echo ${PPID}'
 }
 
+# @FUNCTION: get_nproc
+# @USAGE: [${fallback:-1}]
+# @DESCRIPTION:
+# Attempt to figure out the number of processing units available.
+# If the value can not be determined, prints the provided fallback
+# instead. If no fallback is provided, defaults to 1.
+get_nproc() {
+	local nproc
+
+	# GNU
+	if type -P nproc &>/dev/null; then
+		nproc=$(nproc)
+	fi
+
+	# BSD
+	if [[ -z ${nproc} ]] && type -P sysctl &>/dev/null; then
+		nproc=$(sysctl -n hw.ncpu 2>/dev/null)
+	fi
+
+	# fallback to python2.6+
+	# note: this may fail (raise NotImplementedError)
+	if [[ -z ${nproc} ]] && type -P python &>/dev/null; then
+		nproc=$(python -c 'import multiprocessing; print(multiprocessing.cpu_count());' 2>/dev/null)
+	fi
+
+	if [[ -n ${nproc} ]]; then
+		echo "${nproc}"
+	else
+		echo "${1:-1}"
+	fi
+}
+
 # @FUNCTION: makeopts_jobs
 # @USAGE: [${MAKEOPTS}]
 # @DESCRIPTION:
-- 
2.11.0

[gentoo-dev] [PATCH 3/3] multiprocessing.eclass: Support passing custom inf values for getters

[Michał Górny]

Support passing custom values for 'infinity' in makeopts_jobs()
and makeopts_loadavg(). This can be used e.g. when a build system does
not support --loadavg, and therefore '--jobs 999' would most likely
be a really bad idea. Combined with get_nproc(), this can be used to
provide a sane replacement instead.
---
 eclass/multiprocessing.eclass | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index 0d241cdc15b6..b7d5f435f888 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -79,26 +79,27 @@ get_nproc() {
 }
 
 # @FUNCTION: makeopts_jobs
-# @USAGE: [${MAKEOPTS}]
+# @USAGE: [${MAKEOPTS}] [${inf:-999}]
 # @DESCRIPTION:
 # Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number
 # specified therein.  Useful for running non-make tools in parallel too.
 # i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the
 # number as bash normalizes it to [0, 255].  If the flags haven't specified a
 # -j flag, then "1" is shown as that is the default `make` uses.  Since there's
-# no way to represent infinity, we return 999 if the user has -j without a number.
+# no way to represent infinity, we return ${inf} (defaults to 999) if the user
+# has -j without a number.
 makeopts_jobs() {
 	[[ $# -eq 0 ]] && set -- ${MAKEOPTS}
 	# This assumes the first .* will be more greedy than the second .*
 	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
 	local jobs=$(echo " $* " | sed -r -n \
 		-e 's:.*[[:space:]](-[a-z]*j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
-		-e 's:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:999:p')
+		-e "s:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:${2:-999}:p")
 	echo ${jobs:-1}
 }
 
 # @FUNCTION: makeopts_loadavg
-# @USAGE: [${MAKEOPTS}]
+# @USAGE: [${MAKEOPTS}] [${inf:-999}]
 # @DESCRIPTION:
 # Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
 # for load-average. For make and ninja based builds this will mean new jobs are
@@ -106,15 +107,17 @@ makeopts_jobs() {
 # get excessive due to I/O and not just due to CPU load.
 # Be aware that the returned number might be a floating-point number. Test
 # whether your software supports that.
+# If no limit is specified or --load-average is used without a number, ${inf}
+# (defaults to 999) is returned.
 makeopts_loadavg() {
 	[[ $# -eq 0 ]] && set -- ${MAKEOPTS}
 	# This assumes the first .* will be more greedy than the second .*
 	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
 	local lavg=$(echo " $* " | sed -r -n \
 		-e 's:.*[[:space:]](-[a-z]*l|--(load-average|max-load)[=[:space:]])[[:space:]]*([0-9]+|[0-9]+\.[0-9]+).*:\3:p' \
-		-e 's:.*[[:space:]](-[a-z]*l|--(load-average|max-load))[[:space:]].*:999:p')
-	# Default to 999 since the default is to not use a load limit.
-	echo ${lavg:-999}
+		-e "s:.*[[:space:]](-[a-z]*l|--(load-average|max-load))[[:space:]].*:${2:-999}:p")
+	# Default to ${inf} since the default is to not use a load limit.
+	echo ${lavg:-${2:-999}}
 }
 
 # @FUNCTION: multijob_init
-- 
2.11.0

Re: [gentoo-dev] [PATCH 3/3] multiprocessing.eclass: Support passing custom inf values for getters

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 3531 bytes --]

On Tue, 13 Dec 2016 10:36:16 +0100 Michał Górny wrote:
> Support passing custom values for 'infinity' in makeopts_jobs()
> and makeopts_loadavg(). This can be used e.g. when a build system does
> not support --loadavg, and therefore '--jobs 999' would most likely
> be a really bad idea. Combined with get_nproc(), this can be used to
> provide a sane replacement instead.
> ---
>  eclass/multiprocessing.eclass | 17 ++++++++++-------
>  1 file changed, 10 insertions(+), 7 deletions(-)
> 
> diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
> index 0d241cdc15b6..b7d5f435f888 100644
> --- a/eclass/multiprocessing.eclass
> +++ b/eclass/multiprocessing.eclass
> @@ -79,26 +79,27 @@ get_nproc() {
>  }
>  
>  # @FUNCTION: makeopts_jobs
> -# @USAGE: [${MAKEOPTS}]
> +# @USAGE: [${MAKEOPTS}] [${inf:-999}]
>  # @DESCRIPTION:
>  # Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number
>  # specified therein.  Useful for running non-make tools in parallel too.
>  # i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the
>  # number as bash normalizes it to [0, 255].  If the flags haven't specified a
>  # -j flag, then "1" is shown as that is the default `make` uses.  Since there's
> -# no way to represent infinity, we return 999 if the user has -j without a number.
> +# no way to represent infinity, we return ${inf} (defaults to 999) if the user
> +# has -j without a number.
>  makeopts_jobs() {
>  	[[ $# -eq 0 ]] && set -- ${MAKEOPTS}
>  	# This assumes the first .* will be more greedy than the second .*
>  	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
>  	local jobs=$(echo " $* " | sed -r -n \
>  		-e 's:.*[[:space:]](-[a-z]*j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
> -		-e 's:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:999:p')
> +		-e "s:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:${2:-999}:p")
>  	echo ${jobs:-1}
>  }
>  
>  # @FUNCTION: makeopts_loadavg
> -# @USAGE: [${MAKEOPTS}]
> +# @USAGE: [${MAKEOPTS}] [${inf:-999}]
>  # @DESCRIPTION:
>  # Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
>  # for load-average. For make and ninja based builds this will mean new jobs are
> @@ -106,15 +107,17 @@ makeopts_jobs() {
>  # get excessive due to I/O and not just due to CPU load.
>  # Be aware that the returned number might be a floating-point number. Test
>  # whether your software supports that.
> +# If no limit is specified or --load-average is used without a number, ${inf}
> +# (defaults to 999) is returned.

Why not to feed default ${inf} from get_nproc() by default?
This will make ebuild writing easier.

>  makeopts_loadavg() {
>  	[[ $# -eq 0 ]] && set -- ${MAKEOPTS}
>  	# This assumes the first .* will be more greedy than the second .*
>  	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
>  	local lavg=$(echo " $* " | sed -r -n \
>  		-e 's:.*[[:space:]](-[a-z]*l|--(load-average|max-load)[=[:space:]])[[:space:]]*([0-9]+|[0-9]+\.[0-9]+).*:\3:p' \
> -		-e 's:.*[[:space:]](-[a-z]*l|--(load-average|max-load))[[:space:]].*:999:p')
> -	# Default to 999 since the default is to not use a load limit.
> -	echo ${lavg:-999}
> +		-e "s:.*[[:space:]](-[a-z]*l|--(load-average|max-load))[[:space:]].*:${2:-999}:p")
> +	# Default to ${inf} since the default is to not use a load limit.
> +	echo ${lavg:-${2:-999}}
>  }
>  
>  # @FUNCTION: multijob_init


Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [PATCH 3/3] multiprocessing.eclass: Support passing custom inf values for getters

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3420 bytes --]

On Wed, 14 Dec 2016 15:30:06 +0300
Andrew Savchenko <bircoph@gentoo.org> wrote:

> On Tue, 13 Dec 2016 10:36:16 +0100 Michał Górny wrote:
> > Support passing custom values for 'infinity' in makeopts_jobs()
> > and makeopts_loadavg(). This can be used e.g. when a build system does
> > not support --loadavg, and therefore '--jobs 999' would most likely
> > be a really bad idea. Combined with get_nproc(), this can be used to
> > provide a sane replacement instead.
> > ---
> >  eclass/multiprocessing.eclass | 17 ++++++++++-------
> >  1 file changed, 10 insertions(+), 7 deletions(-)
> > 
> > diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
> > index 0d241cdc15b6..b7d5f435f888 100644
> > --- a/eclass/multiprocessing.eclass
> > +++ b/eclass/multiprocessing.eclass
> > @@ -79,26 +79,27 @@ get_nproc() {
> >  }
> >  
> >  # @FUNCTION: makeopts_jobs
> > -# @USAGE: [${MAKEOPTS}]
> > +# @USAGE: [${MAKEOPTS}] [${inf:-999}]
> >  # @DESCRIPTION:
> >  # Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number
> >  # specified therein.  Useful for running non-make tools in parallel too.
> >  # i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the
> >  # number as bash normalizes it to [0, 255].  If the flags haven't specified a
> >  # -j flag, then "1" is shown as that is the default `make` uses.  Since there's
> > -# no way to represent infinity, we return 999 if the user has -j without a number.
> > +# no way to represent infinity, we return ${inf} (defaults to 999) if the user
> > +# has -j without a number.
> >  makeopts_jobs() {
> >  	[[ $# -eq 0 ]] && set -- ${MAKEOPTS}
> >  	# This assumes the first .* will be more greedy than the second .*
> >  	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
> >  	local jobs=$(echo " $* " | sed -r -n \
> >  		-e 's:.*[[:space:]](-[a-z]*j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
> > -		-e 's:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:999:p')
> > +		-e "s:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:${2:-999}:p")
> >  	echo ${jobs:-1}
> >  }
> >  
> >  # @FUNCTION: makeopts_loadavg
> > -# @USAGE: [${MAKEOPTS}]
> > +# @USAGE: [${MAKEOPTS}] [${inf:-999}]
> >  # @DESCRIPTION:
> >  # Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
> >  # for load-average. For make and ninja based builds this will mean new jobs are
> > @@ -106,15 +107,17 @@ makeopts_jobs() {
> >  # get excessive due to I/O and not just due to CPU load.
> >  # Be aware that the returned number might be a floating-point number. Test
> >  # whether your software supports that.
> > +# If no limit is specified or --load-average is used without a number, ${inf}
> > +# (defaults to 999) is returned.  
> 
> Why not to feed default ${inf} from get_nproc() by default?
> This will make ebuild writing easier.

Maybe it would. However, I didn't want to change the existing behavior.

Most importantly, that would mean that for people with '-j -l XXX'
the eclass would suddenly changed from '-j 999 -l XXX' to
'-j 2 -l XXX', i.e. defeat the purpose of original --jobs.

As the commit message says, the main use for nproc is for stuff that
doesn't support --loadavg or follow regular rules. SCons, various
multiprocessing test suites...

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

[gentoo-dev] [PATCH v2] multiprocessing.eclass: Support passing custom inf values for getters

[Michał Górny]

Support passing custom values for 'infinity' in makeopts_jobs()
and makeopts_loadavg(). This can be used e.g. when a build system does
not support --loadavg, and therefore '--jobs 999' would most likely
be a really bad idea. Combined with get_nproc(), this can be used to
provide a sane replacement instead.

// v2: tests included
---
 eclass/multiprocessing.eclass                    | 17 ++++++++++-------
 eclass/tests/multiprocessing_makeopts_jobs.sh    |  5 ++++-
 eclass/tests/multiprocessing_makeopts_loadavg.sh |  5 ++++-
 3 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/eclass/multiprocessing.eclass b/eclass/multiprocessing.eclass
index 3c5dfff2d11c..70ca475a8c72 100644
--- a/eclass/multiprocessing.eclass
+++ b/eclass/multiprocessing.eclass
@@ -86,26 +86,27 @@ get_nproc() {
 }
 
 # @FUNCTION: makeopts_jobs
-# @USAGE: [${MAKEOPTS}]
+# @USAGE: [${MAKEOPTS}] [${inf:-999}]
 # @DESCRIPTION:
 # Searches the arguments (defaults to ${MAKEOPTS}) and extracts the jobs number
 # specified therein.  Useful for running non-make tools in parallel too.
 # i.e. if the user has MAKEOPTS=-j9, this will echo "9" -- we can't return the
 # number as bash normalizes it to [0, 255].  If the flags haven't specified a
 # -j flag, then "1" is shown as that is the default `make` uses.  Since there's
-# no way to represent infinity, we return 999 if the user has -j without a number.
+# no way to represent infinity, we return ${inf} (defaults to 999) if the user
+# has -j without a number.
 makeopts_jobs() {
 	[[ $# -eq 0 ]] && set -- ${MAKEOPTS}
 	# This assumes the first .* will be more greedy than the second .*
 	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
 	local jobs=$(echo " $* " | sed -r -n \
 		-e 's:.*[[:space:]](-[a-z]*j|--jobs[=[:space:]])[[:space:]]*([0-9]+).*:\2:p' \
-		-e 's:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:999:p')
+		-e "s:.*[[:space:]](-[a-z]*j|--jobs)[[:space:]].*:${2:-999}:p")
 	echo ${jobs:-1}
 }
 
 # @FUNCTION: makeopts_loadavg
-# @USAGE: [${MAKEOPTS}]
+# @USAGE: [${MAKEOPTS}] [${inf:-999}]
 # @DESCRIPTION:
 # Searches the arguments (defaults to ${MAKEOPTS}) and extracts the value set
 # for load-average. For make and ninja based builds this will mean new jobs are
@@ -113,15 +114,17 @@ makeopts_jobs() {
 # get excessive due to I/O and not just due to CPU load.
 # Be aware that the returned number might be a floating-point number. Test
 # whether your software supports that.
+# If no limit is specified or --load-average is used without a number, ${inf}
+# (defaults to 999) is returned.
 makeopts_loadavg() {
 	[[ $# -eq 0 ]] && set -- ${MAKEOPTS}
 	# This assumes the first .* will be more greedy than the second .*
 	# since POSIX doesn't specify a non-greedy match (i.e. ".*?").
 	local lavg=$(echo " $* " | sed -r -n \
 		-e 's:.*[[:space:]](-[a-z]*l|--(load-average|max-load)[=[:space:]])[[:space:]]*([0-9]+|[0-9]+\.[0-9]+).*:\3:p' \
-		-e 's:.*[[:space:]](-[a-z]*l|--(load-average|max-load))[[:space:]].*:999:p')
-	# Default to 999 since the default is to not use a load limit.
-	echo ${lavg:-999}
+		-e "s:.*[[:space:]](-[a-z]*l|--(load-average|max-load))[[:space:]].*:${2:-999}:p")
+	# Default to ${inf} since the default is to not use a load limit.
+	echo ${lavg:-${2:-999}}
 }
 
 # @FUNCTION: multijob_init
diff --git a/eclass/tests/multiprocessing_makeopts_jobs.sh b/eclass/tests/multiprocessing_makeopts_jobs.sh
index a1e43c8b91d7..ef477277ab3b 100755
--- a/eclass/tests/multiprocessing_makeopts_jobs.sh
+++ b/eclass/tests/multiprocessing_makeopts_jobs.sh
@@ -9,7 +9,7 @@ inherit multiprocessing
 
 test-makeopts_jobs() {
 	local exp=$1; shift
-	tbegin "makeopts_jobs($*) == ${exp}"
+	tbegin "makeopts_jobs($1${2+; inf=${2}}) == ${exp}"
 	local act=$(makeopts_jobs "$@")
 	[[ ${act} == "${exp}" ]]
 	tend $? "Got back: ${act}"
@@ -39,4 +39,7 @@ for (( i = 0; i < ${#tests[@]}; i += 2 )) ; do
 	test-makeopts_jobs "${tests[i]}" "${tests[i+1]}"
 done
 
+# test custom inf value
+test-makeopts_jobs 645 "-j" 645
+
 texit
diff --git a/eclass/tests/multiprocessing_makeopts_loadavg.sh b/eclass/tests/multiprocessing_makeopts_loadavg.sh
index 276b7e70d393..6b976beb1aef 100755
--- a/eclass/tests/multiprocessing_makeopts_loadavg.sh
+++ b/eclass/tests/multiprocessing_makeopts_loadavg.sh
@@ -9,7 +9,7 @@ inherit multiprocessing
 
 test-makeopts_loadavg() {
 	local exp=$1; shift
-	tbegin "makeopts_loadavg($*) == ${exp}"
+	tbegin "makeopts_loadavg($1${2+; inf=${2}}) == ${exp}"
 	local act=$(makeopts_loadavg "$@")
 	[[ ${act} == "${exp}" ]]
 	tend $? "Got back: ${act}"
@@ -36,4 +36,7 @@ for (( i = 0; i < ${#tests[@]}; i += 2 )) ; do
 	test-makeopts_loadavg "${tests[i]}" "${tests[i+1]}"
 done
 
+# test custom inf value
+test-makeopts_loadavg 645 "-l" 645
+
 texit
-- 
2.11.0

Re: [gentoo-dev] [PATCH] multiprocessing.eclass: Improvements for wider use

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

On Tue, 13 Dec 2016 10:36:13 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Hello,
> 
> Here's a short batch of improvements to multiprocessing.eclass.
> 
> Patch 1 fixes handling multiple short args, e.g. '-kj'.
> 
> Patch 2 adds get_nproc() (copied from scons-utils.eclass) that tries
> to portably determine the number of available CPUs.
> 
> Patch 3 adds the option to explicitly specify the replacement for
> unlimited jobs/loadvg in makeopts_*() functions.
> 
> 2+3 combined makes it possible to provide a safe replacement for
> unlimited '--jobs' for build systems that do not support limiting
> '--loadavg'.

Pushed now.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]