[gentoo-portage-dev] [PATCH] Default MAKEOPTS to -j(ncpus+1) when unset

[gentoo-portage-dev] [PATCH] Default MAKEOPTS to -j(ncpus+1) when unset

[Michał Górny]

Default MAKEOPTS job number to (number of CPUs + 1) when it is not
provided in the ebuild environment.

Suggested-By: Daniel Robbins <drobbins@funtoo.org>
---
 pym/portage/package/ebuild/doebuild.py |  8 +++++++-
 pym/portage/util/cpuinfo.py            | 19 +++++++++++++++++++
 2 files changed, 26 insertions(+), 1 deletion(-)
 create mode 100644 pym/portage/util/cpuinfo.py

diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index bf97660..f43dddc 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2013 Gentoo Foundation
+# Copyright 2010-2015 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -66,6 +66,7 @@ from portage.package.ebuild.prepare_build_dirs import prepare_build_dirs
 from portage.util import apply_recursive_permissions, \
 	apply_secpass_permissions, noiselimit, normalize_path, \
 	writemsg, writemsg_stdout, write_atomic
+from portage.util.cpuinfo import get_cpu_count
 from portage.util.lafilefixer import rewrite_lafile	
 from portage.versions import _pkgsplit
 from _emerge.BinpkgEnvExtractor import BinpkgEnvExtractor
@@ -463,6 +464,11 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 				mysettings["PATH"] = os.path.join(os.sep, eprefix_lstrip,
 					 "usr", libdir, "ccache", "bin") + ":" + mysettings["PATH"]
 
+		if 'MAKEOPTS' not in mysettings:
+			nproc = get_cpu_count()
+			if nproc:
+				mysettings['MAKEOPTS'] = '-j%d' % (nproc + 1)
+
 		if not eapi_exports_KV(eapi):
 			# Discard KV for EAPIs that don't support it. Cached KV is restored
 			# from the backupenv whenever config.reset() is called.
diff --git a/pym/portage/util/cpuinfo.py b/pym/portage/util/cpuinfo.py
new file mode 100644
index 0000000..157e2e7
--- /dev/null
+++ b/pym/portage/util/cpuinfo.py
@@ -0,0 +1,19 @@
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+__all__ = ['get_cpu_count']
+
+import multiprocessing
+
+
+def get_cpu_count():
+	"""
+	Try to obtain the number of CPUs available.
+
+	@return: Number of CPUs or None if unable to obtain.
+	"""
+
+	try:
+		return multiprocessing.cpu_count()
+	except NotImplementedError:
+		return None
-- 
2.2.1

Re: [gentoo-portage-dev] [PATCH] Default MAKEOPTS to -j(ncpus+1) when unset

[Zac Medico]

On 01/17/2015 02:45 AM, Michał Górny wrote:
> Default MAKEOPTS job number to (number of CPUs + 1) when it is not
> provided in the ebuild environment.
> 
> Suggested-By: Daniel Robbins <drobbins@funtoo.org>
> ---
>  pym/portage/package/ebuild/doebuild.py |  8 +++++++-
>  pym/portage/util/cpuinfo.py            | 19 +++++++++++++++++++
>  2 files changed, 26 insertions(+), 1 deletion(-)
>  create mode 100644 pym/portage/util/cpuinfo.py

LGTM.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] Default MAKEOPTS to -j(ncpus+1) when unset

[Zac Medico]

On 01/17/2015 08:22 PM, Zac Medico wrote:
> On 01/17/2015 02:45 AM, Michał Górny wrote:
>> Default MAKEOPTS job number to (number of CPUs + 1) when it is not
>> provided in the ebuild environment.
>>
>> Suggested-By: Daniel Robbins <drobbins@funtoo.org>
>> ---
>>  pym/portage/package/ebuild/doebuild.py |  8 +++++++-
>>  pym/portage/util/cpuinfo.py            | 19 +++++++++++++++++++
>>  2 files changed, 26 insertions(+), 1 deletion(-)
>>  create mode 100644 pym/portage/util/cpuinfo.py
> 
> LGTM.
> 

Actually, Arfrever tells me that the multiprocessing module is not
available if python is built without threading support. So, we need to
handle the ImportError and either do nothing or parse /proc/cpuinfo or
something like that.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] Default MAKEOPTS to -j(ncpus+1) when unset

[Tim Harder]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

On 2015-01-17 23:48, Zac Medico wrote:
> Actually, Arfrever tells me that the multiprocessing module is not
> available if python is built without threading support. So, we need to
> handle the ImportError and either do nothing or parse /proc/cpuinfo or
> something like that.

Feel free to borrow/steal/copy the relevant methods from
snakeoil.process since we parse /proc/cpuinfo there probably how you
want. Patches/pull reqs welcome if you have improvements.

Thanks,
Tim

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-portage-dev] [PATCH] Default MAKEOPTS to -j(ncpus+1) when unset

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

Dnia 2015-01-17, o godz. 20:48:11
Zac Medico <zmedico@gentoo.org> napisał(a):

> On 01/17/2015 08:22 PM, Zac Medico wrote:
> > On 01/17/2015 02:45 AM, Michał Górny wrote:
> >> Default MAKEOPTS job number to (number of CPUs + 1) when it is not
> >> provided in the ebuild environment.
> >>
> >> Suggested-By: Daniel Robbins <drobbins@funtoo.org>
> >> ---
> >>  pym/portage/package/ebuild/doebuild.py |  8 +++++++-
> >>  pym/portage/util/cpuinfo.py            | 19 +++++++++++++++++++
> >>  2 files changed, 26 insertions(+), 1 deletion(-)
> >>  create mode 100644 pym/portage/util/cpuinfo.py
> > 
> > LGTM.
> > 
> 
> Actually, Arfrever tells me that the multiprocessing module is not
> available if python is built without threading support. So, we need to
> handle the ImportError and either do nothing or parse /proc/cpuinfo or
> something like that.

I don't think there's a point in having more complexity for a use case
that doesn't really get used. Except maybe for the few 'USE=-*' people
who aren't cautious enough but they're going to be hit by some failure
sooner or later.

I'm going to remove the flag on the Python end after verifying
the USE-deps.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-portage-dev] [PATCH v2] Default MAKEOPTS to -j(ncpus+1) when unset

[Michał Górny]

Default MAKEOPTS job number to (number of CPUs + 1) when it is not
provided in the ebuild environment.

Suggested-By: Daniel Robbins <drobbins@funtoo.org>
---
 pym/portage/package/ebuild/doebuild.py |  8 +++++++-
 pym/portage/util/cpuinfo.py            | 18 ++++++++++++++++++
 2 files changed, 25 insertions(+), 1 deletion(-)
 create mode 100644 pym/portage/util/cpuinfo.py

diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index bf97660..f43dddc 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2013 Gentoo Foundation
+# Copyright 2010-2015 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -66,6 +66,7 @@ from portage.package.ebuild.prepare_build_dirs import prepare_build_dirs
 from portage.util import apply_recursive_permissions, \
 	apply_secpass_permissions, noiselimit, normalize_path, \
 	writemsg, writemsg_stdout, write_atomic
+from portage.util.cpuinfo import get_cpu_count
 from portage.util.lafilefixer import rewrite_lafile	
 from portage.versions import _pkgsplit
 from _emerge.BinpkgEnvExtractor import BinpkgEnvExtractor
@@ -463,6 +464,11 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 				mysettings["PATH"] = os.path.join(os.sep, eprefix_lstrip,
 					 "usr", libdir, "ccache", "bin") + ":" + mysettings["PATH"]
 
+		if 'MAKEOPTS' not in mysettings:
+			nproc = get_cpu_count()
+			if nproc:
+				mysettings['MAKEOPTS'] = '-j%d' % (nproc + 1)
+
 		if not eapi_exports_KV(eapi):
 			# Discard KV for EAPIs that don't support it. Cached KV is restored
 			# from the backupenv whenever config.reset() is called.
diff --git a/pym/portage/util/cpuinfo.py b/pym/portage/util/cpuinfo.py
new file mode 100644
index 0000000..669e707
--- /dev/null
+++ b/pym/portage/util/cpuinfo.py
@@ -0,0 +1,18 @@
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+__all__ = ['get_cpu_count']
+
+
+def get_cpu_count():
+	"""
+	Try to obtain the number of CPUs available.
+
+	@return: Number of CPUs or None if unable to obtain.
+	"""
+
+	try:
+		import multiprocessing
+		return multiprocessing.cpu_count()
+	except (ImportError, NotImplementedError):
+		return None
-- 
2.2.1

Re: [gentoo-portage-dev] [PATCH v2] Default MAKEOPTS to -j(ncpus+1) when unset

[Brian Dolbec]

On Sun, 18 Jan 2015 11:38:38 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Default MAKEOPTS job number to (number of CPUs + 1) when it is not
> provided in the ebuild environment.
> 
> Suggested-By: Daniel Robbins <drobbins@funtoo.org>
> ---
>  pym/portage/package/ebuild/doebuild.py |  8 +++++++-
>  pym/portage/util/cpuinfo.py            | 18 ++++++++++++++++++
>  2 files changed, 25 insertions(+), 1 deletion(-)
>  create mode 100644 pym/portage/util/cpuinfo.py
>


looks good, merge please

-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] [PATCH v2] Default MAKEOPTS to -j(ncpus+1) when unset

[Rick "Zero_Chaos" Farina]

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

On 01/18/2015 11:58 AM, Brian Dolbec wrote:
> On Sun, 18 Jan 2015 11:38:38 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
>> Default MAKEOPTS job number to (number of CPUs + 1) when it is not
>> provided in the ebuild environment.
>>
>> Suggested-By: Daniel Robbins <drobbins@funtoo.org>
>> ---
>>  pym/portage/package/ebuild/doebuild.py |  8 +++++++-
>>  pym/portage/util/cpuinfo.py            | 18 ++++++++++++++++++
>>  2 files changed, 25 insertions(+), 1 deletion(-)
>>  create mode 100644 pym/portage/util/cpuinfo.py
>>
> 
> 
> looks good, merge please
> 
I realize I'm a little bit late here, but after extensive testing with
multiple different schedulers that NUMCPU provides equivalent
performance, or even outperforms NUMCPU+1.  I think that changes from 1
cpu to NUMCPU is good enough, and we don't need to test thrashing the
scheduler.

Thanks,
Zero


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-portage-dev] [PATCH v2] Default MAKEOPTS to -j(ncpus+1) when unset

[Zac Medico]

On 01/28/2015 07:06 PM, Rick "Zero_Chaos" Farina wrote:
> On 01/18/2015 11:58 AM, Brian Dolbec wrote:
>> On Sun, 18 Jan 2015 11:38:38 +0100
>> Michał Górny <mgorny@gentoo.org> wrote:
>>
>>> Default MAKEOPTS job number to (number of CPUs + 1) when it is not
>>> provided in the ebuild environment.
>>>
>>> Suggested-By: Daniel Robbins <drobbins@funtoo.org>
>>> ---
>>>  pym/portage/package/ebuild/doebuild.py |  8 +++++++-
>>>  pym/portage/util/cpuinfo.py            | 18 ++++++++++++++++++
>>>  2 files changed, 25 insertions(+), 1 deletion(-)
>>>  create mode 100644 pym/portage/util/cpuinfo.py
>>>
>>
>>
>> looks good, merge please
>>
> I realize I'm a little bit late here, but after extensive testing with
> multiple different schedulers that NUMCPU provides equivalent
> performance, or even outperforms NUMCPU+1.  I think that changes from 1
> cpu to NUMCPU is good enough, and we don't need to test thrashing the
> scheduler.

Yeah, that's what I was thinking when the patch was submitted, but I
forgot to mention it.
-- 
Thanks,
Zac