* [gentoo-python] [PATCH] Support overriding job-count for distutils-r1 multiprocessing.
@ 2013-01-01 16:57 Michał Górny
2013-01-01 19:15 ` [gentoo-python] " Mike Gilbert
0 siblings, 1 reply; 6+ messages in thread
From: Michał Górny @ 2013-01-01 16:57 UTC (permalink / raw
To: gentoo-python; +Cc: python, Michał Górny
---
gx86/eclass/distutils-r1.eclass | 44 ++++++++++++++++++++++++++++-------------
1 file changed, 30 insertions(+), 14 deletions(-)
diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
index e288c2c..3312505 100644
--- a/gx86/eclass/distutils-r1.eclass
+++ b/gx86/eclass/distutils-r1.eclass
@@ -68,6 +68,14 @@ if [[ ! ${_DISTUTILS_R1} ]]; then
RDEPEND=${PYTHON_DEPS}
DEPEND=${PYTHON_DEPS}
+# @ECLASS-VARIABLE: DISTUTILS_JOBS
+# @DEFAULT_UNSET
+# @DECRIPTION:
+# The number of parallel jobs to run for distutils-r1 parallel builds.
+# If unset, the job-count in ${MAKEOPTS} will be used.
+#
+# This variable is intended to be set in make.conf.
+
# @ECLASS-VARIABLE: PATCHES
# @DEFAULT_UNSET
# @DESCRIPTION:
@@ -144,15 +152,6 @@ DEPEND=${PYTHON_DEPS}
# This variable can be used to disable the afore-mentioned feature
# in case it causes issues with the package.
-#
-# If in-source builds are used, the eclass will create a copy of package
-# sources for each Python implementation in python_prepare_all(),
-# and work on that copy afterwards.
-#
-# If out-of-source builds are used, the eclass will instead work
-# on the sources directly, prepending setup.py arguments with
-# 'build --build-base ${BUILD_DIR}' to enforce keeping & using built
-# files in the specific root.
# @ECLASS-VARIABLE: mydistutilsargs
# @DEFAULT_UNSET
# @DESCRIPTION:
@@ -425,6 +424,23 @@ _distutils-r1_run_common_phase() {
"${@}"
}
+# @FUNCTION: _distutils-r1_multijob_init
+# @INTERNAL
+# @DESCRIPTION:
+# Init multijob, taking the job-count from ${DISTUTILS_JOBS}.
+_distutils-r1_multijob_init() {
+ debug-print-function ${FUNCNAME} "${@}"
+
+ local opts
+ if [[ ${DISTUTILS_JOBS} ]]; then
+ opts=-j${DISTUTILS_JOBS}
+ else
+ opts=${MAKEOPTS}
+ fi
+
+ multijob_init "${opts}"
+}
+
distutils-r1_src_prepare() {
debug-print-function ${FUNCNAME} "${@}"
@@ -435,7 +451,7 @@ distutils-r1_src_prepare() {
distutils-r1_python_prepare_all
fi
- multijob_init
+ _distutils-r1_multijob_init
if declare -f python_prepare >/dev/null; then
python_foreach_impl distutils-r1_run_phase python_prepare
else
@@ -445,7 +461,7 @@ distutils-r1_src_prepare() {
}
distutils-r1_src_configure() {
- multijob_init
+ _distutils-r1_multijob_init
if declare -f python_configure >/dev/null; then
python_foreach_impl distutils-r1_run_phase python_configure
else
@@ -461,7 +477,7 @@ distutils-r1_src_configure() {
distutils-r1_src_compile() {
debug-print-function ${FUNCNAME} "${@}"
- multijob_init
+ _distutils-r1_multijob_init
if declare -f python_compile >/dev/null; then
python_foreach_impl distutils-r1_run_phase python_compile
else
@@ -477,7 +493,7 @@ distutils-r1_src_compile() {
distutils-r1_src_test() {
debug-print-function ${FUNCNAME} "${@}"
- multijob_init
+ _distutils-r1_multijob_init
if declare -f python_test >/dev/null; then
python_foreach_impl distutils-r1_run_phase python_test
else
@@ -493,7 +509,7 @@ distutils-r1_src_test() {
distutils-r1_src_install() {
debug-print-function ${FUNCNAME} "${@}"
- multijob_init
+ _distutils-r1_multijob_init
if declare -f python_install >/dev/null; then
python_foreach_impl distutils-r1_run_phase python_install
else
--
1.8.0.2
^ permalink raw reply related [flat|nested] 6+ messages in thread* [gentoo-python] Re: [PATCH] Support overriding job-count for distutils-r1 multiprocessing.
2013-01-01 16:57 [gentoo-python] [PATCH] Support overriding job-count for distutils-r1 multiprocessing Michał Górny
@ 2013-01-01 19:15 ` Mike Gilbert
2013-01-01 19:55 ` Michał Górny
0 siblings, 1 reply; 6+ messages in thread
From: Mike Gilbert @ 2013-01-01 19:15 UTC (permalink / raw
To: gentoo-python
On Tue, Jan 1, 2013 at 11:57 AM, Michał Górny <mgorny@gentoo.org> wrote:
> ---
> gx86/eclass/distutils-r1.eclass | 44 ++++++++++++++++++++++++++++-------------
> 1 file changed, 30 insertions(+), 14 deletions(-)
>
> diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
> index e288c2c..3312505 100644
> --- a/gx86/eclass/distutils-r1.eclass
> +++ b/gx86/eclass/distutils-r1.eclass
> @@ -68,6 +68,14 @@ if [[ ! ${_DISTUTILS_R1} ]]; then
> RDEPEND=${PYTHON_DEPS}
> DEPEND=${PYTHON_DEPS}
>
> +# @ECLASS-VARIABLE: DISTUTILS_JOBS
> +# @DEFAULT_UNSET
> +# @DECRIPTION:
> +# The number of parallel jobs to run for distutils-r1 parallel builds.
> +# If unset, the job-count in ${MAKEOPTS} will be used.
> +#
> +# This variable is intended to be set in make.conf.
> +
> # @ECLASS-VARIABLE: PATCHES
> # @DEFAULT_UNSET
> # @DESCRIPTION:
> @@ -144,15 +152,6 @@ DEPEND=${PYTHON_DEPS}
> # This variable can be used to disable the afore-mentioned feature
> # in case it causes issues with the package.
>
> -#
> -# If in-source builds are used, the eclass will create a copy of package
> -# sources for each Python implementation in python_prepare_all(),
> -# and work on that copy afterwards.
> -#
> -# If out-of-source builds are used, the eclass will instead work
> -# on the sources directly, prepending setup.py arguments with
> -# 'build --build-base ${BUILD_DIR}' to enforce keeping & using built
> -# files in the specific root.
> # @ECLASS-VARIABLE: mydistutilsargs
> # @DEFAULT_UNSET
> # @DESCRIPTION:
> @@ -425,6 +424,23 @@ _distutils-r1_run_common_phase() {
> "${@}"
> }
>
> +# @FUNCTION: _distutils-r1_multijob_init
> +# @INTERNAL
> +# @DESCRIPTION:
> +# Init multijob, taking the job-count from ${DISTUTILS_JOBS}.
> +_distutils-r1_multijob_init() {
> + debug-print-function ${FUNCNAME} "${@}"
> +
> + local opts
> + if [[ ${DISTUTILS_JOBS} ]]; then
> + opts=-j${DISTUTILS_JOBS}
> + else
> + opts=${MAKEOPTS}
> + fi
> +
> + multijob_init "${opts}"
> +}
> +
> distutils-r1_src_prepare() {
> debug-print-function ${FUNCNAME} "${@}"
>
> @@ -435,7 +451,7 @@ distutils-r1_src_prepare() {
> distutils-r1_python_prepare_all
> fi
>
> - multijob_init
> + _distutils-r1_multijob_init
> if declare -f python_prepare >/dev/null; then
> python_foreach_impl distutils-r1_run_phase python_prepare
> else
> @@ -445,7 +461,7 @@ distutils-r1_src_prepare() {
> }
>
> distutils-r1_src_configure() {
> - multijob_init
> + _distutils-r1_multijob_init
> if declare -f python_configure >/dev/null; then
> python_foreach_impl distutils-r1_run_phase python_configure
> else
> @@ -461,7 +477,7 @@ distutils-r1_src_configure() {
> distutils-r1_src_compile() {
> debug-print-function ${FUNCNAME} "${@}"
>
> - multijob_init
> + _distutils-r1_multijob_init
> if declare -f python_compile >/dev/null; then
> python_foreach_impl distutils-r1_run_phase python_compile
> else
> @@ -477,7 +493,7 @@ distutils-r1_src_compile() {
> distutils-r1_src_test() {
> debug-print-function ${FUNCNAME} "${@}"
>
> - multijob_init
> + _distutils-r1_multijob_init
> if declare -f python_test >/dev/null; then
> python_foreach_impl distutils-r1_run_phase python_test
> else
> @@ -493,7 +509,7 @@ distutils-r1_src_test() {
> distutils-r1_src_install() {
> debug-print-function ${FUNCNAME} "${@}"
>
> - multijob_init
> + _distutils-r1_multijob_init
> if declare -f python_install >/dev/null; then
> python_foreach_impl distutils-r1_run_phase python_install
> else
> --
> 1.8.0.2
>
Seems ok. Was this requested by someone?
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [gentoo-python] Re: [PATCH] Support overriding job-count for distutils-r1 multiprocessing.
2013-01-01 19:15 ` [gentoo-python] " Mike Gilbert
@ 2013-01-01 19:55 ` Michał Górny
2013-01-01 20:15 ` Rafael Goncalves Martins
0 siblings, 1 reply; 6+ messages in thread
From: Michał Górny @ 2013-01-01 19:55 UTC (permalink / raw
To: Mike Gilbert; +Cc: gentoo-python
[-- Attachment #1: Type: text/plain, Size: 4419 bytes --]
On Tue, 1 Jan 2013 14:15:35 -0500
Mike Gilbert <floppym@gentoo.org> wrote:
> On Tue, Jan 1, 2013 at 11:57 AM, Michał Górny <mgorny@gentoo.org> wrote:
> > ---
> > gx86/eclass/distutils-r1.eclass | 44 ++++++++++++++++++++++++++++-------------
> > 1 file changed, 30 insertions(+), 14 deletions(-)
> >
> > diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
> > index e288c2c..3312505 100644
> > --- a/gx86/eclass/distutils-r1.eclass
> > +++ b/gx86/eclass/distutils-r1.eclass
> > @@ -68,6 +68,14 @@ if [[ ! ${_DISTUTILS_R1} ]]; then
> > RDEPEND=${PYTHON_DEPS}
> > DEPEND=${PYTHON_DEPS}
> >
> > +# @ECLASS-VARIABLE: DISTUTILS_JOBS
> > +# @DEFAULT_UNSET
> > +# @DECRIPTION:
> > +# The number of parallel jobs to run for distutils-r1 parallel builds.
> > +# If unset, the job-count in ${MAKEOPTS} will be used.
> > +#
> > +# This variable is intended to be set in make.conf.
> > +
> > # @ECLASS-VARIABLE: PATCHES
> > # @DEFAULT_UNSET
> > # @DESCRIPTION:
> > @@ -144,15 +152,6 @@ DEPEND=${PYTHON_DEPS}
> > # This variable can be used to disable the afore-mentioned feature
> > # in case it causes issues with the package.
> >
> > -#
> > -# If in-source builds are used, the eclass will create a copy of package
> > -# sources for each Python implementation in python_prepare_all(),
> > -# and work on that copy afterwards.
> > -#
> > -# If out-of-source builds are used, the eclass will instead work
> > -# on the sources directly, prepending setup.py arguments with
> > -# 'build --build-base ${BUILD_DIR}' to enforce keeping & using built
> > -# files in the specific root.
> > # @ECLASS-VARIABLE: mydistutilsargs
> > # @DEFAULT_UNSET
> > # @DESCRIPTION:
> > @@ -425,6 +424,23 @@ _distutils-r1_run_common_phase() {
> > "${@}"
> > }
> >
> > +# @FUNCTION: _distutils-r1_multijob_init
> > +# @INTERNAL
> > +# @DESCRIPTION:
> > +# Init multijob, taking the job-count from ${DISTUTILS_JOBS}.
> > +_distutils-r1_multijob_init() {
> > + debug-print-function ${FUNCNAME} "${@}"
> > +
> > + local opts
> > + if [[ ${DISTUTILS_JOBS} ]]; then
> > + opts=-j${DISTUTILS_JOBS}
> > + else
> > + opts=${MAKEOPTS}
> > + fi
> > +
> > + multijob_init "${opts}"
> > +}
> > +
> > distutils-r1_src_prepare() {
> > debug-print-function ${FUNCNAME} "${@}"
> >
> > @@ -435,7 +451,7 @@ distutils-r1_src_prepare() {
> > distutils-r1_python_prepare_all
> > fi
> >
> > - multijob_init
> > + _distutils-r1_multijob_init
> > if declare -f python_prepare >/dev/null; then
> > python_foreach_impl distutils-r1_run_phase python_prepare
> > else
> > @@ -445,7 +461,7 @@ distutils-r1_src_prepare() {
> > }
> >
> > distutils-r1_src_configure() {
> > - multijob_init
> > + _distutils-r1_multijob_init
> > if declare -f python_configure >/dev/null; then
> > python_foreach_impl distutils-r1_run_phase python_configure
> > else
> > @@ -461,7 +477,7 @@ distutils-r1_src_configure() {
> > distutils-r1_src_compile() {
> > debug-print-function ${FUNCNAME} "${@}"
> >
> > - multijob_init
> > + _distutils-r1_multijob_init
> > if declare -f python_compile >/dev/null; then
> > python_foreach_impl distutils-r1_run_phase python_compile
> > else
> > @@ -477,7 +493,7 @@ distutils-r1_src_compile() {
> > distutils-r1_src_test() {
> > debug-print-function ${FUNCNAME} "${@}"
> >
> > - multijob_init
> > + _distutils-r1_multijob_init
> > if declare -f python_test >/dev/null; then
> > python_foreach_impl distutils-r1_run_phase python_test
> > else
> > @@ -493,7 +509,7 @@ distutils-r1_src_test() {
> > distutils-r1_src_install() {
> > debug-print-function ${FUNCNAME} "${@}"
> >
> > - multijob_init
> > + _distutils-r1_multijob_init
> > if declare -f python_install >/dev/null; then
> > python_foreach_impl distutils-r1_run_phase python_install
> > else
> > --
> > 1.8.0.2
> >
>
> Seems ok. Was this requested by someone?
By me ;).
I have MAKEOPTS=-j6 because of distcc, and it kills my system with
nose's tests.
--
Best regards,
Michał Górny
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [gentoo-python] Re: [PATCH] Support overriding job-count for distutils-r1 multiprocessing.
2013-01-01 19:55 ` Michał Górny
@ 2013-01-01 20:15 ` Rafael Goncalves Martins
2013-01-01 21:17 ` Mike Gilbert
0 siblings, 1 reply; 6+ messages in thread
From: Rafael Goncalves Martins @ 2013-01-01 20:15 UTC (permalink / raw
To: Michał Górny; +Cc: Mike Gilbert, gentoo-python
On Tue, Jan 1, 2013 at 5:55 PM, Michał Górny <mgorny@gentoo.org> wrote:
> On Tue, 1 Jan 2013 14:15:35 -0500
> Mike Gilbert <floppym@gentoo.org> wrote:
>
>> On Tue, Jan 1, 2013 at 11:57 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> > ---
>> > gx86/eclass/distutils-r1.eclass | 44 ++++++++++++++++++++++++++++-------------
>> > 1 file changed, 30 insertions(+), 14 deletions(-)
>> >
>> > diff --git a/gx86/eclass/distutils-r1.eclass b/gx86/eclass/distutils-r1.eclass
>> > index e288c2c..3312505 100644
>> > --- a/gx86/eclass/distutils-r1.eclass
>> > +++ b/gx86/eclass/distutils-r1.eclass
>> > @@ -68,6 +68,14 @@ if [[ ! ${_DISTUTILS_R1} ]]; then
>> > RDEPEND=${PYTHON_DEPS}
>> > DEPEND=${PYTHON_DEPS}
>> >
>> > +# @ECLASS-VARIABLE: DISTUTILS_JOBS
>> > +# @DEFAULT_UNSET
>> > +# @DECRIPTION:
>> > +# The number of parallel jobs to run for distutils-r1 parallel builds.
>> > +# If unset, the job-count in ${MAKEOPTS} will be used.
>> > +#
>> > +# This variable is intended to be set in make.conf.
>> > +
>> > # @ECLASS-VARIABLE: PATCHES
>> > # @DEFAULT_UNSET
>> > # @DESCRIPTION:
>> > @@ -144,15 +152,6 @@ DEPEND=${PYTHON_DEPS}
>> > # This variable can be used to disable the afore-mentioned feature
>> > # in case it causes issues with the package.
>> >
>> > -#
>> > -# If in-source builds are used, the eclass will create a copy of package
>> > -# sources for each Python implementation in python_prepare_all(),
>> > -# and work on that copy afterwards.
>> > -#
>> > -# If out-of-source builds are used, the eclass will instead work
>> > -# on the sources directly, prepending setup.py arguments with
>> > -# 'build --build-base ${BUILD_DIR}' to enforce keeping & using built
>> > -# files in the specific root.
>> > # @ECLASS-VARIABLE: mydistutilsargs
>> > # @DEFAULT_UNSET
>> > # @DESCRIPTION:
>> > @@ -425,6 +424,23 @@ _distutils-r1_run_common_phase() {
>> > "${@}"
>> > }
>> >
>> > +# @FUNCTION: _distutils-r1_multijob_init
>> > +# @INTERNAL
>> > +# @DESCRIPTION:
>> > +# Init multijob, taking the job-count from ${DISTUTILS_JOBS}.
>> > +_distutils-r1_multijob_init() {
>> > + debug-print-function ${FUNCNAME} "${@}"
>> > +
>> > + local opts
>> > + if [[ ${DISTUTILS_JOBS} ]]; then
>> > + opts=-j${DISTUTILS_JOBS}
>> > + else
>> > + opts=${MAKEOPTS}
I don't know the eclass enough and the question may look silly for you
guys, but what happens if people adds other make options to MAKEOPTS
instead of just -jX ?
>> > + fi
>> > +
>> > + multijob_init "${opts}"
>> > +}
>> > +
>> > distutils-r1_src_prepare() {
>> > debug-print-function ${FUNCNAME} "${@}"
>> >
>> > @@ -435,7 +451,7 @@ distutils-r1_src_prepare() {
>> > distutils-r1_python_prepare_all
>> > fi
>> >
>> > - multijob_init
>> > + _distutils-r1_multijob_init
>> > if declare -f python_prepare >/dev/null; then
>> > python_foreach_impl distutils-r1_run_phase python_prepare
>> > else
>> > @@ -445,7 +461,7 @@ distutils-r1_src_prepare() {
>> > }
>> >
>> > distutils-r1_src_configure() {
>> > - multijob_init
>> > + _distutils-r1_multijob_init
>> > if declare -f python_configure >/dev/null; then
>> > python_foreach_impl distutils-r1_run_phase python_configure
>> > else
>> > @@ -461,7 +477,7 @@ distutils-r1_src_configure() {
>> > distutils-r1_src_compile() {
>> > debug-print-function ${FUNCNAME} "${@}"
>> >
>> > - multijob_init
>> > + _distutils-r1_multijob_init
>> > if declare -f python_compile >/dev/null; then
>> > python_foreach_impl distutils-r1_run_phase python_compile
>> > else
>> > @@ -477,7 +493,7 @@ distutils-r1_src_compile() {
>> > distutils-r1_src_test() {
>> > debug-print-function ${FUNCNAME} "${@}"
>> >
>> > - multijob_init
>> > + _distutils-r1_multijob_init
>> > if declare -f python_test >/dev/null; then
>> > python_foreach_impl distutils-r1_run_phase python_test
>> > else
>> > @@ -493,7 +509,7 @@ distutils-r1_src_test() {
>> > distutils-r1_src_install() {
>> > debug-print-function ${FUNCNAME} "${@}"
>> >
>> > - multijob_init
>> > + _distutils-r1_multijob_init
>> > if declare -f python_install >/dev/null; then
>> > python_foreach_impl distutils-r1_run_phase python_install
>> > else
>> > --
>> > 1.8.0.2
>> >
>>
>> Seems ok. Was this requested by someone?
>
> By me ;).
>
> I have MAKEOPTS=-j6 because of distcc, and it kills my system with
> nose's tests.
>
> --
> Best regards,
> Michał Górny
--
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [gentoo-python] Re: [PATCH] Support overriding job-count for distutils-r1 multiprocessing.
2013-01-01 20:15 ` Rafael Goncalves Martins
@ 2013-01-01 21:17 ` Mike Gilbert
2013-01-01 21:18 ` Rafael Goncalves Martins
0 siblings, 1 reply; 6+ messages in thread
From: Mike Gilbert @ 2013-01-01 21:17 UTC (permalink / raw
To: Rafael Goncalves Martins; +Cc: Michał Górny, gentoo-python
On Tue, Jan 1, 2013 at 3:15 PM, Rafael Goncalves Martins
<rafaelmartins@gentoo.org> wrote:
> I don't know the eclass enough and the question may look silly for you
> guys, but what happens if people adds other make options to MAKEOPTS
> instead of just -jX ?
We use the makeopts_jobs function from multiprocessing.eclass to parse
out the number of jobs from MAKEOPTS.
It currently ignores the load-average flag, mostly because there
really isn't any way to implement that in a generic way.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [gentoo-python] Re: [PATCH] Support overriding job-count for distutils-r1 multiprocessing.
2013-01-01 21:17 ` Mike Gilbert
@ 2013-01-01 21:18 ` Rafael Goncalves Martins
0 siblings, 0 replies; 6+ messages in thread
From: Rafael Goncalves Martins @ 2013-01-01 21:18 UTC (permalink / raw
To: Mike Gilbert; +Cc: Michał Górny, gentoo-python
On Tue, Jan 1, 2013 at 7:17 PM, Mike Gilbert <floppym@gentoo.org> wrote:
> On Tue, Jan 1, 2013 at 3:15 PM, Rafael Goncalves Martins
> <rafaelmartins@gentoo.org> wrote:
>> I don't know the eclass enough and the question may look silly for you
>> guys, but what happens if people adds other make options to MAKEOPTS
>> instead of just -jX ?
>
> We use the makeopts_jobs function from multiprocessing.eclass to parse
> out the number of jobs from MAKEOPTS.
>
> It currently ignores the load-average flag, mostly because there
> really isn't any way to implement that in a generic way.
Ah, got it! Thanks for explaining :)
--
Rafael Goncalves Martins
Gentoo Linux developer
http://rafaelmartins.eng.br/
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-01-01 21:18 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-01 16:57 [gentoo-python] [PATCH] Support overriding job-count for distutils-r1 multiprocessing Michał Górny
2013-01-01 19:15 ` [gentoo-python] " Mike Gilbert
2013-01-01 19:55 ` Michał Górny
2013-01-01 20:15 ` Rafael Goncalves Martins
2013-01-01 21:17 ` Mike Gilbert
2013-01-01 21:18 ` Rafael Goncalves Martins
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox