[gentoo-dev] Setting number of parallel builds for other build-systems than 'make'

[gentoo-dev] Setting number of parallel builds for other build-systems than 'make'

[Tiziano Müller]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone

It seems that setting the number of parallel builds using '-jN' does not
only work for make, but also for scons and bjam (and maybe others as well).
Since it isn't save to assume that '-jN' is the only option in MAKEOPTS,
some filtering is needed.
Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf and used
whenever one of those build-systems is being used. But we would probably
have to add a ...OPTS for every build-system.

What do you think? Better ideas?

Greets,
Tiziano

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFH7z7GwVqY66cHjARAm9cAJ9ia3Og4tdXQzYzeoWD3BUvQYM4/wCeIX7U
5x0so3XwIAj/gWCwD6xNaCw=
=EFaX
-----END PGP SIGNATURE-----

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Setting number of parallel builds for other build-systems than 'make'

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 322 bytes --]

On Sunday 01 October 2006 15:04, Tiziano Müller wrote:
> What do you think? Better ideas?
Look at app-i18n/skim . I do parse MAKEOPTS to use the -j* and -s options with 
scons.

-- 
Diego "Flameeyes" Pettenò - http://farragut.flameeyes.is-a-geek.org/
Gentoo/Alt lead, Gentoo/FreeBSD, Video, AMD64, Sound, PAM, KDE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Setting number of parallel builds for other build-systems than 'make'

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

On Monday 02 October 2006 17:32, Ilya A. Volynets-Evenbakh wrote:
> Actually this is not such a good idea. I usually use -l instead of -j, to
> limit number of jobs based on load average.
As there's no equivalent function in scons, the safest route is not to enable 
parallel jobs in that case.

-- 
Diego "Flameeyes" Pettenò - http://farragut.flameeyes.is-a-geek.org/
Gentoo/Alt lead, Gentoo/FreeBSD, Video, AMD64, Sound, PAM, KDE

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Setting number of parallel builds for other build-systems than 'make'

[Donnie Berkholz]

Tiziano Müller wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hi everyone
> 
> It seems that setting the number of parallel builds using '-jN' does not
> only work for make, but also for scons and bjam (and maybe others as well).
> Since it isn't save to assume that '-jN' is the only option in MAKEOPTS,
> some filtering is needed.
> Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf and used
> whenever one of those build-systems is being used. But we would probably
> have to add a ...OPTS for every build-system.
> 
> What do you think? Better ideas?

I think adding it for each build system is probably a good idea, 
nobody's guaranteeing option-level compatibility with make. Optionally 
falling back to using the few valid MAKEOPTS might be a nice addition.

Thanks,
Donnie
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Setting number of parallel builds for other build-systems than 'make'

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2343 bytes --]

On Sun, Oct 01, 2006 at 09:52:14AM -0700, Donnie Berkholz wrote:
> Tiziano Müller wrote:
> >-----BEGIN PGP SIGNED MESSAGE-----
> >Hash: SHA1
> >
> >Hi everyone
> >
> >It seems that setting the number of parallel builds using '-jN' does not
> >only work for make, but also for scons and bjam (and maybe others as well).
> >Since it isn't save to assume that '-jN' is the only option in MAKEOPTS,
> >some filtering is needed.
> >Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf and used
> >whenever one of those build-systems is being used. But we would probably
> >have to add a ...OPTS for every build-system.
> >
> >What do you think? Better ideas?
> 
> I think adding it for each build system is probably a good idea, 
> nobody's guaranteeing option-level compatibility with make. Optionally 
> falling back to using the few valid MAKEOPTS might be a nice addition.

I might be daft (likely), but why not just introduce a var indicating 
max parallelization instead?  Tweak portage to push that setting into 
MAKEOPTS="${MAKEOPTS+${MAKEOPTS} } -j${PARALLELIZATION}".

Might sound daft, but -j is hardcoded parallelization; if you're 
trying to run 3 build processes, the original var of -j isn't all that 
useful, nor will the hardcoded -j# var set for 3 package build 
processes be useful if you're doing a single build.

Depending on number of seperate package build processes possible, the 
# of build processes allocated per build process needs to vary 
(essentially).

Now... that's a bit ahead of what portage is doing atm, but folks 
*will* parallelize portage building (see bug 147516 if in doubt), so 
its not too far out.

Getting back to the actual topic, set this var, it's a raw int that a 
scons/bjam eclass can use to easily set appropriate var with the 
parallelization requested, thus unifying this particular knob; more 
importantly, it gives them a way to do what they're after while 
exposing a knob that the pkg manager can easily fiddle with for global 
parallelization needs.

Only downside to it I see is that it requires mangling the pkg manager 
to translate the parallelization setting into MAKEOPTS+=-j#; can't 
really get around that however due to the fact MAKEOPTS is already 
forced in installed configuration though.

Thoughts?
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Setting number of parallel builds for other build-systems than 'make'

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 461 bytes --]

On Sunday 01 October 2006 15:27, Brian Harring wrote:
> I might be daft (likely), but why not just introduce a var indicating
> max parallelization instead?

seems like the best thing to me ... then in things like GNOME packages, they 
can force the jobs to 1 rather than having to mung MAKEOPTS ...

i'd put the munging into the wrapper binaries ... so have `emake` parse the 
variable rather than screwing with MAKEOPTS in portage or profiles
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

[gentoo-dev] Re: Setting number of parallel builds for other build-systems than 'make'

[Tiziano Müller]

Ciaran McCreesh wrote:

> On Sun, 01 Oct 2006 15:04:52 +0200 Tiziano Müller <dev-zero@gentoo.org>
> wrote:
> | Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf
> | and used whenever one of those build-systems is being used. But we
> | would probably have to add a ...OPTS for every build-system.
> 
> Does BJAM parallelisation actually work reliably these days?
> 
Well, it seems to work for boost.


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Setting number of parallel builds for other build-systems than 'make'

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

On Sun, 01 Oct 2006 19:23:16 +0200 Tiziano Müller <dev-zero@gentoo.org>
wrote:
| Ciaran McCreesh wrote:
| > On Sun, 01 Oct 2006 15:04:52 +0200 Tiziano Müller
| > <dev-zero@gentoo.org> wrote:
| > | Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf
| > | and used whenever one of those build-systems is being used. But we
| > | would probably have to add a ...OPTS for every build-system.
| > 
| > Does BJAM parallelisation actually work reliably these days?
|
| Well, it seems to work for boost.

How well have you tested? It used to have issues on real SMP systems
with lots of CPUs (-j32), which is why I ask...

-- 
Ciaran McCreesh
Mail                : ciaranm at ciaranm.org
Web                 : http://ciaranm.org/
as-needed is broken : http://ciaranm.org/show_post.pl?post_id=13


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Re: Setting number of parallel builds for other build-systems than 'make'

[Tiziano Müller]

Ciaran McCreesh wrote:

> On Sun, 01 Oct 2006 19:23:16 +0200 Tiziano Müller <dev-zero@gentoo.org>
> wrote:
> | Ciaran McCreesh wrote:
> | > On Sun, 01 Oct 2006 15:04:52 +0200 Tiziano Müller
> | > <dev-zero@gentoo.org> wrote:
> | > | Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf
> | > | and used whenever one of those build-systems is being used. But we
> | > | would probably have to add a ...OPTS for every build-system.
> | > 
> | > Does BJAM parallelisation actually work reliably these days?
> |
> | Well, it seems to work for boost.
> 
> How well have you tested? It used to have issues on real SMP systems
> with lots of CPUs (-j32), which is why I ask...
> 
Well, I have a poor P4 CPU here and building with -j3. But if you have a
login for me on such a machine, I'll test it.

Tiziano


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Setting number of parallel builds for other build-systems than 'make'

[Ilya A. Volynets-Evenbakh]

Well, O2K is up and running, so someone can go ahead and
give it a shot at -j20


Ciaran McCreesh wrote:
> On Sun, 01 Oct 2006 19:23:16 +0200 Tiziano Müller <dev-zero@gentoo.org>
> wrote:
> | Ciaran McCreesh wrote:
> | > On Sun, 01 Oct 2006 15:04:52 +0200 Tiziano Müller
> | > <dev-zero@gentoo.org> wrote:
> | > | Now, SCONSOPTS (BJAMOPTS respectively) could be added to make.conf
> | > | and used whenever one of those build-systems is being used. But we
> | > | would probably have to add a ...OPTS for every build-system.
> | > 
> | > Does BJAM parallelisation actually work reliably these days?
> |
> | Well, it seems to work for boost.
>
> How well have you tested? It used to have issues on real SMP systems
> with lots of CPUs (-j32), which is why I ask...
>
>   

-- 
gentoo-dev@gentoo.org mailing list