[gentoo-dev] [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[gentoo-dev] [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]


Since not all the buildsystem we support use make for the actual build,
and they don't necessarily support make-like options (-jX -s and so on),
it would be nice to be able to express a JOBS variable that could be
used for parallel build with any build systems.

Right now there are ebuilds like openoffice or some scons-based ebuilds
that parse MAKEOPTS and get out of that the number of jobs from the -j
option, but this is a) suboptimal b) error-prone.

One has to consider people might be using -l for parallel building too,
for which reasons I'd be suggesting doing something like this to make
the change transparent:

 - ebuilds using non-make build systems would use JOBS;
 - ebuilds using make builds systems would just use emake as usual;
 - Portage takes care, if JOBS is unset, to parse it out of MAKEOPTS;
 - if user has set JOBS but not MAKEOPTS this defaults to -j${JOBS};
 - if user has JOBS and MAKEOPTS, MAKEOPTS keeps the same (for -l).

The result is that you can finally combine -l with parallel build on
OpenOffice and other packages, with a fallback number of maximum jobs
instead of using load-based decisions.

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/


[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Luca Barbato]

On 4-12-2008 21:29, Diego 'Flameeyes' Pettenò wrote:
> One has to consider people might be using -l for parallel building too,

I'd have it in a separate variable as well, IFF another build system is 
as nice as make towards parallel build.

lu

[gentoo-dev] Re: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Tiziano Müller]

Diego 'Flameeyes' =?utf-8?Q?Petten=C3=B2?= wrote:

>
> Since not all the buildsystem we support use make for the actual build,
> and they don't necessarily support make-like options (-jX -s and so on),
> it would be nice to be able to express a JOBS variable that could be
> used for parallel build with any build systems.
>
> Right now there are ebuilds like openoffice or some scons-based ebuilds
> that parse MAKEOPTS and get out of that the number of jobs from the -j
> option, but this is a) suboptimal b) error-prone.
>
> One has to consider people might be using -l for parallel building too,
> for which reasons I'd be suggesting doing something like this to make
> the change transparent:
>
>  - ebuilds using non-make build systems would use JOBS;
>  - ebuilds using make builds systems would just use emake as usual;
>  - Portage takes care, if JOBS is unset, to parse it out of MAKEOPTS;
>  - if user has set JOBS but not MAKEOPTS this defaults to -j${JOBS};
>  - if user has JOBS and MAKEOPTS, MAKEOPTS keeps the same (for -l).
>
> The result is that you can finally combine -l with parallel build on
> OpenOffice and other packages, with a fallback number of maximum jobs
> instead of using load-based decisions.

Sounds good for me.
What do you do for other build systems which also decide on load-basis
how many jobs to run? Parse again?
In that case I'd like to see a more abstract definition of "how many
jobs to run in parallel" which gets translated to the correct
make-options for emake and which is query'able either via a variable or
a function (something like JOBS="load=0.7||max=3" maybe?).

[gentoo-dev] Re: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

Tiziano Müller <dev-zero@gentoo.org> writes:

> What do you do for other build systems which also decide on load-basis
> how many jobs to run? Parse again?
> In that case I'd like to see a more abstract definition of "how many
> jobs to run in parallel" which gets translated to the correct
> make-options for emake and which is query'able either via a variable or
> a function (something like JOBS="load=0.7||max=3" maybe?).

If you can come up with a good enough syntax.

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Alec Warner]

On Thu, Dec 4, 2008 at 12:29 PM, Diego 'Flameeyes' Pettenò
<flameeyes@gmail.com> wrote:
>
> Since not all the buildsystem we support use make for the actual build,
> and they don't necessarily support make-like options (-jX -s and so on),
> it would be nice to be able to express a JOBS variable that could be
> used for parallel build with any build systems.
>
> Right now there are ebuilds like openoffice or some scons-based ebuilds
> that parse MAKEOPTS and get out of that the number of jobs from the -j
> option, but this is a) suboptimal b) error-prone.
>
> One has to consider people might be using -l for parallel building too,
> for which reasons I'd be suggesting doing something like this to make
> the change transparent:
>
>  - ebuilds using non-make build systems would use JOBS;
>  - ebuilds using make builds systems would just use emake as usual;
>  - Portage takes care, if JOBS is unset, to parse it out of MAKEOPTS;
>  - if user has set JOBS but not MAKEOPTS this defaults to -j${JOBS};
>  - if user has JOBS and MAKEOPTS, MAKEOPTS keeps the same (for -l).
>
> The result is that you can finally combine -l with parallel build on
> OpenOffice and other packages, with a fallback number of maximum jobs
> instead of using load-based decisions.

Looks Good To Me, but I would prefix the JOBS variable with some sort
of namespace (EJOBS, GENTOO_JOBS, etc.) to avoid conflicts with other
systems that may use JOBS internally already (seems vaguely likely).

-Alec

>
> --
> Diego "Flameeyes" Pettenò
> http://blog.flameeyes.eu/
>
>

[gentoo-dev] Re: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 372 bytes --]

"Alec Warner" <antarus@gentoo.org> writes:

> Looks Good To Me, but I would prefix the JOBS variable with some sort
> of namespace (EJOBS, GENTOO_JOBS, etc.) to avoid conflicts with other
> systems that may use JOBS internally already (seems vaguely likely).

Good point, GENTOO_JOBS sounds good to me.

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] Re: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Joe Peterson]

Diego 'Flameeyes' Pettenò wrote:
> "Alec Warner" <antarus@gentoo.org> writes:
> 
>> Looks Good To Me, but I would prefix the JOBS variable with some sort
>> of namespace (EJOBS, GENTOO_JOBS, etc.) to avoid conflicts with other
>> systems that may use JOBS internally already (seems vaguely likely).
> 
> Good point, GENTOO_JOBS sounds good to me.

How about "PORTAGE_JOBS" to go along with "PORTAGE_OVERLAY",
"PORTAGE_NICENESS", etc.

	-Joe

Re: [gentoo-dev] Re: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Rémi Cardona]

Le 05/12/2008 05:33, Joe Peterson a écrit :
> How about "PORTAGE_JOBS" to go along with "PORTAGE_OVERLAY",
> "PORTAGE_NICENESS", etc.

While this part of the thread has a lot of bikeshedding potential, Joe's 
name sounds more consistent with what we already have.

Naming issues appart, it's a good idea.

+1 from me :)

Rémi

Re: [gentoo-dev] Re: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Peter Volkov]

В Птн, 05/12/2008 в 08:23 +0100, Rémi Cardona пишет:
> Le 05/12/2008 05:33, Joe Peterson a écrit :
> > How about "PORTAGE_JOBS" to go along with "PORTAGE_OVERLAY",
> > "PORTAGE_NICENESS", etc.
> 
> While this part of the thread has a lot of bikeshedding potential, Joe's 
> name sounds more consistent with what we already have.

This variable will be used in eclasses so it's better to avoid portage
in it's name, since not only portage will work with it. GENTOO_JOBS is
really something new in sense of consistency. EJOBS looks most terse for
me but I'll be ok with any name...

For load average may be it's good to use another variable to avoid
overloading of "JOBS" meaning?

-- 
Peter.

[gentoo-dev] Re: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

Peter Volkov <pva@gentoo.org> writes:

> This variable will be used in eclasses so it's better to avoid portage
> in it's name, since not only portage will work with it. GENTOO_JOBS is
> really something new in sense of consistency. EJOBS looks most terse for
> me but I'll be ok with any name...

It's just I don't want to end up colliding with other stuff like
EclipseJobs or something else.

And my reason to not propose PORTAGE_JOBS is actually that it suggests
relation with -j option to emerge rather than make ;)

> For load average may be it's good to use another variable to avoid
> overloading of "JOBS" meaning?

Agreed, easier than ending up having to parse a variable content, after
all environment variables are kinda cheap.

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

[gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Ulrich Mueller]

>>>>> On Thu, 4 Dec 2008, Diego 'Flameeyes' Pettenò wrote:

> Since not all the buildsystem we support use make for the actual
> build, and they don't necessarily support make-like options (-jX -s
> and so on), it would be nice to be able to express a JOBS variable
> that could be used for parallel build with any build systems.

> Right now there are ebuilds like openoffice or some scons-based
> ebuilds that parse MAKEOPTS and get out of that the number of jobs
> from the -j option, but this is a) suboptimal b) error-prone.

> One has to consider people might be using -l for parallel building
> too, for which reasons I'd be suggesting doing something like this to
> make the change transparent:

>  - ebuilds using non-make build systems would use JOBS;
>  - ebuilds using make builds systems would just use emake as usual;
>  - Portage takes care, if JOBS is unset, to parse it out of MAKEOPTS;
>  - if user has set JOBS but not MAKEOPTS this defaults to -j${JOBS};
>  - if user has JOBS and MAKEOPTS, MAKEOPTS keeps the same (for -l).

> The result is that you can finally combine -l with parallel build on
> OpenOffice and other packages, with a fallback number of maximum jobs
> instead of using load-based decisions.

Coming back to this old topic [1]. Is there still consensus that we
should have such an EJOBS variable? (It shouldn't be called JOBS because
this name is too generic, see the old discussion.) Then we could add it
to EAPI 5.

Ulrich

[1] <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

On Fri, 31 Aug 2012 10:21:15 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:
> Coming back to this old topic [1]. Is there still consensus that we
> should have such an EJOBS variable? (It shouldn't be called JOBS
> because this name is too generic, see the old discussion.) Then we
> could add it to EAPI 5.
> 
> Ulrich
> 
> [1]
> <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>

If we're doing this, do we tell users to stop setting MAKEOPTS for
EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and
greater instead? Do we put fancy code in the package mangler to deal
with it?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Alexis Ballier]

On Fri, 31 Aug 2012 15:45:21 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Fri, 31 Aug 2012 10:21:15 +0200
> Ulrich Mueller <ulm@gentoo.org> wrote:
> > Coming back to this old topic [1]. Is there still consensus that we
> > should have such an EJOBS variable? (It shouldn't be called JOBS
> > because this name is too generic, see the old discussion.) Then we
> > could add it to EAPI 5.
> > 
> > Ulrich
> > 
> > [1]
> > <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>
> 
> If we're doing this, do we tell users to stop setting MAKEOPTS for
> EAPIs 5 and greater?

How can this work ? I cant think of any simple solution.

> Do we change the name of MAKEOPTS for EAPIs 5 and
> greater instead? Do we put fancy code in the package mangler to deal
> with it?

IMHO EAPI-5 compliant PMs should do MAKEOPTS="$MAKEOPTS -j$EJOBS" for
every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5
and greater.
This is retroactive but could be classified 'PM internals' so its fine
imho.

People using such a PM and not reading the news will get the old
MAKEOPTS which will still work with makefile based build systems but
will get serial builds for e.g. EAPI5 ebuilds + waf based build systems.
Not a very big deal.

A.

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Alexandre Rostovtsev]

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]

On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote:
> On Fri, 31 Aug 2012 10:21:15 +0200
> Ulrich Mueller <ulm@gentoo.org> wrote:
> > Coming back to this old topic [1]. Is there still consensus that we
> > should have such an EJOBS variable? (It shouldn't be called JOBS
> > because this name is too generic, see the old discussion.) Then we
> > could add it to EAPI 5.
> > 
> > Ulrich
> > 
> > [1]
> > <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>
> 
> If we're doing this, do we tell users to stop setting MAKEOPTS for
> EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs 5 and
> greater instead? Do we put fancy code in the package mangler to deal
> with it?

Users typically set MAKEOPTS systemwide in /etc/make.conf. If EJOBS will
have no effect for <EAPI5 ebuilds, then obviously we should not be
advising users to stop using MAKEOPTS until the whole tree has migrated
to EAPI5. And if EJOBS will be recognized by a future version of portage
for all EAPIs, then we still should allow MAKEOPTS because some users
may want to use --load-average.

Changing the name of MAKEOPTS in >=EAPI5 makes no sense. First, because
it's a standard environment variable used by gnu make. Second, because
having 3 different settings for parallel building (EJOBS, MAKEOPTS, and
"MAKEOPTS_EAPI5") would be insane.

Fancy code in the package manager would be the way to go IMHO. Ulrich's
message contains a reasonable description of the algorithm.

-Alexandre.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 31/08/12 11:27 AM, Alexandre Rostovtsev wrote:
> On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote:
>> On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller
>> <ulm@gentoo.org> wrote:
>>> Coming back to this old topic [1]. Is there still consensus
>>> that we should have such an EJOBS variable? (It shouldn't be
>>> called JOBS because this name is too generic, see the old
>>> discussion.) Then we could add it to EAPI 5.
>>> 
>>> Ulrich
>>> 
>>> [1] 
>>> <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>
>>
>>
>>> 
If we're doing this, do we tell users to stop setting MAKEOPTS for
>> EAPIs 5 and greater? Do we change the name of MAKEOPTS for EAPIs
>> 5 and greater instead? Do we put fancy code in the package
>> mangler to deal with it?
> 
> Users typically set MAKEOPTS systemwide in /etc/make.conf. If EJOBS
> will have no effect for <EAPI5 ebuilds, then obviously we should
> not be advising users to stop using MAKEOPTS until the whole tree
> has migrated to EAPI5. And if EJOBS will be recognized by a future
> version of portage for all EAPIs, then we still should allow
> MAKEOPTS because some users may want to use --load-average.
> 
> Changing the name of MAKEOPTS in >=EAPI5 makes no sense. First,
> because it's a standard environment variable used by gnu make.
> Second, because having 3 different settings for parallel building
> (EJOBS, MAKEOPTS, and "MAKEOPTS_EAPI5") would be insane.
> 
> Fancy code in the package manager would be the way to go IMHO.
> Ulrich's message contains a reasonable description of the
> algorithm.
> 
> -Alexandre.

I think, if i read the previous response to this correctly, that the
suggestion isn't the removal of MAKEOPTS, but simply that the '-j'
specification currently set in MAKEOPTS should instead be set in EJOBS
in everyone's make.conf.  This would then be appended to MAKEOPTS (for
all EAPI) -and- be used for non-make build systems (for EAPI>=5) alike.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBA4YUACgkQ2ugaI38ACPD96gD+Pu9f9SVG//0yhioO0LGP/W8o
sIGpiMFIEddXvhUsDAwA/0EJkZF8jrN7zmt/LdZy3nlCGKTIkPNxp5ukUGDDWIJB
=Dlem
-----END PGP SIGNATURE-----

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 31/08/12 12:08 PM, Ian Stakenvicius wrote:
> On 31/08/12 11:27 AM, Alexandre Rostovtsev wrote:
>> On Fri, 2012-08-31 at 15:45 +0100, Ciaran McCreesh wrote:
>>> On Fri, 31 Aug 2012 10:21:15 +0200 Ulrich Mueller 
>>> <ulm@gentoo.org> wrote:
>>>> Coming back to this old topic [1]. Is there still consensus 
>>>> that we should have such an EJOBS variable? (It shouldn't be 
>>>> called JOBS because this name is too generic, see the old 
>>>> discussion.) Then we could add it to EAPI 5.
>>>> 
>>>> Ulrich
>>>> 
>>>> [1] 
>>>> <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>
>>>
>>>
>>>>
>
>>>> 
If we're doing this, do we tell users to stop setting MAKEOPTS for
>>> EAPIs 5 and greater? Do we change the name of MAKEOPTS for
>>> EAPIs 5 and greater instead? Do we put fancy code in the
>>> package mangler to deal with it?
> 
>> Users typically set MAKEOPTS systemwide in /etc/make.conf. If
>> EJOBS will have no effect for <EAPI5 ebuilds, then obviously we
>> should not be advising users to stop using MAKEOPTS until the
>> whole tree has migrated to EAPI5. And if EJOBS will be recognized
>> by a future version of portage for all EAPIs, then we still
>> should allow MAKEOPTS because some users may want to use
>> --load-average.
> 
>> Changing the name of MAKEOPTS in >=EAPI5 makes no sense. First, 
>> because it's a standard environment variable used by gnu make. 
>> Second, because having 3 different settings for parallel
>> building (EJOBS, MAKEOPTS, and "MAKEOPTS_EAPI5") would be
>> insane.
> 
>> Fancy code in the package manager would be the way to go IMHO. 
>> Ulrich's message contains a reasonable description of the 
>> algorithm.
> 
>> -Alexandre.
> 
> I think, if i read the previous response to this correctly, that
> the suggestion isn't the removal of MAKEOPTS, but simply that the
> '-j' specification currently set in MAKEOPTS should instead be set
> in EJOBS in everyone's make.conf.  This would then be appended to
> MAKEOPTS (for all EAPI) -and- be used for non-make build systems
> (for EAPI>=5) alike.
> 


..hit send to soon...

So if users stick with setting -j in MAKEOPTS, then in EAPI=5 and
above this would only affect make-based builds; for parallel
compilation in non-make builds they would need to convert to using
EJOBS for EAPI=5 and above, otherwise those build systems will compile
serially instead of in parallel.





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBA4jQACgkQ2ugaI38ACPAFrAEArp7MM5w4Mv/TLKb058HzB9oN
NtQeSVoCQ8X5PuxjjJ0BAKbTJXEkLlZ0hMr09RyTKzK0XtdQq6cf2fbeFFgFb5eV
=+vtW
-----END PGP SIGNATURE-----

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Brian Harring]

On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote:
> On Fri, 31 Aug 2012 15:45:21 +0100
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> 
> > On Fri, 31 Aug 2012 10:21:15 +0200
> > Ulrich Mueller <ulm@gentoo.org> wrote:
> > > Coming back to this old topic [1]. Is there still consensus that we
> > > should have such an EJOBS variable? (It shouldn't be called JOBS
> > > because this name is too generic, see the old discussion.) Then we
> > > could add it to EAPI 5.
> > > 
> > > Ulrich
> > > 
> > > [1]
> > > <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>
> > 
> > If we're doing this, do we tell users to stop setting MAKEOPTS for
> > EAPIs 5 and greater?
> 
> How can this work ? I cant think of any simple solution.
> 
> > Do we change the name of MAKEOPTS for EAPIs 5 and
> > greater instead? Do we put fancy code in the package mangler to deal
> > with it?
> 
> IMHO EAPI-5 compliant PMs should do MAKEOPTS="$MAKEOPTS -j$EJOBS" for
> every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5
> and greater.
> This is retroactive but could be classified 'PM internals' so its fine
> imho.

This approach is fine imo, although I'd *potentially* look at adding a 
magic $PROC_COUNT var that is the # of cpu threads on the system; 
either that or defaulting jobs to it.

I rather dislike requiring users to go jam a 2/4/8 in there when it's 
easy to compute.  That said, it's minor.

Either way, yes, I think EJOBS should be in EAPI5.
~harring

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Michael Mol]

On Sat, Sep 1, 2012 at 8:20 PM, Brian Harring <ferringb@gmail.com> wrote:
> On Fri, Aug 31, 2012 at 11:12:44AM -0400, Alexis Ballier wrote:
>> On Fri, 31 Aug 2012 15:45:21 +0100
>> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
>>
>> > On Fri, 31 Aug 2012 10:21:15 +0200
>> > Ulrich Mueller <ulm@gentoo.org> wrote:
>> > > Coming back to this old topic [1]. Is there still consensus that we
>> > > should have such an EJOBS variable? (It shouldn't be called JOBS
>> > > because this name is too generic, see the old discussion.) Then we
>> > > could add it to EAPI 5.
>> > >
>> > > Ulrich
>> > >
>> > > [1]
>> > > <http://archives.gentoo.org/gentoo-dev/msg_750e33f68b16d971dff1f40dd9145e56.xml>
>> >
>> > If we're doing this, do we tell users to stop setting MAKEOPTS for
>> > EAPIs 5 and greater?
>>
>> How can this work ? I cant think of any simple solution.
>>
>> > Do we change the name of MAKEOPTS for EAPIs 5 and
>> > greater instead? Do we put fancy code in the package mangler to deal
>> > with it?
>>
>> IMHO EAPI-5 compliant PMs should do MAKEOPTS="$MAKEOPTS -j$EJOBS" for
>> every EAPI; using EJOBS from ebuilds/eclasses is allowed only in EAPI 5
>> and greater.
>> This is retroactive but could be classified 'PM internals' so its fine
>> imho.
>
> This approach is fine imo, although I'd *potentially* look at adding a
> magic $PROC_COUNT var that is the # of cpu threads on the system;
> either that or defaulting jobs to it.
>
> I rather dislike requiring users to go jam a 2/4/8 in there when it's
> easy to compute.  That said, it's minor.

On this point, I use dynamic load-based job counting. I'd be using
distcc, too, but I don't have the spare hardware for a second build
box right now.

For a single box, I target a load average of $PROC_COUNT, with a max
jobs in the vicinity of 2x$PROC_COUNT, to prevent I/O hangs from
causing job count explosions. (Where I/O hiccups aren't a concern,
I've found leaving --jobs open-ended worked fine, but builds always
eventually expand to consume available I/O as upstream things get
larger) I find this works very well; emerge and make both manage to
lurk near the optimum point of keeping the CPU busy without spending
too much time in context switches.

In a distcc context, I would most likely target a load average of
$LOCAL_PROC_COUNT, with a max jobs of
(2*$LOCAL_PROC_COUNT+$REMOTE_PROC_COUNT).

If ebuilds are try to be more aware of parallel-build contexts, I
think it's important they're not limited to static job counts.

-- 
:wq

Re: [gentoo-dev] EJOBS variable for EAPI 5? (was: [RFC] Create a JOBS variable to replace -jX in MAKEOPTS)

[Walter Dnes]

On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote

> This approach is fine imo, although I'd *potentially* look at adding a 
> magic $PROC_COUNT var that is the # of cpu threads on the system; 
> either that or defaulting jobs to it.
> 
> I rather dislike requiring users to go jam a 2/4/8 in there when it's 
> easy to compute.  That said, it's minor.
> 
> Either way, yes, I think EJOBS should be in EAPI5.

  One question about the suggested EJOBS variable; will it over-ride
MAKEOPTS?  Every so often on the Gentoo-user list, someone comes along
with a mysterious build failure.  The first suggestion is to reset
MAKEOPTS to -j1.  And on some occasions, that is indeed the solution to
the mysterious build failure.  Even the Gentoo manual agrees that the
"CPUs + 1" rule-of-thumb doesn't always work...
http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?full=1#installing_portagesays...

> With MAKEOPTS you define how many parallel compilations should occur
> when you install a package.  A good choice is the number of CPUs
> (or CPU cores) in your system plus one, ***BUT THIS GUIDELINE ISN'T
> ALWAYS PERFECT.*** (emphasis mine)

  I set -j1 and leave it that way.  Yes, the builds take longer, but the
resulting binary is just as fast.  And the amount of time I "save" will
be blown away the first time I end up screwing around a couple of hours
trying to fix a mysterious build failure.  That's why I want the user to
have the option of over-riding EJOBS, should it ever be implemented.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Zac Medico]

On 09/04/2012 04:00 AM, Walter Dnes wrote:
> On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote
> 
>> This approach is fine imo, although I'd *potentially* look at adding a 
>> magic $PROC_COUNT var that is the # of cpu threads on the system; 
>> either that or defaulting jobs to it.
>>
>> I rather dislike requiring users to go jam a 2/4/8 in there when it's 
>> easy to compute.  That said, it's minor.
>>
>> Either way, yes, I think EJOBS should be in EAPI5.
> 
>   One question about the suggested EJOBS variable; will it over-ride
> MAKEOPTS?  Every so often on the Gentoo-user list, someone comes along
> with a mysterious build failure.  The first suggestion is to reset
> MAKEOPTS to -j1.  And on some occasions, that is indeed the solution to
> the mysterious build failure.

That would be due to a missing dependency in the Makefiles, and using
-j1 is just a workaround. The ebuild can be hardcoded to use emake -j1
until the Makefile gets fixed.

>   I set -j1 and leave it that way.  Yes, the builds take longer, but the
> resulting binary is just as fast.  And the amount of time I "save" will
> be blown away the first time I end up screwing around a couple of hours
> trying to fix a mysterious build failure.  That's why I want the user to
> have the option of over-riding EJOBS, should it ever be implemented.

You could use EXTRA_EMAKE for that. You can do per-package settings via
/etc/portage/package.env.
-- 
Thanks,
Zac

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[vivo75]

Il 04/09/2012 19:15, Zac Medico ha scritto:
> On 09/04/2012 04:00 AM, Walter Dnes wrote:
>> On Sat, Sep 01, 2012 at 05:20:02PM -0700, Brian Harring wrote
>>
>>> This approach is fine imo, although I'd *potentially* look at adding a
>>> magic $PROC_COUNT var that is the # of cpu threads on the system;
>>> either that or defaulting jobs to it.
>>>
>>> I rather dislike requiring users to go jam a 2/4/8 in there when it's
>>> easy to compute.  That said, it's minor.
>>>
>>> Either way, yes, I think EJOBS should be in EAPI5.
>>    One question about the suggested EJOBS variable; will it over-ride
>> MAKEOPTS?  Every so often on the Gentoo-user list, someone comes along
>> with a mysterious build failure.  The first suggestion is to reset
>> MAKEOPTS to -j1.  And on some occasions, that is indeed the solution to
>> the mysterious build failure.
> That would be due to a missing dependency in the Makefiles, and using
> -j1 is just a workaround. The ebuild can be hardcoded to use emake -j1
> until the Makefile gets fixed.
>
>>    I set -j1 and leave it that way.  Yes, the builds take longer, but the
>> resulting binary is just as fast.  And the amount of time I "save" will
>> be blown away the first time I end up screwing around a couple of hours
>> trying to fix a mysterious build failure.  That's why I want the user to
>> have the option of over-riding EJOBS, should it ever be implemented.
> You could use EXTRA_EMAKE for that. You can do per-package settings via
> /etc/portage/package.env.

Dunno where to place this request, but if we go for something like EJOBS 
could we also make it phase specific?
So compile, install and test could have a different number of jobs running.
Possibly three different variables that override a predefined EJOBS.

TIA

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Zac Medico]

On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:
> Dunno where to place this request, but if we go for something like EJOBS
> could we also make it phase specific?
> So compile, install and test could have a different number of jobs running.
> Possibly three different variables that override a predefined EJOBS.

Per-phase sounds a little to fine-grained. Instead, I'd suggest to add
an ELOADAVG variable that's analogous to make's --load-average option.
That should be enough to compensate for any differences between phases.
-- 
Thanks,
Zac

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 11/09/12 12:43 PM, Zac Medico wrote:
> On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:
>> Dunno where to place this request, but if we go for something
>> like EJOBS could we also make it phase specific? So compile,
>> install and test could have a different number of jobs running. 
>> Possibly three different variables that override a predefined
>> EJOBS.
> 
> Per-phase sounds a little to fine-grained. Instead, I'd suggest to
> add an ELOADAVG variable that's analogous to make's --load-average
> option. That should be enough to compensate for any differences
> between phases.

I personally wonder about why this would be necessary from the
perspective of the user; if the user's system at emerge time can
handle X concurrent processes per emerge-job , i don't see why it
would matter what phase these jobs would be launched from.

At the ebuild level, certainly, but that's one of the reasons for
EJOBS in the first place, so that it can be overridden consistently
within a phase, if necessary for the ebuild (regardless of build
system type), right?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBPbLEACgkQ2ugaI38ACPA1qAD/bvjy7aB6nk5YboJHnCpQ8C56
QolKD9BPHL9eN8Xf41oA/iTZU+tyB+BDl+woZAlVGbaa6AR2r6Qp8rwOzkUWSwV/
=FhKc
-----END PGP SIGNATURE-----

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Zac Medico]

On 09/11/2012 09:54 AM, Ian Stakenvicius wrote:
> On 11/09/12 12:43 PM, Zac Medico wrote:
>> On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:
>>> Dunno where to place this request, but if we go for something
>>> like EJOBS could we also make it phase specific? So compile,
>>> install and test could have a different number of jobs running. 
>>> Possibly three different variables that override a predefined
>>> EJOBS.
> 
>> Per-phase sounds a little to fine-grained. Instead, I'd suggest to
>> add an ELOADAVG variable that's analogous to make's --load-average
>> option. That should be enough to compensate for any differences
>> between phases.
> 
> I personally wonder about why this would be necessary from the
> perspective of the user; if the user's system at emerge time can
> handle X concurrent processes per emerge-job , i don't see why it
> would matter what phase these jobs would be launched from.

Right, what matters is the system load, which is why I suggested ELOADAVG.

> At the ebuild level, certainly, but that's one of the reasons for
> EJOBS in the first place, so that it can be overridden consistently
> within a phase, if necessary for the ebuild (regardless of build
> system type), right?

Right. I'm surprised that ELOADAVG wasn't proposed in tandem with EJOBS
though, since overloading is not a good idea, and can happen easily any
time that you doing lots of things in parallel.
-- 
Thanks,
Zac

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Rich Freeman]

On Tue, Sep 11, 2012 at 1:07 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On 09/11/2012 09:54 AM, Ian Stakenvicius wrote:
>> At the ebuild level, certainly, but that's one of the reasons for
>> EJOBS in the first place, so that it can be overridden consistently
>> within a phase, if necessary for the ebuild (regardless of build
>> system type), right?
>
> Right. I'm surprised that ELOADAVG wasn't proposed in tandem with EJOBS
> though, since overloading is not a good idea, and can happen easily any
> time that you doing lots of things in parallel.

I tend to agree that load average matters more, although that doesn't
factor in RAM use.

I don't suggest that this is something that is easily remedied, but I
have run into a situation where WHAT you're doing greatly influences
how many jobs you can run - distcc.  I once tried to implement distcc
in a fairly large cluster, and then run make with VERY high levels of
parallelization - such as -j32.  That worked great - if the package
used C.  Then portage would try to build something that used ant and
suddenly the host was trying to run 32 jvms in parallel - just killing
the system.  Ditto for python/etc.  There was no way to tell the build
system to go nuts with anything using distcc but not everything else.

I think this is just a fundamental limitation of make - I don't
suggest that we can fix this at the portage level.  However, that is a
use case where WHAT you're doing influences how many jobs you can run.

Rich

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Gregory M. Turner]

On 9/11/2012 9:54 AM, Ian Stakenvicius wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 11/09/12 12:43 PM, Zac Medico wrote:
>> On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:
>>> Dunno where to place this request, but if we go for something
>>> like EJOBS could we also make it phase specific? So compile,
>>> install and test could have a different number of jobs running.
>>> Possibly three different variables that override a predefined
>>> EJOBS.
>>
>> Per-phase sounds a little to fine-grained. Instead, I'd suggest to
>> add an ELOADAVG variable that's analogous to make's --load-average
>> option. That should be enough to compensate for any differences
>> between phases.
>
> I personally wonder about why this would be necessary from the
> perspective of the user; if the user's system at emerge time can
> handle X concurrent processes per emerge-job , i don't see why it
> would matter what phase these jobs would be launched from.
>
> At the ebuild level, certainly, but that's one of the reasons for
> EJOBS in the first place, so that it can be overridden consistently
> within a phase, if necessary for the ebuild (regardless of build
> system type), right?

I would hope so.  There are definitely real-world reasons to want to 
restrict jobs -- usually to just one -- in a particular phase... several 
ebuilds, notably several in @system, do this already by injecting -j1 
somewhere.

To say it's all about performance seems to be forgetting a major reason 
somebody might want to do this as a user or an ebuild author.

Hmm...

I should preface this by saying that it really pains me to say this, 
because now this just starts to seem so fucking complicated -- but now 
that I think about it, this really seems to highlight a kind of semantic 
discrepancy that this thread has mostly dusted under the rug.

There are really TWO things we seem to care about -- one is some kind of 
global build-parallelism frob-set, and the other is the ability to turn 
off all parallelism for certain parts of an ebuild and those are 
actually kind-of orthogonal.

In other words, imagine if we had some kind of "global parallellism 
arbiter" thingy that looked at EJOBS or ELOADAVG or maybe some other 
things we haven't thought of yet, and decided things like:

   o should portage start a new parallel emerge process?
   o What is the appropriate "MAKEOPTS" for the "emake"
     that foo-1.0.ebuild just issued?

or even, one could imagine, such things as:

   o should portage decide that parallelism has gotten out of control
     somehow and suspend or kill a running ebuild for subsequent
     resuming or restarting?

   o should portage re-nice some of its processes?

Note that, effectively, we have this already, and it's called "portage".
But one could certainly make a case for modularizing it better, since, 
in truth, we are talking about a very common, very abstract problem here 
which portage shares with any number of batch-build systems.

Such an engine could very well do exactly the right thing if it were 
faced with a constraint that a certain part of a certain build needed to 
proceed without parallelism due to limitations coming from the build.

Also, there are very large parts of most builds -- configure comes to 
mind -- that don't parallelize even if, perhaps, they should.  In such 
cases, a really smart global parallelism arbiter could easily respond by 
spawning more jobs from other builds.

Not sure what I'm suggesting we do about it, exactly, but just pointing 
out that maybe a completely "correct" solution requires a much more 
elaborate implementation than just a bunch of syntactic sugar around 
what we currently call MAKEOPTS.  Whether or not Gentoo wants to take 
that all on, right now, as part of the next EAPI, is certainly debatable 
-- in fact, I'm inclined to say maybe it's not the best idea.

Perhaps we should really be asking: is the status-quo really so 
problematic/inelegant that it needs fixing? -- before we decide how best 
to fix it.

-gmt

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 12/09/12 05:55 AM, Gregory M. Turner wrote:
> 
> Note that, effectively, we have this already, and it's called
> "portage". But one could certainly make a case for modularizing it
> better, since, in truth, we are talking about a very common, very
> abstract problem here which portage shares with any number of
> batch-build systems.
> 
> Such an engine could very well do exactly the right thing if it
> were faced with a constraint that a certain part of a certain build
> needed to proceed without parallelism due to limitations coming
> from the build.
> 
> Also, there are very large parts of most builds -- configure comes
> to mind -- that don't parallelize even if, perhaps, they should.
> In such cases, a really smart global parallelism arbiter could
> easily respond by spawning more jobs from other builds.
> 

So essentially what you're saying here is that it might be worthwhile
to look into parallelism as a whole and possibly come up with a
solution that combines 'emerge --jobs' and build-system parallelism
together to maximum benefit?

Advanced HPC systems (sys-cluster/torque along with an appropriate
scheduler, for instance) can do such things with their jobs when the
jobs are properly built; I could see portage being able to handle this
as well given most of what is necessary is already known (ebuild
phases, build system type (via eclass), etc).   However, given the
limitations already put on parallelism in terms of emerge order, etc,
I could see this solution needing to be -very- complex and integration
needing to occur on multiple levels.  We'd also need to consider
distcc (and other cluster-shared compilation methods if there are
any??)..  It would be an interesting project, though.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBQhwAACgkQ2ugaI38ACPCKrgD+JNlHPUl7ETYDC6u3lYWRSz8J
fpWC/puDfCYu51yNOVIA/0E+U6x9Ds8GV8r/RinkTqss3/fcd06w24GRvZOda3Mj
=XONZ
-----END PGP SIGNATURE-----

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[vivo75]

Il 11/09/2012 18:43, Zac Medico ha scritto:
> On 09/11/2012 09:36 AM, vivo75@gmail.com wrote:
>> Dunno where to place this request, but if we go for something like EJOBS
>> could we also make it phase specific?
>> So compile, install and test could have a different number of jobs running.
>> Possibly three different variables that override a predefined EJOBS.
> Per-phase sounds a little to fine-grained. Instead, I'd suggest to add
> an ELOADAVG variable that's analogous to make's --load-average option.
> That should be enough to compensate for any differences between phases.
ok, but in my experience load-average really is too limited so I 
relaunch with the ability to control the following:
- disk io
- network
- memory
- cpu
- jobs

just tough that being able to control just jobs in a phase specific 
manner could have been sufficed ;-)
Also this seem is a good job for containers, already implemented in the 
linux kernel, but will let someone with experience with them comment on 
the mattter.

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Hans de Graaff]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]

On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote:

> So essentially what you're saying here is that it might be worthwhile
> to look into parallelism as a whole and possibly come up with a
> solution that combines 'emerge --jobs' and build-system parallelism
> together to maximum benefit?

Forget about jobs and load average, and just keep starting jobs all
around until there is only 20% (or whatever tuneable amount) free memory
left. As far as I can tell this is always the real bottleneck in the
end. Once you hit swap overall throughput has to go down quite a bit.

Hans

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Michael Mol]

[-- Attachment #1: Type: text/plain, Size: 1284 bytes --]

On Wed, Sep 12, 2012 at 12:33 PM, Hans de Graaff <graaff@gentoo.org> wrote:

> On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote:
>
> > So essentially what you're saying here is that it might be worthwhile
> > to look into parallelism as a whole and possibly come up with a
> > solution that combines 'emerge --jobs' and build-system parallelism
> > together to maximum benefit?
>
> Forget about jobs and load average, and just keep starting jobs all
> around until there is only 20% (or whatever tuneable amount) free memory
> left. As far as I can tell this is always the real bottleneck in the
> end. Once you hit swap overall throughput has to go down quite a bit.
>
> I've been thinking about this, but that only works until you get to the
huge link step of, e.g. chromium, firefox, libreoffice.

I've had programs with memory leaks in the past, but I've never seen a
program validly consume as much memory as ld during those builds.

To cover something like that, you would need to be able to freeze and swap
out an entire process (such as ld) to allow something else to complete
quickly...but there's no good way I can think of to prioritize sanely
between the one big process and the few dozen smaller ones which might be
allowed to spawn and complete first.

-- 
:wq

[-- Attachment #2: Type: text/html, Size: 1756 bytes --]

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Zac Medico]

On 09/12/2012 09:33 AM, Hans de Graaff wrote:
> On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote:
> 
>> So essentially what you're saying here is that it might be worthwhile
>> to look into parallelism as a whole and possibly come up with a
>> solution that combines 'emerge --jobs' and build-system parallelism
>> together to maximum benefit?
> 
> Forget about jobs and load average, and just keep starting jobs all
> around until there is only 20% (or whatever tuneable amount) free memory
> left. As far as I can tell this is always the real bottleneck in the
> end. Once you hit swap overall throughput has to go down quite a bit.

Well, I think it's still good to limit the number of jobs at least,
since otherwise you could become overloaded with processes that don't
consume a lot of memory at first but by the time they complete they have
consumed much more memory than desired (using swap).
-- 
Thanks,
Zac

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 12/09/12 12:58 PM, Zac Medico wrote:
> On 09/12/2012 09:33 AM, Hans de Graaff wrote:
>> On Wed, 2012-09-12 at 08:58 -0400, Ian Stakenvicius wrote:
>> 
>>> So essentially what you're saying here is that it might be
>>> worthwhile to look into parallelism as a whole and possibly
>>> come up with a solution that combines 'emerge --jobs' and
>>> build-system parallelism together to maximum benefit?
>> 
>> Forget about jobs and load average, and just keep starting jobs
>> all around until there is only 20% (or whatever tuneable amount)
>> free memory left. As far as I can tell this is always the real
>> bottleneck in the end. Once you hit swap overall throughput has
>> to go down quite a bit.
> 
> Well, I think it's still good to limit the number of jobs at
> least, since otherwise you could become overloaded with processes
> that don't consume a lot of memory at first but by the time they
> complete they have consumed much more memory than desired (using
> swap).

I think this would need to be dealt with by having the parent emerge
process monitor all children and specifically block individual
processes (ie, 'make' , 'ld' , etc) once resources are unavailable
until they become so.  Swap may be hit by the big processes but they
wouldn't continue to be processed while in swap, at least.

I don't have a solution to the potential 'thrashing' issue, though,
which i expect would be a problem even if there's enough memory.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBQwFoACgkQ2ugaI38ACPAiAwD/foU8Xw1BQM3jeO6IiVfCGOnw
xHtIufwVmMpsGVdJQRIA/3W7Utg92foSc6ZtKMzBP5Fj0qB2BXMt/RS2I4pLsCQm
=gy9K
-----END PGP SIGNATURE-----

Re: [gentoo-dev] EJOBS variable for EAPI 5?

[Gregory M. Turner]

On 9/12/2012 5:58 AM, Ian Stakenvicius wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 12/09/12 05:55 AM, Gregory M. Turner wrote:
>>
>> Note that, effectively, we have this already, and it's called
>> "portage". But one could certainly make a case for modularizing it
>> better, since, in truth, we are talking about a very common, very
>> abstract problem here which portage shares with any number of
>> batch-build systems.
>>
>> Such an engine could very well do exactly the right thing if it
>> were faced with a constraint that a certain part of a certain build
>> needed to proceed without parallelism due to limitations coming
>> from the build.
>>
>> Also, there are very large parts of most builds -- configure comes
>> to mind -- that don't parallelize even if, perhaps, they should.
>> In such cases, a really smart global parallelism arbiter could
>> easily respond by spawning more jobs from other builds.
>>
>
> So essentially what you're saying here is that it might be worthwhile
> to look into parallelism as a whole and possibly come up with a
> solution that combines 'emerge --jobs' and build-system parallelism
> together to maximum benefit?

Yeah, couldn't have said it better myself ... apparently :)

> Advanced HPC systems (sys-cluster/torque along with an appropriate
> scheduler, for instance) can do such things with their jobs when the
> jobs are properly built; I could see portage being able to handle this
> as well given most of what is necessary is already known (ebuild
> phases, build system type (via eclass), etc).   However, given the
> limitations already put on parallelism in terms of emerge order, etc,
> I could see this solution needing to be -very- complex and integration
> needing to occur on multiple levels.  We'd also need to consider
> distcc (and other cluster-shared compilation methods if there are
> any??)..  It would be an interesting project, though.

ACK all of the above.

Tempting to think more deeply about this but probably the last thing I 
need to do right now is to talk myself into another speculative project.

I've hurt my wrist a bit -- probably an RSI -- so should help deter me :S

Only a few major sources of parallelism exist in portage: --jobs / 
--load-average in emerge opts, multiprocessing eclass & equiv. ebuild 
helper, distcc, and make...  Infrastructure is already in place for all 
of those, so perhaps a good holistic solution exists that isn't /too/ 
complicated.

...OK another f!#!%$^ brainstorm incoming :)

For "JOBS" syntax... what really seems missing in portage are:

   o a clean way to say "dont parallelize this particular make
     invocation" in ebuilds

   o a clean way to globally say "try to use this parallelization
     strategy when emerging."

So what about something like:

   o EMERGE_JOBS and EMERGE_LOAD_AVERAGE make.conf vars equiv. to
     --jobs and --load-average emerge options

   o EBUILD_JOBS and EBUILD_LOAD_AVERAGE make.conf vars

   o If the latter are not specified, they are copied respectively from
     the former (debatable for *_JOBS, since now we get 16 processes when
     we asked for four).

   o MAKEOPTS is auto-extended to reflect EBUILD_JOBS/EBUILD_LOAD_AVERAGE
     if & only if -j|--jobs|-l|--load-average options aren't provided in
     make.conf/profile/envvar MAKEOPTS

   o however, if MAKEOPTS "override" EBUILD_JOBS or EBUILD_LOAD_AVERAGE,
     issue a conspicuous yellow-stars warning

   o extend "emake" to accept a "--non-parallel" option which will
     strip all -j|--jobs|-l|--load-average options from MAKEOPTS;
     perhaps support an equivalent EBUILD_NON_PARALLEL envvar as well,
     with support for override in profile.bashrc. Don't warn about this
     overriding EBUILD_JOBS -- treat as SOP.

   o debatable: respect EBUILD_NON_PARALLEL in multiprocessing, etc?
     or, perhaps, something like:

     EMAKE_NON_PARALLEL=${EMAKE_NON_PARALLEL:-${EBUILD_NON_PARALLEL:-no}}

     could be used to distinguish between "don't use any parallelism"
     and "don't use GNU's make parallelism in emake".  Also maybe a
     better name exists that doesn't use double-negatives.

?

Seems to me something vaguely like the above would provide

   o backward compatibility for ebuilds and make.conf

   o not so vastly different than what we have

   o a decent way to specify what "we really want" globally;
     insofar as portage doesn't do the best job effecting the requested
     parallelization strategy, more ambitious tactics could be
     implemented later, hopefully without huge interface revisions.

-gmt

P.S.:

(Kind-of-crazy additional idea: put ceil(sqrt(EMERGE_JOBS)) into 
EBUILD_JOBS when only the former is specified, and then let 
effective_emerge_jobs equal floor(EMERGE_JOBS/EBUILD_JOBS).... but maybe 
too much automagic for this to be a good idea.)