[gentoo-user] Simultaneously emerging multiple packages with same dependencies

[gentoo-user] Simultaneously emerging multiple packages with same dependencies

[PK]

[-- Attachment #1: Type: text/plain, Size: 492 bytes --]

Hi,

Is there any way to simultaneously emerge multiple packages (multiple
instances of 'emerge') that share common dependencies ?

I'm aware that portage uses locking mechanism before modifying 'world' file,
but what about the actual building process ? I'd expect emerge to check if
dependency package is already build/installed (or currently being build by
another instance) and just skip it in this case, however I haven't tried it
yet.. Can anybody shred some light on this ?

Thanks,
P.

[-- Attachment #2: Type: text/html, Size: 605 bytes --]

Re: [gentoo-user] Simultaneously emerging multiple packages with same dependencies

[Petri Rosenström]

On Wed, Jan 26, 2011 at 2:12 PM, PK <pkugrinas@gmail.com> wrote:
>
> Hi,
> Is there any way to simultaneously emerge multiple packages (multiple instances of 'emerge') that share common dependencies ?
> I'm aware that portage uses locking mechanism before modifying 'world' file, but what about the actual building process ? I'd expect emerge to check if dependency package is already build/installed (or currently being build by another instance) and just skip it in this case, however I haven't tried it yet.. Can anybody shred some light on this ?
> Thanks,
> P.

Hi,

--jobs isn't enough? example emerge -j kde-meta

Best regards
Petri Rosenström

Re: [gentoo-user] Simultaneously emerging multiple packages with same dependencies

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

On Wed, 26 Jan 2011 12:12:57 +0000, PK wrote:

> Is there any way to simultaneously emerge multiple packages (multiple
> instances of 'emerge') that share common dependencies ?

Don't use multiple instances of emerge, they'll probably all try to merge
the same dependencies. Do it the way the portage developers intended, use
the --jobs option with a single emerge instance.


-- 
Neil Bothwick

The severity of the itch is inversely proportional to the reach.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Simultaneously emerging multiple packages with same dependencies

[PK]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]

>
> Don't use multiple instances of emerge, they'll probably all try to merge
> the same dependencies. Do it the way the portage developers intended, use
> the --jobs option with a single emerge instance.
>
>
Sure, but sometimes while emerge is building a bunch of packages I remember
that I've forgotten to specify a few more, my only option then is to wait
till building process finishes. I think it should be possible to be more
efficient than that.

Cheers,
P.

[-- Attachment #2: Type: text/html, Size: 736 bytes --]

Re: [gentoo-user] Simultaneously emerging multiple packages with same dependencies

[Jacob Todd]

[-- Attachment #1: Type: text/plain, Size: 581 bytes --]

Then ^c and add/remove what you need.
On Jan 26, 2011 1:30 PM, "PK" <pkugrinas@gmail.com> wrote:
>>
>> Don't use multiple instances of emerge, they'll probably all try to merge
>> the same dependencies. Do it the way the portage developers intended, use
>> the --jobs option with a single emerge instance.
>>
>>
> Sure, but sometimes while emerge is building a bunch of packages I
remember
> that I've forgotten to specify a few more, my only option then is to wait
> till building process finishes. I think it should be possible to be more
> efficient than that.
>
> Cheers,
> P.

[-- Attachment #2: Type: text/html, Size: 816 bytes --]

[gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Nikos Chantziaras]

On 01/26/2011 02:12 PM, PK wrote:
> Hi,
>
> Is there any way to simultaneously emerge multiple packages (multiple
> instances of 'emerge') that share common dependencies ?
>
> I'm aware that portage uses locking mechanism before modifying 'world'
> file, but what about the actual building process ? I'd expect emerge to
> check if dependency package is already build/installed (or currently
> being build by another instance) and just skip it in this case, however
> I haven't tried it yet.. Can anybody shred some light on this ?

You can try, but the second instance with simply block until the lock 
has been removed.

I'm not aware of any package system that supports this.  I don't think 
adding support for this justifies the added complexity.

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

On Thu, 27 Jan 2011 07:12:24 +0200, Nikos Chantziaras wrote:

> > I'm aware that portage uses locking mechanism before modifying 'world'
> > file, but what about the actual building process ? I'd expect emerge
> > to check if dependency package is already build/installed (or
> > currently being build by another instance) and just skip it in this
> > case, however I haven't tried it yet.. Can anybody shred some light
> > on this ?  
> 
> You can try, but the second instance with simply block until the lock 
> has been removed.

The lock is not there for the entire emerge, I have run two emerges at
the same time, such as when I needed to install something while a world
update is in progress. It is possible, but not recommended as a general
strategy. That's what --jobs is for.


-- 
Neil Bothwick

Top Oxymorons Number 21: "Now, then ..."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Dale]

Neil Bothwick wrote:
> On Thu, 27 Jan 2011 07:12:24 +0200, Nikos Chantziaras wrote:
>
>    
>>> I'm aware that portage uses locking mechanism before modifying 'world'
>>> file, but what about the actual building process ? I'd expect emerge
>>> to check if dependency package is already build/installed (or
>>> currently being build by another instance) and just skip it in this
>>> case, however I haven't tried it yet.. Can anybody shred some light
>>> on this ?
>>>        
>> You can try, but the second instance with simply block until the lock
>> has been removed.
>>      
> The lock is not there for the entire emerge, I have run two emerges at
> the same time, such as when I needed to install something while a world
> update is in progress. It is possible, but not recommended as a general
> strategy. That's what --jobs is for.
>
>
>    

I have done the same thing and as long as the dependencies don't clash, 
it works fine.  However, if you start one emerge with a set of 
dependencies, then start another and they clash somewhere in the middle, 
portage has issues.  That is where the locks would kick in I guess.  I 
would also imagine that portage could emerge the same package twice 
too.  If one instance of emerge doesn't know what the other instance has 
already done, then the second one could emerge it again.  Doesn't emerge 
do all the calculating at the beginning and runs with that until the end?

I am using the -j option for the first time now.  I'm updating KDE.  It 
seems to work fine.  It doesn't scroll all the stuff like with a regular 
emerges but this new rig is so fast, I can't read it anyway.  I did have 
a package to fail and it spit out the error for me to read.

I agree, using --jobs is the best way to do this.  It works really well 
if you have a fast multi-core CPU.  I wish I had got me a 6 core one 
now.  ;-)

Dale

:-)  :-)

[gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Nikos Chantziaras]

On 01/27/2011 03:11 PM, Dale wrote:
> [...]
> I am using the -j option for the first time now. I'm updating KDE. It
> seems to work fine. It doesn't scroll all the stuff like with a regular
> emerges but this new rig is so fast, I can't read it anyway. I did have
> a package to fail and it spit out the error for me to read.

You don't need that if you have MAKEOPTS set in your make.conf, which is 
preferred.  The -j option of emerge emerges multiple packages, while 
with MAKEOPTS set to "-j4" or whatever does a parallel build in the same 
package (meaning compiling multiple source files at the same time).

It's preferred because with "emerge -jN" the last package will only use 
one CPU, while with "-jN" in MAKEOPTS even the last package will use N 
CPUs.  Furthermore, emerge can't always build N packages at the same 
time because one can depend on the other, so it will have to wait until 
the dependency is built.

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[YoYo Siska]

On Thu, Jan 27, 2011 at 03:33:21PM +0200, Nikos Chantziaras wrote:
> On 01/27/2011 03:11 PM, Dale wrote:
> >[...]
> >I am using the -j option for the first time now. I'm updating KDE. It
> >seems to work fine. It doesn't scroll all the stuff like with a regular
> >emerges but this new rig is so fast, I can't read it anyway. I did have
> >a package to fail and it spit out the error for me to read.
> 
> You don't need that if you have MAKEOPTS set in your make.conf,
> which is preferred.  The -j option of emerge emerges multiple
> packages, while with MAKEOPTS set to "-j4" or whatever does a
> parallel build in the same package (meaning compiling multiple
> source files at the same time).
> 
> It's preferred because with "emerge -jN" the last package will only
> use one CPU, while with "-jN" in MAKEOPTS even the last package will
> use N CPUs.  Furthermore, emerge can't always build N packages at
> the same time because one can depend on the other, so it will have
> to wait until the dependency is built.

On the other hand, unpacking, configure and install stages are not
parallel and emerge can do those in parallel for different packages... 
The best would be somewhere in the middle ;)


There are also the load-average options to -j, i.e.:
MAKEOPTS="-j -l5" emerge -j --load-average=5  ....

which makes make spawn parallel processes while load average is below 5
and the same for emerge spawning parallel ebuilds (when make isn't
parallel enough)

yoyo

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[J. Roeleveld]

On Thursday 27 January 2011 15:05:25 YoYo Siska wrote:
> On Thu, Jan 27, 2011 at 03:33:21PM +0200, Nikos Chantziaras wrote:
> > On 01/27/2011 03:11 PM, Dale wrote:
> > >[...]
> > >I am using the -j option for the first time now. I'm updating KDE. It
> > >seems to work fine. It doesn't scroll all the stuff like with a regular
> > >emerges but this new rig is so fast, I can't read it anyway. I did have
> > >a package to fail and it spit out the error for me to read.
> > 
> > You don't need that if you have MAKEOPTS set in your make.conf,
> > which is preferred.  The -j option of emerge emerges multiple
> > packages, while with MAKEOPTS set to "-j4" or whatever does a
> > parallel build in the same package (meaning compiling multiple
> > source files at the same time).
> > 
> > It's preferred because with "emerge -jN" the last package will only
> > use one CPU, while with "-jN" in MAKEOPTS even the last package will
> > use N CPUs.  Furthermore, emerge can't always build N packages at
> > the same time because one can depend on the other, so it will have
> > to wait until the dependency is built.
> 
> On the other hand, unpacking, configure and install stages are not
> parallel and emerge can do those in parallel for different packages...
> The best would be somewhere in the middle ;)
> 
> 
> There are also the load-average options to -j, i.e.:
> MAKEOPTS="-j -l5" emerge -j --load-average=5  ....
> 
> which makes make spawn parallel processes while load average is below 5
> and the same for emerge spawning parallel ebuilds (when make isn't
> parallel enough)
> 
> yoyo

Hmmm... didn't know about that one yet.
Does that mean that by doing it like that, the emerge-process (and compile-
processes) will try to keep the load average at 5 and if that is lower, it 
will keep adding more processes?

--
Joost

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

On Thu, 27 Jan 2011 15:33:21 +0200, Nikos Chantziaras wrote:

> > I am using the -j option for the first time now. I'm updating KDE. It
> > seems to work fine. It doesn't scroll all the stuff like with a
> > regular emerges but this new rig is so fast, I can't read it anyway.
> > I did have a package to fail and it spit out the error for me to
> > read.  
> 
> You don't need that if you have MAKEOPTS set in your make.conf, which
> is preferred.  The -j option of emerge emerges multiple packages, while 
> with MAKEOPTS set to "-j4" or whatever does a parallel build in the
> same package (meaning compiling multiple source files at the same time).

And how many CPU cores are used during the configure and install stages?

Using --jobs does a better job of making use of your CPU because one
package can use it fully for compiling while another is configuring.


-- 
Neil Bothwick

Beware of the opinion of someone without any facts.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Nikos Chantziaras]

On 01/27/2011 04:16 PM, Neil Bothwick wrote:
> On Thu, 27 Jan 2011 15:33:21 +0200, Nikos Chantziaras wrote:
>
>>> I am using the -j option for the first time now. I'm updating KDE. It
>>> seems to work fine. It doesn't scroll all the stuff like with a
>>> regular emerges but this new rig is so fast, I can't read it anyway.
>>> I did have a package to fail and it spit out the error for me to
>>> read.
>>
>> You don't need that if you have MAKEOPTS set in your make.conf, which
>> is preferred.  The -j option of emerge emerges multiple packages, while
>> with MAKEOPTS set to "-j4" or whatever does a parallel build in the
>> same package (meaning compiling multiple source files at the same time).
>
> And how many CPU cores are used during the configure and install stages?
>
> Using --jobs does a better job of making use of your CPU because one
> package can use it fully for compiling while another is configuring.

And what about the last package?  The time you gained for faster 
configure and install (which don't take too much time anyway) is wasted 
again on the last package.

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Thu, 27 Jan 2011 16:30:30 +0200, Nikos Chantziaras wrote:

> > Using --jobs does a better job of making use of your CPU because one
> > package can use it fully for compiling while another is configuring.  
> 
> And what about the last package?  The time you gained for faster 
> configure and install (which don't take too much time anyway) is wasted 
> again on the last package.

So on a 20 package world update, only 19 are faster while the 20th runs
at the same speed? Where's the loss there? Even if the last were slower,
it would be worth it.

Note that I never use --jobs without an argument, 2 by default, although
I could let emerge decide how many parallel processes to run.


-- 
Neil Bothwick

Top Oxymorons Number 47: Act naturally

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Nikos Chantziaras]

On 01/27/2011 04:53 PM, Neil Bothwick wrote:
> On Thu, 27 Jan 2011 16:30:30 +0200, Nikos Chantziaras wrote:
>
>>> Using --jobs does a better job of making use of your CPU because one
>>> package can use it fully for compiling while another is configuring.
>>
>> And what about the last package?  The time you gained for faster
>> configure and install (which don't take too much time anyway) is wasted
>> again on the last package.
>
> So on a 20 package world update, only 19 are faster while the 20th runs
> at the same speed? Where's the loss there? Even if the last were slower,
> it would be worth it.

Given the amount of time unpack/configure/install of most packages needs 
(very short), my observation is that it would not be worth it.

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Paul Hartman]

On Thu, Jan 27, 2011 at 7:33 AM, Nikos Chantziaras <realnc@arcor.de> wrote:
> On 01/27/2011 03:11 PM, Dale wrote:
>>
>> [...]
>> I am using the -j option for the first time now. I'm updating KDE. It
>> seems to work fine. It doesn't scroll all the stuff like with a regular
>> emerges but this new rig is so fast, I can't read it anyway. I did have
>> a package to fail and it spit out the error for me to read.
>
> You don't need that if you have MAKEOPTS set in your make.conf, which is
> preferred.  The -j option of emerge emerges multiple packages, while with
> MAKEOPTS set to "-j4" or whatever does a parallel build in the same package
> (meaning compiling multiple source files at the same time).
>
> It's preferred because with "emerge -jN" the last package will only use one
> CPU, while with "-jN" in MAKEOPTS even the last package will use N CPUs.
>  Furthermore, emerge can't always build N packages at the same time because
> one can depend on the other, so it will have to wait until the dependency is
> built.

I use MAKEOPTS = -j12 along with emerge --jobs for portage. Needless
to say my CPU is fully utilized during an emerge of KDE. :)

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Alan McKinnon]

Apparently, though unproven, at 17:09 on Thursday 27 January 2011, Nikos 
Chantziaras did opine thusly:

> On 01/27/2011 04:53 PM, Neil Bothwick wrote:
> > On Thu, 27 Jan 2011 16:30:30 +0200, Nikos Chantziaras wrote:
> >>> Using --jobs does a better job of making use of your CPU because one
> >>> package can use it fully for compiling while another is configuring.
> >> 
> >> And what about the last package?  The time you gained for faster
> >> configure and install (which don't take too much time anyway) is wasted
> >> again on the last package.
> > 
> > So on a 20 package world update, only 19 are faster while the 20th runs
> > at the same speed? Where's the loss there? Even if the last were slower,
> > it would be worth it.
> 
> Given the amount of time unpack/configure/install of most packages needs
> (very short), my observation is that it would not be worth it.

KDE.

unpack/configure/install takes up a significant amount of time for KDE


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[PK]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Thu, Jan 27, 2011 at 5:12 AM, Nikos Chantziaras <realnc@arcor.de> wrote:

>
> You can try, but the second instance with simply block until the lock has
> been removed.
>
> I'm not aware of any package system that supports this.  I don't think
> adding support for this justifies the added complexity.
>
>
Correct me if I'm wrong but the only added complexity I can think of is an
additional lock for package (ebuild) which is currently being built. Emerge
would skip the locked packages during "calculating dependencies" and
building phases and possibly wait for them to install.

Just a naive idea though.

Cheers,
P.

[-- Attachment #2: Type: text/html, Size: 995 bytes --]

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote:

> > So on a 20 package world update, only 19 are faster while the 20th
> > runs at the same speed? Where's the loss there? Even if the last were
> > slower, it would be worth it.  
> 
> Given the amount of time unpack/configure/install of most packages
> needs (very short), my observation is that it would not be worth it.

Even if that were true, how much time would you have to save to justify
adding -j 2 to EMERGE_DEFAULTS in make.conf?

But it's not true, large packages spend a lot of time on these phases of
the install.


-- 
Neil Bothwick

Head: (n.) the part of a disk drive which detects sectors and decides
which of the two possible values to return: 'lose a turn' or 'bankrupt.'

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Dale]

Alan McKinnon wrote:
> Apparently, though unproven, at 17:09 on Thursday 27 January 2011, Nikos
> Chantziaras did opine thusly:
>
>    
>> Given the amount of time unpack/configure/install of most packages needs
>> (very short), my observation is that it would not be worth it.
>>      
> KDE.
>
> unpack/configure/install takes up a significant amount of time for KDE
>
>
>    

Putting reply in just one post this time.  This is a discussion now and 
not a technical problem.

The package that failed had nothing to do with it building more than one 
package at a time.  For some reason, it didn't have one of the patches 
downloaded.  I guess it was a failure between here and where the mirror 
is.  When I restarted the emerge, it found it and no problems from 
there.  It would have done the same thing if I wasn't using -j is the 
point here.

Tthis was a KDE upgrade, it saved a LOT of time.  Most of the time only 
a couple cores are really working especially when they are smaller 
packages.  When using the -j option, all 4 cores were running and was 
pretty busy all the time.  At one time, it was doing >20 packages at 
once.  I also noticed the hard drive light was pretty steady too.

All in all, using the -j option seems to have saved a lot of time here.  
This is a fairly new install so I can recall how long it took to install 
KDE the last time.  This was much faster.

Just reporting a real world experience here.  I wish I had got a 6 core 
CPU now for sure.  Maybe later.

Dale

:-)  :-)

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[David W Noon]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On Thu, 27 Jan 2011 11:10:02 +0100, Neil Bothwick wrote about Re:
[gentoo-user] Re: Simultaneously emerging multiple packages  with same
dependencies:

>On Thu, 27 Jan 2011 07:12:24 +0200, Nikos Chantziaras wrote:
[snip]
>> You can try, but the second instance with simply block until the
>> lock has been removed.
>
>The lock is not there for the entire emerge, I have run two emerges at
>the same time, such as when I needed to install something while a world
>update is in progress. It is possible, but not recommended as a general
>strategy. That's what --jobs is for.

Ditto here.

Moreover, it is safer to perform all the package installations in a
single emerge, as the command will schedule dependencies correctly
(most of the time).
-- 
Regards,

Dave  [RLU #314465]
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*
dwnoon@ntlworld.com (David W Noon)
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[YoYo Siska]

On Thu, Jan 27, 2011 at 03:12:49PM +0100, J. Roeleveld wrote:
> On Thursday 27 January 2011 15:05:25 YoYo Siska wrote:
> > On Thu, Jan 27, 2011 at 03:33:21PM +0200, Nikos Chantziaras wrote:
> > > On 01/27/2011 03:11 PM, Dale wrote:
> > > >[...]
> > > >I am using the -j option for the first time now. I'm updating KDE. It
> > > >seems to work fine. It doesn't scroll all the stuff like with a regular
> > > >emerges but this new rig is so fast, I can't read it anyway. I did have
> > > >a package to fail and it spit out the error for me to read.
> > > 
> > > You don't need that if you have MAKEOPTS set in your make.conf,
> > > which is preferred.  The -j option of emerge emerges multiple
> > > packages, while with MAKEOPTS set to "-j4" or whatever does a
> > > parallel build in the same package (meaning compiling multiple
> > > source files at the same time).
> > > 
> > > It's preferred because with "emerge -jN" the last package will only
> > > use one CPU, while with "-jN" in MAKEOPTS even the last package will
> > > use N CPUs.  Furthermore, emerge can't always build N packages at
> > > the same time because one can depend on the other, so it will have
> > > to wait until the dependency is built.
> > 
> > On the other hand, unpacking, configure and install stages are not
> > parallel and emerge can do those in parallel for different packages...
> > The best would be somewhere in the middle ;)
> > 
> > 
> > There are also the load-average options to -j, i.e.:
> > MAKEOPTS="-j -l5" emerge -j --load-average=5  ....
> > 
> > which makes make spawn parallel processes while load average is below 5
> > and the same for emerge spawning parallel ebuilds (when make isn't
> > parallel enough)
> > 
> > yoyo
> 
> Hmmm... didn't know about that one yet.
> Does that mean that by doing it like that, the emerge-process (and compile-
> processes) will try to keep the load average at 5 and if that is lower, it 
> will keep adding more processes?
> 


Yes.
It might not be perfect, but mostly it works pretty well.
Once make started 10 or so process, which ate all my ram, because I
forgot to reenable swap, when I was playing with something before that
:)

yoyo

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Allan Gottlieb]

On Thu, Jan 27 2011, Neil Bothwick wrote:

> On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote:
>
>> > So on a 20 package world update, only 19 are faster while the 20th
>> > runs at the same speed? Where's the loss there? Even if the last were
>> > slower, it would be worth it.  
>> 
>> Given the amount of time unpack/configure/install of most packages
>> needs (very short), my observation is that it would not be worth it.
>
> Even if that were true, how much time would you have to save to justify
> adding -j 2 to EMERGE_DEFAULTS in make.conf?
>
> But it's not true, large packages spend a lot of time on these phases of
> the install.

OK I'm convinced since I know that those phases do take noticeable time.

I have a "4" processor i7 model 620 (2 cores, doubled for
hyperthreading) and have set MAKEOPTS="-j5".
If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower
MAKEOPTS to 3 (to 4)?

thanks,
allan

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[J. Roeleveld]

On Thursday 27 January 2011 19:56:23 Allan Gottlieb wrote:
> On Thu, Jan 27 2011, Neil Bothwick wrote:
> > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote:
> >> > So on a 20 package world update, only 19 are faster while the 20th
> >> > runs at the same speed? Where's the loss there? Even if the last were
> >> > slower, it would be worth it.
> >> 
> >> Given the amount of time unpack/configure/install of most packages
> >> needs (very short), my observation is that it would not be worth it.
> > 
> > Even if that were true, how much time would you have to save to justify
> > adding -j 2 to EMERGE_DEFAULTS in make.conf?
> > 
> > But it's not true, large packages spend a lot of time on these phases of
> > the install.
> 
> OK I'm convinced since I know that those phases do take noticeable time.
> 
> I have a "4" processor i7 model 620 (2 cores, doubled for
> hyperthreading) and have set MAKEOPTS="-j5".
> If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower
> MAKEOPTS to 3 (to 4)?
> 
> thanks,
> allan

You could, as if you leave it at -j5, you can end up with 2 * 5 = 10 
processed, eg: similar as if running with MAKEOPTS="-j10"

I think the option that YoYo came with is a good compromise:
# MAKEOPTS="-j -l5" emerge -j --load-average=5

Next time I am doing a big upgrade, I'm going to test that to see how it 
behaves.

--
Joost

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Dale]

YoYo Siska wrote:
> Yes.
> It might not be perfect, but mostly it works pretty well.
> Once make started 10 or so process, which ate all my ram, because I
> forgot to reenable swap, when I was playing with something before that
> :)
>
> yoyo
>    

I noticed the same thing with mine.  It used a LOT of ram.  I have 4Gbs 
and it was up to about 3Gbs at one point and using some swap as well.  
I'm hoping to max out to 16Gbs as soon as I can.  May upgrade to a 6 
core CPU too.

I wonder how much faster it would be if the work directory is put on 
tmpfs?  With 16Gbs, that should work even for OOo.

Dale

:-)  :-)

[gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Nikos Chantziaras]

On 01/27/2011 09:41 PM, Dale wrote:
> YoYo Siska wrote:
>> Yes.
>> It might not be perfect, but mostly it works pretty well.
>> Once make started 10 or so process, which ate all my ram, because I
>> forgot to reenable swap, when I was playing with something before that
>> :)
>>
>> yoyo
>
> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs
> and it was up to about 3Gbs at one point and using some swap as well.
> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core
> CPU too.
>
> I wonder how much faster it would be if the work directory is put on
> tmpfs? With 16Gbs, that should work even for OOo.
>

Btw, if you're using more instances than the amount of CPUs, the result 
will be slow-down.

With the default kernel scheduler, best if amount of CPUs + 1.  (On a 
4-core, that's -j5).

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Paul Hartman]

On Thu, Jan 27, 2011 at 2:15 PM, Nikos Chantziaras <realnc@arcor.de> wrote:
> On 01/27/2011 09:41 PM, Dale wrote:
>>
>> YoYo Siska wrote:
>>>
>>> Yes.
>>> It might not be perfect, but mostly it works pretty well.
>>> Once make started 10 or so process, which ate all my ram, because I
>>> forgot to reenable swap, when I was playing with something before that
>>> :)
>>>
>>> yoyo
>>
>> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs
>> and it was up to about 3Gbs at one point and using some swap as well.
>> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core
>> CPU too.
>>
>> I wonder how much faster it would be if the work directory is put on
>> tmpfs? With 16Gbs, that should work even for OOo.
>>
>
> Btw, if you're using more instances than the amount of CPUs, the result will
> be slow-down.
>
> With the default kernel scheduler, best if amount of CPUs + 1.  (On a
> 4-core, that's -j5).

Once, when building my kernel, I accidentally forgot to specify the
number of makes and ran "make -j all". That was a really bad idea, the
system became totally unresponsive for quite a long time, much longer
than normal kernel build time, but it did eventually finish!

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[YoYo Siska]

On Thu, Jan 27, 2011 at 08:18:34PM +0100, J. Roeleveld wrote:
> On Thursday 27 January 2011 19:56:23 Allan Gottlieb wrote:
> > On Thu, Jan 27 2011, Neil Bothwick wrote:
> > > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote:
> > >> > So on a 20 package world update, only 19 are faster while the 20th
> > >> > runs at the same speed? Where's the loss there? Even if the last were
> > >> > slower, it would be worth it.
> > >> 
> > >> Given the amount of time unpack/configure/install of most packages
> > >> needs (very short), my observation is that it would not be worth it.
> > > 
> > > Even if that were true, how much time would you have to save to justify
> > > adding -j 2 to EMERGE_DEFAULTS in make.conf?
> > > 
> > > But it's not true, large packages spend a lot of time on these phases of
> > > the install.
> > 
> > OK I'm convinced since I know that those phases do take noticeable time.
> > 
> > I have a "4" processor i7 model 620 (2 cores, doubled for
> > hyperthreading) and have set MAKEOPTS="-j5".
> > If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower
> > MAKEOPTS to 3 (to 4)?
> > 
> > thanks,
> > allan
> 
> You could, as if you leave it at -j5, you can end up with 2 * 5 = 10 
> processed, eg: similar as if running with MAKEOPTS="-j10"
> 
> I think the option that YoYo came with is a good compromise:
> # MAKEOPTS="-j -l5" emerge -j --load-average=5
> 
> Next time I am doing a big upgrade, I'm going to test that to see how it 
> behaves.

I was just building the whole system for my notebook in a chroot on my
desktop machine ( I use FEATURES=buildpkg to build binary packages in
the chroot on a fast desktop machine and then upgrade the notebook with
the binary packages) and I used exactly that (-j -l5 for bot make and
emerge). Can't say if it really is better or not ;) but most of the time
all four cores were busy, though sometimes I saw even 6 or 7 gcc-s
simultaneously in top ;)
emerge was running 3 to 4 jobs most of the time, sometimes dropping to 1
and once I saw it emerging about 10 parallel packages ;) (mostly small
things, which I guess were doing a lot of
unpacking/configuring/installing but almost nothing of compiling ;)


btw, just now I got this error from dev-lang/v8:

SCons error: option -j: invalid integer value: '-l4'

seems scons honors  MAKEOPTS, but doesn't understand the "loadaverage"
version (-j -l4)


yoyo

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Bill Longman]

On 01/27/2011 12:15 PM, Nikos Chantziaras wrote:
> 
> Btw, if you're using more instances than the amount of CPUs, the result
> will be slow-down.
> 
> With the default kernel scheduler, best if amount of CPUs + 1.  (On a
> 4-core, that's -j5).

And if you use emerge's "--jobs 2", each of those jobs will get the
MAKEOPTS values sent to it. So, if you have MAKEOPTS="-s -j4" and you
use "--jobs 2" with emerge, then you'll get two jobs running "gcc -s
-j4". Enjoy.

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[J. Roeleveld]

On Thursday 27 January 2011 21:25:02 Paul Hartman wrote:
> On Thu, Jan 27, 2011 at 2:15 PM, Nikos Chantziaras <realnc@arcor.de> wrote:
> > On 01/27/2011 09:41 PM, Dale wrote:
> >> YoYo Siska wrote:
> >>> Yes.
> >>> It might not be perfect, but mostly it works pretty well.
> >>> Once make started 10 or so process, which ate all my ram, because I
> >>> forgot to reenable swap, when I was playing with something before that
> >>> 
> >>> :)
> >>> 
> >>> yoyo
> >> 
> >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs
> >> and it was up to about 3Gbs at one point and using some swap as well.
> >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core
> >> CPU too.
> >> 
> >> I wonder how much faster it would be if the work directory is put on
> >> tmpfs? With 16Gbs, that should work even for OOo.
> > 
> > Btw, if you're using more instances than the amount of CPUs, the result
> > will be slow-down.
> > 
> > With the default kernel scheduler, best if amount of CPUs + 1.  (On a
> > 4-core, that's -j5).
> 
> Once, when building my kernel, I accidentally forgot to specify the
> number of makes and ran "make -j all". That was a really bad idea, the
> system became totally unresponsive for quite a long time, much longer
> than normal kernel build time, but it did eventually finish!

I have found that multi-core systems with sufficient memory can handle "-j" 
(no value) a lot better then sindle-core systems. I do on occasion do it with 
the kernel and can still continue using the system. (For comparison, my 
desktop is a 4-core AMD64 with 8GB memory)

--
Joost

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[J. Roeleveld]

On Thursday 27 January 2011 22:06:30 YoYo Siska wrote:
> On Thu, Jan 27, 2011 at 08:18:34PM +0100, J. Roeleveld wrote:
> > On Thursday 27 January 2011 19:56:23 Allan Gottlieb wrote:
> > > On Thu, Jan 27 2011, Neil Bothwick wrote:
> > > > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote:
> > > >> > So on a 20 package world update, only 19 are faster while the 20th
> > > >> > runs at the same speed? Where's the loss there? Even if the last
> > > >> > were slower, it would be worth it.
> > > >> 
> > > >> Given the amount of time unpack/configure/install of most packages
> > > >> needs (very short), my observation is that it would not be worth it.
> > > > 
> > > > Even if that were true, how much time would you have to save to
> > > > justify adding -j 2 to EMERGE_DEFAULTS in make.conf?
> > > > 
> > > > But it's not true, large packages spend a lot of time on these phases
> > > > of the install.
> > > 
> > > OK I'm convinced since I know that those phases do take noticeable
> > > time.
> > > 
> > > I have a "4" processor i7 model 620 (2 cores, doubled for
> > > hyperthreading) and have set MAKEOPTS="-j5".
> > > If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower
> > > MAKEOPTS to 3 (to 4)?
> > > 
> > > thanks,
> > > allan
> > 
> > You could, as if you leave it at -j5, you can end up with 2 * 5 = 10
> > processed, eg: similar as if running with MAKEOPTS="-j10"
> > 
> > I think the option that YoYo came with is a good compromise:
> > # MAKEOPTS="-j -l5" emerge -j --load-average=5
> > 
> > Next time I am doing a big upgrade, I'm going to test that to see how it
> > behaves.
> 
> I was just building the whole system for my notebook in a chroot on my
> desktop machine ( I use FEATURES=buildpkg to build binary packages in
> the chroot on a fast desktop machine and then upgrade the notebook with
> the binary packages) and I used exactly that (-j -l5 for bot make and
> emerge). Can't say if it really is better or not ;) but most of the time
> all four cores were busy, though sometimes I saw even 6 or 7 gcc-s
> simultaneously in top ;)
> emerge was running 3 to 4 jobs most of the time, sometimes dropping to 1
> and once I saw it emerging about 10 parallel packages ;) (mostly small
> things, which I guess were doing a lot of
> unpacking/configuring/installing but almost nothing of compiling ;)
> 
> 
> btw, just now I got this error from dev-lang/v8:
> 
> SCons error: option -j: invalid integer value: '-l4'
> 
> seems scons honors  MAKEOPTS, but doesn't understand the "loadaverage"
> version (-j -l4)

You might be able to avoid this by using the "long" version in the MAKEOPTS 
for "-l"?

Eg:
MAKEOPTS="-j --load-average=5" emerge -j --load-average=5 

--
Joost

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Paul Hartman]

On Thu, Jan 27, 2011 at 3:46 PM, J. Roeleveld <joost@antarean.org> wrote:
> On Thursday 27 January 2011 21:25:02 Paul Hartman wrote:
>> On Thu, Jan 27, 2011 at 2:15 PM, Nikos Chantziaras <realnc@arcor.de> wrote:
>> > On 01/27/2011 09:41 PM, Dale wrote:
>> >> YoYo Siska wrote:
>> >>> Yes.
>> >>> It might not be perfect, but mostly it works pretty well.
>> >>> Once make started 10 or so process, which ate all my ram, because I
>> >>> forgot to reenable swap, when I was playing with something before that
>> >>>
>> >>> :)
>> >>>
>> >>> yoyo
>> >>
>> >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs
>> >> and it was up to about 3Gbs at one point and using some swap as well.
>> >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core
>> >> CPU too.
>> >>
>> >> I wonder how much faster it would be if the work directory is put on
>> >> tmpfs? With 16Gbs, that should work even for OOo.
>> >
>> > Btw, if you're using more instances than the amount of CPUs, the result
>> > will be slow-down.
>> >
>> > With the default kernel scheduler, best if amount of CPUs + 1.  (On a
>> > 4-core, that's -j5).
>>
>> Once, when building my kernel, I accidentally forgot to specify the
>> number of makes and ran "make -j all". That was a really bad idea, the
>> system became totally unresponsive for quite a long time, much longer
>> than normal kernel build time, but it did eventually finish!
>
> I have found that multi-core systems with sufficient memory can handle "-j"
> (no value) a lot better then sindle-core systems. I do on occasion do it with
> the kernel and can still continue using the system. (For comparison, my
> desktop is a 4-core AMD64 with 8GB memory)

Strange, in my case it was an i7 920 (4 cores, hyperthreaded, appears
as 8 CPUs to Linux) with 12GB of RAM. Maybe if I prefixed it
with"nice" it would not have brought my computer to its knees... or
maybe related to the schedulers and other kernel voodoo that I don't
understand. I might try it again someday :)

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[J. Roeleveld]

On Thursday 27 January 2011 22:06:30 YoYo Siska wrote:
> On Thu, Jan 27, 2011 at 08:18:34PM +0100, J. Roeleveld wrote:
> > On Thursday 27 January 2011 19:56:23 Allan Gottlieb wrote:
> > > On Thu, Jan 27 2011, Neil Bothwick wrote:
> > > > On Thu, 27 Jan 2011 17:09:27 +0200, Nikos Chantziaras wrote:
> > > >> > So on a 20 package world update, only 19 are faster while the 20th
> > > >> > runs at the same speed? Where's the loss there? Even if the last
> > > >> > were slower, it would be worth it.
> > > >> 
> > > >> Given the amount of time unpack/configure/install of most packages
> > > >> needs (very short), my observation is that it would not be worth it.
> > > > 
> > > > Even if that were true, how much time would you have to save to
> > > > justify adding -j 2 to EMERGE_DEFAULTS in make.conf?
> > > > 
> > > > But it's not true, large packages spend a lot of time on these phases
> > > > of the install.
> > > 
> > > OK I'm convinced since I know that those phases do take noticeable
> > > time.
> > > 
> > > I have a "4" processor i7 model 620 (2 cores, doubled for
> > > hyperthreading) and have set MAKEOPTS="-j5".
> > > If I add -jobs=2 to EMERGE_DEFAULT_OPTS, should I lower
> > > MAKEOPTS to 3 (to 4)?
> > > 
> > > thanks,
> > > allan
> > 
> > You could, as if you leave it at -j5, you can end up with 2 * 5 = 10
> > processed, eg: similar as if running with MAKEOPTS="-j10"
> > 
> > I think the option that YoYo came with is a good compromise:
> > # MAKEOPTS="-j -l5" emerge -j --load-average=5
> > 
> > Next time I am doing a big upgrade, I'm going to test that to see how it
> > behaves.
> 
> I was just building the whole system for my notebook in a chroot on my
> desktop machine ( I use FEATURES=buildpkg to build binary packages in
> the chroot on a fast desktop machine and then upgrade the notebook with
> the binary packages) and I used exactly that (-j -l5 for bot make and
> emerge). Can't say if it really is better or not ;) but most of the time
> all four cores were busy, though sometimes I saw even 6 or 7 gcc-s
> simultaneously in top ;)
> emerge was running 3 to 4 jobs most of the time, sometimes dropping to 1
> and once I saw it emerging about 10 parallel packages ;) (mostly small
> things, which I guess were doing a lot of
> unpacking/configuring/installing but almost nothing of compiling ;)
> 
> 
> btw, just now I got this error from dev-lang/v8:
> 
> SCons error: option -j: invalid integer value: '-l4'
> 
> seems scons honors  MAKEOPTS, but doesn't understand the "loadaverage"
> version (-j -l4)
> 
> 
> yoyo

Doing an update now, just had this:
***
top - 23:09:47 up 15:12,  4 users,  load average: 5.92, 3.62, 1.87
Tasks: 321 total,  35 running, 285 sleeping,   0 stopped,   1 zombie
Cpu(s): 89.5%us, 10.5%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8114712k total,  5266456k used,  2848256k free,   162600k buffers
Swap: 12582904k total,        0k used, 12582904k free,  2812008k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                         
24196 root      20   0 63756  39m 6932 R   15  0.5   0:00.45 cc1plus                                                                                                                                                                         
24092 root      20   0 65532  40m 6896 R   15  0.5   0:00.64 cc1plus                                                                                                                                                                         
24158 root      20   0 60572  36m 6832 R   15  0.5   0:00.50 cc1plus                                                                                                                                                                         
24187 root      20   0 61296  36m 6904 R   15  0.5   0:00.45 cc1plus                                                                                                                                                                         
24202 root      20   0 59404  34m 6832 R   14  0.4   0:00.43 cc1plus                                                                                                                                                                         
24096 root      20   0 61272  36m 6868 R   14  0.5   0:00.56 cc1plus                                                                                                                                                                         
24245 root      20   0 43364  20m 6372 R   12  0.3   0:00.36 cc1                                                                                                                                                                             
24240 root      20   0 41636  18m 6340 R   12  0.2   0:00.35 cc1                                                                                                                                                                             
24271 root      20   0 42776  19m 6296 R   11  0.2   0:00.34 cc1                                                                                                                                                                             
24281 root      20   0 41132  17m 6308 R    8  0.2   0:00.25 cc1                                                                                                                                                                             
24302 root      20   0 40996  17m 6268 R    8  0.2   0:00.24 cc1                                                                                                                                                                             
24320 root      20   0 42356  18m 6192 R    8  0.2   0:00.23 cc1                                                                                                                                                                             
24304 root      20   0 42000  18m 6832 R    7  0.2   0:00.22 cc1plus                                                                                                                                                                         
24331 root      20   0 52232  27m 6388 R    7  0.3   0:00.22 cc1plus                                                                                                                                                                         
24330 root      20   0 54068  27m 4016 R    6  0.3   0:00.19 cc1plus                                                                                                                                                                         
24338 root      20   0 47924  23m 6160 R    6  0.3   0:00.19 cc1plus                                                                                                                                                                         
24329 root      20   0 45212  20m 5852 R    6  0.3   0:00.18 cc1                                                                                                                                                                             
24335 root      20   0 51212  26m 6300 R    6  0.3   0:00.18 cc1plus                                                                                                                                                                         
24360 root      20   0 37316  14m 6240 R    6  0.2   0:00.17 cc1                                                                                                                                                                             
24344 root      20   0 48028  24m 6432 R    5  0.3   0:00.16 cc1plus                                                                                                                                                                         
24399 root      20   0 47652  20m 3532 R    4  0.3   0:00.13 cc1plus                                                                                                                                                                         
24377 root      20   0 45552  18m 3536 R    3  0.2   0:00.10 cc1plus                                                                                                                                                                         
24380 root      20   0 45536  18m 3528 R    3  0.2   0:00.10 cc1plus                                                                                                                                                                         
 4574 root      20   0  170m  62m  15m S    3  0.8   8:09.14 X                                                                                                                                                                               
24413 root      20   0 38308   9m 2620 R    1  0.1   0:00.04 cc1plus                                                                                                                                                                         
11522 joost     20   0 93144  36m  13m S    1  0.5   1:51.05 skype                                                                                                                                                                           
24417 root      20   0 36456 8628 2620 R    1  0.1   0:00.03 cc1plus                                                                                                                                                                         
24424 root      20   0 34724 8588 2908 R    1  0.1   0:00.03 cc1                                                                                                                                                                             
24426 root      20   0 35752 9088 2928 R    1  0.1   0:00.03 cc1                                                                                                                                                                             
11573 joost     20   0  849m 403m  31m S    1  5.1   3:07.35 firefox                                                                                                                                                                         
13388 root      20   0 30216 5144 1452 S    1  0.1   0:00.89 configure                                                                                                                                                                       
19920 root      20   0 20696 2316 1336 S    1  0.0   0:00.38 sh                                                                                                                                                                              
11490 joost     20   0  453m  46m  31m S    0  0.6   0:04.61 kopete                                                                                                                                                                          
12922 root      20   0  205m 102m 3788 S    0  1.3   0:21.50 emerge                                                                                                                                                                          
14463 joost     20   0  449m  40m  18m R    0  0.5   0:20.04 konsole                                                                                                                                                                         
18085 joost     20   0 38456 2604 1724 R    0  0.0   0:00.88 top                                                                                                                                                                             
22015 root      20   0 21732 3332  908 S    0  0.0   0:00.12 make                                                                                                                                                                            
24418 root      20   0 34708 7056 2620 R    0  0.1   0:00.01 cc1plus                                                                                                                                                                         
24429 root      20   0 34680 7048 2616 R    0  0.1   0:00.01 cc1plus                                                                                                                                                                         
***

Looks nice :)

--
Joost

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[J. Roeleveld]

On Thursday 27 January 2011 23:05:22 Paul Hartman wrote:
> On Thu, Jan 27, 2011 at 3:46 PM, J. Roeleveld <joost@antarean.org> wrote:
> > On Thursday 27 January 2011 21:25:02 Paul Hartman wrote:
> >> On Thu, Jan 27, 2011 at 2:15 PM, Nikos Chantziaras <realnc@arcor.de> 
wrote:
> >> > On 01/27/2011 09:41 PM, Dale wrote:
> >> >> YoYo Siska wrote:
> >> >>> Yes.
> >> >>> It might not be perfect, but mostly it works pretty well.
> >> >>> Once make started 10 or so process, which ate all my ram, because I
> >> >>> forgot to reenable swap, when I was playing with something before
> >> >>> that
> >> >>> 
> >> >>> :)
> >> >>> 
> >> >>> yoyo
> >> >> 
> >> >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs
> >> >> and it was up to about 3Gbs at one point and using some swap as well.
> >> >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6
> >> >> core CPU too.
> >> >> 
> >> >> I wonder how much faster it would be if the work directory is put on
> >> >> tmpfs? With 16Gbs, that should work even for OOo.
> >> > 
> >> > Btw, if you're using more instances than the amount of CPUs, the
> >> > result will be slow-down.
> >> > 
> >> > With the default kernel scheduler, best if amount of CPUs + 1.  (On a
> >> > 4-core, that's -j5).
> >> 
> >> Once, when building my kernel, I accidentally forgot to specify the
> >> number of makes and ran "make -j all". That was a really bad idea, the
> >> system became totally unresponsive for quite a long time, much longer
> >> than normal kernel build time, but it did eventually finish!
> > 
> > I have found that multi-core systems with sufficient memory can handle
> > "-j" (no value) a lot better then sindle-core systems. I do on occasion
> > do it with the kernel and can still continue using the system. (For
> > comparison, my desktop is a 4-core AMD64 with 8GB memory)
> 
> Strange, in my case it was an i7 920 (4 cores, hyperthreaded, appears
> as 8 CPUs to Linux) with 12GB of RAM. Maybe if I prefixed it
> with"nice" it would not have brought my computer to its knees... or
> maybe related to the schedulers and other kernel voodoo that I don't
> understand. I might try it again someday :)

That is strange, unless your harddrive is really underperforming?
Or do you have all the options in the kernel selected?

Btw, HyperThreading doesn't work too well when you have a lot of identical 
tasks. In that case, you might end up with lesser performance as there are no 
"usable unused" parts in your cores, but the CPU-schedules (the hardware one 
for HT) is looking for things to fill those last few bits with.

--
Joost

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Dale]

Nikos Chantziaras wrote:
> On 01/27/2011 09:41 PM, Dale wrote:
>> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs
>> and it was up to about 3Gbs at one point and using some swap as well.
>> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core
>> CPU too.
>>
>> I wonder how much faster it would be if the work directory is put on
>> tmpfs? With 16Gbs, that should work even for OOo.
>>
>
> Btw, if you're using more instances than the amount of CPUs, the 
> result will be slow-down.
>
> With the default kernel scheduler, best if amount of CPUs + 1.  (On a 
> 4-core, that's -j5).
>

I just used -j with no number.  It worked fine and I played Solitaire 
and checked my email while it was running.  The only thing I noticed was 
it using swap.  That could slow things down but otherwise, it worked fine.

I do have the nice and ionice settings in make.conf tho.  That helps a 
lot for sure.  Without those, it would likely slow down a lot.

Dale

:-)  :-)

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1968 bytes --]

On Thursday 27 January 2011 22:18:22 J. Roeleveld wrote:
> On Thursday 27 January 2011 23:05:22 Paul Hartman wrote:
> > On Thu, Jan 27, 2011 at 3:46 PM, J. Roeleveld <joost@antarean.org> wrote:

> > >> Once, when building my kernel, I accidentally forgot to specify the
> > >> number of makes and ran "make -j all". That was a really bad idea, the
> > >> system became totally unresponsive for quite a long time, much longer
> > >> than normal kernel build time, but it did eventually finish!
> > > 
> > > I have found that multi-core systems with sufficient memory can handle
> > > "-j" (no value) a lot better then sindle-core systems. I do on occasion
> > > do it with the kernel and can still continue using the system. (For
> > > comparison, my desktop is a 4-core AMD64 with 8GB memory)
> > 
> > Strange, in my case it was an i7 920 (4 cores, hyperthreaded, appears
> > as 8 CPUs to Linux) with 12GB of RAM. Maybe if I prefixed it
> > with"nice" it would not have brought my computer to its knees... or
> > maybe related to the schedulers and other kernel voodoo that I don't
> > understand. I might try it again someday :)
> 
> That is strange, unless your harddrive is really underperforming?
> Or do you have all the options in the kernel selected?
> 
> Btw, HyperThreading doesn't work too well when you have a lot of identical
> tasks. In that case, you might end up with lesser performance as there are
> no "usable unused" parts in your cores, but the CPU-schedules (the
> hardware one for HT) is looking for things to fill those last few bits
> with.

I'm running i7 Q 720 (4 cores, hyperthreaded) and have MAKEOPTS="-j9" without 
any slowdown.  One or two packages (like OpenOffice) will fail and need -j=1 
to emerge.  Otherwise no noticeable drop in desktop responsiveness.

I have not set up portage niceness so it runs with default value.

Given the above what shall I set --load-average as?
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[J. Roeleveld]

On Thursday 27 January 2011 23:59:24 Mick wrote:
> I'm running i7 Q 720 (4 cores, hyperthreaded) and have MAKEOPTS="-j9"
> without any slowdown.  One or two packages (like OpenOffice) will fail and
> need -j=1 to emerge.  Otherwise no noticeable drop in desktop
> responsiveness.
> 
> I have not set up portage niceness so it runs with default value.
> 
> Given the above what shall I set --load-average as?

Not sure, I tested last night with a value of 12 and my system was still 
responsive.

It depends, I think, on how well your system handles higher load.

The best suggestion I can give is to actually test with values and see how 
well it responds.

--
Joost

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[J. Roeleveld]

On Thursday 27 January 2011 23:53:04 Dale wrote:
> Nikos Chantziaras wrote:
> > On 01/27/2011 09:41 PM, Dale wrote:
> >> I noticed the same thing with mine. It used a LOT of ram. I have 4Gbs
> >> and it was up to about 3Gbs at one point and using some swap as well.
> >> I'm hoping to max out to 16Gbs as soon as I can. May upgrade to a 6 core
> >> CPU too.
> >> 
> >> I wonder how much faster it would be if the work directory is put on
> >> tmpfs? With 16Gbs, that should work even for OOo.
> > 
> > Btw, if you're using more instances than the amount of CPUs, the
> > result will be slow-down.
> > 
> > With the default kernel scheduler, best if amount of CPUs + 1.  (On a
> > 4-core, that's -j5).
> 
> I just used -j with no number.  It worked fine and I played Solitaire
> and checked my email while it was running.  The only thing I noticed was
> it using swap.  That could slow things down but otherwise, it worked fine.
> 
> I do have the nice and ionice settings in make.conf tho.  That helps a
> lot for sure.  Without those, it would likely slow down a lot.

Which settings do you use for that?

--
Joost

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Dale]

J. Roeleveld wrote:
> On Thursday 27 January 2011 23:53:04 Dale wrote:
>    
>> I just used -j with no number.  It worked fine and I played Solitaire
>> and checked my email while it was running.  The only thing I noticed was
>> it using swap.  That could slow things down but otherwise, it worked fine.
>>
>> I do have the nice and ionice settings in make.conf tho.  That helps a
>> lot for sure.  Without those, it would likely slow down a lot.
>>      
> Which settings do you use for that?
>
> --
> Joost
>
>    

This is my settings at the moment:

PORTAGE_NICENESS=5
PORTAGE_IONICE_COMMAND="ionice -c -3 -p \${PID}"
MAKEOPTS="-j6"
FEATURES="buildpkg sandbox fixpackages parallel-fetch --keep-going"
EMERGE_DEFAULT_OPTS="--with-bdeps y --backtrack=30"

That works very well here for me.  Rig, AMD 4 core CPU at 3.2Ghz, 4Gbs 
of ram at DDR3 1600 and SATA 3.0Gbs/sec.  Your mileage may vary tho.

Dale

:-)  :-)

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Enrico Weigelt]

* Dale <rdalek1967@gmail.com> wrote:
> If one instance of emerge doesn't know what the other instance has 
> already done, then the second one could emerge it again.  Doesn't emerge 
> do all the calculating at the beginning and runs with that until the end?

That's also one of my questions - does parallel emerge instances
know if some dependency had already been built/installed in the middle ?

> I am using the -j option for the first time now.  I'm updating KDE.  It 
> seems to work fine.  It doesn't scroll all the stuff like with a regular 
> emerges but this new rig is so fast, I can't read it anyway.  I did have 
> a package to fail and it spit out the error for me to read.

BTW: does -j / --jobs also cause a parallel make (-j parameter to gmake) ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-user] Re: Simultaneously emerging multiple packages with same dependencies

[Enrico Weigelt]

Nikos Chantziaras <realnc@arcor.de> wrote:

> I'm not aware of any package system that supports this. 

Briegel does this. It can even build the same package (maybe with
different feature flags) in parallel. Basicly it walks the 
dependency tree from leaves to root, builds binpkg's yet missing
(at the point in it reaches them in the graph) and puts them
into an archive. Perhaps important to note that everything happens
in an sysroot, so there eg. cannot be any issues w/ upgrading
some deps in the middle (along w/ other common issues kicked of 
of the game).


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------