[gentoo-user] For what is emerge waiting for...?

[gentoo-user] For what is emerge waiting for...?

[tuxic]

Hi,

no critism or anything else negative implied here...
,..I am just asking out of coriosity:

This day I am busy compiling all the application,
which are installed on my old system.

I gave emerge 12 threads and 6 cores to do its work
(The CPU is a 6 core one).

I start emerge with a bunch of packages to compile and
install.

And it happens from time to time that all cores seem to be 
on holiday: No load at all. And emerge sits there and waits...
And the harddisk is not busy at all...no blinkenlights,
nothing.

Would it be possible, that there is kind of temporary
dead lock of some kind...when emerge is started with more
then one core?

What else than writing to disk and using the cores?

Cheers!
Meino

Re: [gentoo-user] For what is emerge waiting for...?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

On Sun, 5 Apr 2020 21:10:57 +0200, tuxic@posteo.de wrote:

> And it happens from time to time that all cores seem to be 
> on holiday: No load at all. And emerge sits there and waits...
> And the harddisk is not busy at all...no blinkenlights,
> nothing.

Is $PORTAGE_TMPDIR on a ramdisk?


-- 
Neil Bothwick

I'm Bugs Bunny of Borg.  What's up Collective?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] For what is emerge waiting for...?

[tuxic]

On 04/05 08:38, Neil Bothwick wrote:
> On Sun, 5 Apr 2020 21:10:57 +0200, tuxic@posteo.de wrote:
> 
> > And it happens from time to time that all cores seem to be 
> > on holiday: No load at all. And emerge sits there and waits...
> > And the harddisk is not busy at all...no blinkenlights,
> > nothing.
> 
> Is $PORTAGE_TMPDIR on a ramdisk?
> 

no...it is planned for the new system, when that one (or better the
mount point)  is on the SSD...

> 
> -- 
> Neil Bothwick
> 
> I'm Bugs Bunny of Borg.  What's up Collective?

Re: [gentoo-user] For what is emerge waiting for...?

[Dale]

tuxic@posteo.de wrote:
> Hi,
>
> no critism or anything else negative implied here...
> ,..I am just asking out of coriosity:
>
> This day I am busy compiling all the application,
> which are installed on my old system.
>
> I gave emerge 12 threads and 6 cores to do its work
> (The CPU is a 6 core one).
>
> I start emerge with a bunch of packages to compile and
> install.
>
> And it happens from time to time that all cores seem to be 
> on holiday: No load at all. And emerge sits there and waits...
> And the harddisk is not busy at all...no blinkenlights,
> nothing.
>
> Would it be possible, that there is kind of temporary
> dead lock of some kind...when emerge is started with more
> then one core?
>
> What else than writing to disk and using the cores?
>
> Cheers!
> Meino

I have seen this too. I read ages ago that some packages don't like
parallel jobs so the dev sets it to 1 or 2 depending on what works best
to limit those and build failures.  I don't know if that still applies
to many packages or not but it is one possibility.  I think it is set in
the ebuild so one may could look and see if that is the case.

Another option, it could be something the build process is doing that
can't be done in parallel and is set by upstream.  It may not be in the
ebuild but is in the sources itself.  It does seem to be a bottleneck
but if it is set that way, there may be a good reason.  When compiling,
I like to see my CPU utilized to the max but if it isn't possible, then
I let it do its thing.  I know LOo does some strange things when it is
compiling.  It will compile for a long time with little memory usage but
all CPU cores at the max, even tho the work space is in tmpfs, and then
all of a sudden, it starts eating memory until it is using several GBs. 
Then it compiles some more and starts the install process.  I have no
idea what makes it do that but it's the usual thing for LOo.

I'm sure someone else will have more ideas. 

Dale

:-)  :-) 

P. S. Back to the garden.  We supposed to stay home anyway so I'm taking
down the electric fence, posts and getting ready to plow.  Dang it's wet
out there tho.  At least I have a lift disc this year tho.  Maybe I
won't get stuck. :/

Re: [gentoo-user] For what is emerge waiting for...?

[tuxic]

On 04/05 02:41, Dale wrote:
> tuxic@posteo.de wrote:
> > Hi,
> >
> > no critism or anything else negative implied here...
> > ,..I am just asking out of coriosity:
> >
> > This day I am busy compiling all the application,
> > which are installed on my old system.
> >
> > I gave emerge 12 threads and 6 cores to do its work
> > (The CPU is a 6 core one).
> >
> > I start emerge with a bunch of packages to compile and
> > install.
> >
> > And it happens from time to time that all cores seem to be 
> > on holiday: No load at all. And emerge sits there and waits...
> > And the harddisk is not busy at all...no blinkenlights,
> > nothing.
> >
> > Would it be possible, that there is kind of temporary
> > dead lock of some kind...when emerge is started with more
> > then one core?
> >
> > What else than writing to disk and using the cores?
> >
> > Cheers!
> > Meino
> 
> I have seen this too. I read ages ago that some packages don't like
> parallel jobs so the dev sets it to 1 or 2 depending on what works best
> to limit those and build failures.  I don't know if that still applies
> to many packages or not but it is one possibility.  I think it is set in
> the ebuild so one may could look and see if that is the case.
> 
> Another option, it could be something the build process is doing that
> can't be done in parallel and is set by upstream.  It may not be in the
> ebuild but is in the sources itself.  It does seem to be a bottleneck
> but if it is set that way, there may be a good reason.  When compiling,
> I like to see my CPU utilized to the max but if it isn't possible, then
> I let it do its thing.  I know LOo does some strange things when it is
> compiling.  It will compile for a long time with little memory usage but
> all CPU cores at the max, even tho the work space is in tmpfs, and then
> all of a sudden, it starts eating memory until it is using several GBs. 
> Then it compiles some more and starts the install process.  I have no
> idea what makes it do that but it's the usual thing for LOo.
> 
> I'm sure someone else will have more ideas. 
> 
> Dale
> 
> :-)  :-) 
> 
> P. S. Back to the garden.  We supposed to stay home anyway so I'm taking
> down the electric fence, posts and getting ready to plow.  Dang it's wet
> out there tho.  At least I have a lift disc this year tho.  Maybe I
> won't get stuck. :/
> 
> 

Hi Dale,

if emerge encounters a have-to-use-one-core-onlu ebiuld and that one
is not constantlu writing to the disk I would expect one core busy
at least.

But no...nothing...as I said: All cores on holidays...:)

Cheers!
Meino

Re: [gentoo-user] For what is emerge waiting for...?

[Stefan Schmiedl]

"tuxic@posteo.de" <tuxic@posteo.de>, 05.04.2020, 21:58:

> if emerge encounters a have-to-use-one-core-onlu ebiuld and that one
> is not constantlu writing to the disk I would expect one core busy
> at least.

> But no...nothing...as I said: All cores on holidays...:)

I ran into this a few times when some host was unresponsive,
so a package could not be downloaded, hence emerge did not
have anything to build.

Check with htop's tree view or "ps faux" if emerge
has a wget child process.

s.

Re: [gentoo-user] For what is emerge waiting for...?

[tastytea]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

On Sun, 5 Apr 2020 22:13:57 +0200 
Stefan Schmiedl <s@xss.de> wrote:

> "tuxic@posteo.de" <tuxic@posteo.de>, 05.04.2020, 21:58:
> 
> > if emerge encounters a have-to-use-one-core-onlu ebiuld and that one
> > is not constantlu writing to the disk I would expect one core busy
> > at least.  
> 
> > But no...nothing...as I said: All cores on holidays...:)  
> 
> I ran into this a few times when some host was unresponsive,
> so a package could not be downloaded, hence emerge did not
> have anything to build.
> 
> Check with htop's tree view or "ps faux" if emerge
> has a wget child process.
> 
> s.

The fetch process writes a log to /var/log/emerge-fetch.log. If
fetching a package is the problem, you should see it there.

tastytea

-- 
Get my PGP key with `gpg --locate-keys tastytea@tastytea.de` or at
<https://tastytea.de/tastytea.asc>.

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

[gentoo-user] Re: For what is emerge waiting for...?

[Nikos Chantziaras]

On 05/04/2020 22:10, tuxic@posteo.de wrote:
> [...]
> I start emerge with a bunch of packages to compile and
> install.
> 
> And it happens from time to time that all cores seem to be
> on holiday: No load at all. And emerge sits there and waits...
> And the harddisk is not busy at all...no blinkenlights,
> nothing.
> 
> Would it be possible, that there is kind of temporary
> dead lock of some kind...when emerge is started with more
> then one core?

It would have been helpful if you posted the output of emerge at the 
point it hangs.