[gentoo-user] Emerge takes a long time calculating dependencies...

[gentoo-user] Emerge takes a long time calculating dependencies...

[BRM]

I've noticed recently that Emerge is taking a long time (e.g. >20 minutes) to calculate the dependencies when I run "emerge world -vuDNp".
I'm running on a PII-233 w/ 190MB RAM and using the 2008.0 server profile. (I've thought about moving to the hardened profile, but want to at least get things finished configured first, which I'm taking my slow time doing.)
It use to run pretty fast (e.g. < 4 minutes). This started probably back in July some time, I haven't really added much of anything 

The system originally ran the 2007.0 desktop profile (late 2007), then I moved it to the 2008.0 desktop profile (June 2008 - before the final release by a week or so I think...it was when a lot of questions started cycling on this list about the 2008.0 profile but before the final release was made), and later to the 2008.0 server profile (within a couple weeks). In case it matter, I was an late-early adopter of the 2008.0 profile set. (In moving to the server profile, I first moved to the 2008.0 profile, let it sit a week or so, and then moved to the server profile.)

The software is pretty up-to-date. It does have a couple ~x86 programs - but very very few. (Can't remember for sure off the top of my head though.)

I've gone through a few kernels on it, and have been simply deleting the kernel sources in /usr/src for the kernels I didn't use.

Any ideas on what might be making it go so slow?
I've thought about doing some clean-ups with e-merge (e.g. emerge --deep-clean), but haven't risked it yet.

In all other respects, the systems seems to be performing fine. It runs 24-7, hosts my internet connection and network services for my other systems.

TIA,

Ben

[gentoo-user] Re: Emerge takes a long time calculating dependencies...

[Nikos Chantziaras]

BRM wrote:
> I've noticed recently that Emerge is taking a long time (e.g. >20 minutes) to calculate the dependencies when I run "emerge world -vuDNp".
> I'm running on a PII-233 w/ 190MB RAM and using the 2008.0 server profile. (I've thought about moving to the hardened profile, but want to at least get things finished configured first, which I'm taking my slow time doing.)
> It use to run pretty fast (e.g. < 4 minutes). This started probably back in July some time, I haven't really added much of anything 
> 
> The system originally ran the 2007.0 desktop profile (late 2007), then I moved it to the 2008.0 desktop profile (June 2008 - before the final release by a week or so I think...it was when a lot of questions started cycling on this list about the 2008.0 profile but before the final release was made), and later to the 2008.0 server profile (within a couple weeks). In case it matter, I was an late-early adopter of the 2008.0 profile set. (In moving to the server profile, I first moved to the 2008.0 profile, let it sit a week or so, and then moved to the server profile.)
> 
> The software is pretty up-to-date. It does have a couple ~x86 programs - but very very few. (Can't remember for sure off the top of my head though.)
> 
> I've gone through a few kernels on it, and have been simply deleting the kernel sources in /usr/src for the kernels I didn't use.
> 
> Any ideas on what might be making it go so slow?
> I've thought about doing some clean-ups with e-merge (e.g. emerge --deep-clean), but haven't risked it yet.
> 
> In all other respects, the systems seems to be performing fine. It runs 24-7, hosts my internet connection and network services for my other systems.

Try to "defragment" the portage directories.  That is, move them to 
another disk/partition and then back again.  This happens here too after 
some time.  After I do this, it's fast again.  I'm on ext3.

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[Daniel Pielmeier]

2008/9/18 BRM <bm_witness@yahoo.com>:
> Any ideas on what might be making it go so slow?
> I've thought about doing some clean-ups with e-merge (e.g. emerge --deep-clean), but haven't risked it yet.

Did you mean "emerge --depclean"?

> In all other respects, the systems seems to be performing fine. It runs 24-7, hosts my internet connection and network services for my other systems.

How do you sync your portage tree with this machine? How often? Do you
use some kind of script?

Regards,

Daniel

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[BRM]

Yes - emerge --depclean.

And syncing portage is done nightly using a cron job to do "emerge --sync"; however, I run "emerge world -vuDNp" manually.

(The system also hosts portage via rsync for my other gentoo system on my network too, which I manually sync against it; I might be adding another system too in the same manner, not sure yet.)

Ben



----- Original Message ----
From: Daniel Pielmeier <daniel.pielmeier@googlemail.com>
To: gentoo-user@lists.gentoo.org
Sent: Thursday, September 18, 2008 9:17:35 AM
Subject: Re: [gentoo-user] Emerge takes a long time calculating dependencies...

2008/9/18 BRM <bm_witness@yahoo.com>:
> Any ideas on what might be making it go so slow?
> I've thought about doing some clean-ups with e-merge (e.g. emerge --deep-clean), but haven't risked it yet.

Did you mean "emerge --depclean"?

> In all other respects, the systems seems to be performing fine. It runs 24-7, hosts my internet connection and network services for my other systems.

How do you sync your portage tree with this machine? How often? Do you
use some kind of script?

Regards,

Daniel

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[Robert Bridge]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

On Thu, 18 Sep 2008 06:24:34 -0700 (PDT)
BRM <bm_witness@yahoo.com> wrote:

> (The system also hosts portage via rsync for my other gentoo system
> on my network too, which I manually sync against it; I might be
> adding another system too in the same manner, not sure yet.)

Not particularly relevant to the topic at hand, but had you considered
and NFS share for the portage tree? Simplifies tree management within
the network.

Rob.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[Daniel Pielmeier]

2008/9/18 BRM <bm_witness@yahoo.com>:
> And syncing portage is done nightly using a cron job to do "emerge --sync"; however, I run "emerge world -vuDNp" manually.

How often does this cron job run and what are the contents of your
cron script. Do you use "emerge --sync" or probably "eix-sync". I am
asking this because if you sync to often and you use eix-sync the
portage cache is deleted in advantage of the sync and is not recreated
if it fails because of identical timestamps between the local machine
and the rsync server.

So when calling emerge after this the cache has to be created before
and thus it seems dependency calculation takes very long.

> (The system also hosts portage via rsync for my other gentoo system on my network too, which I manually sync against it; I might be adding another system too in the same manner, not sure yet.)
>
> Ben

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[Ricardo Saffi Marques]

Daniel Pielmeier wrote:
> How often does this cron job run and what are the contents of your
> cron script. Do you use "emerge --sync" or probably "eix-sync". I am
> asking this because if you sync to often and you use eix-sync the
> portage cache is deleted in advantage of the sync and is not recreated
> if it fails because of identical timestamps between the local machine
> and the rsync server.

In my case, my cron runs "eix-sync" every night at 00h and I just did an "time emerge -DNuvp world"
here, getting:

Total: 129 packages (103 upgrades, 18 new, 8 reinstalls), Size of downloads: 187,761 kB

real	0m21.074s
user	0m18.913s
sys	0m0.724s

So I assume once a day (let's say, every night) is a good choice for syncing, right? :-)
Never ever had a problem with slow dependency calculation on emerge.

-- 
Ricardo Saffi Marques
http://www.las.ic.unicamp.br/~saffi/
======================================================
Laboratory of System Administration and Security - LAS
Institute of Computing - IC
P.O. Box: 6176
University of Campinas - UNICAMP
13083-852, Campinas, SP, Brazil
======================================================

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[BRM]

I only use the standard emerge utility. I can't access the system at the moment since I'm at work. I was hoping to have some ideas to try out tonight.

I believe my cron job just does a straight call to "emerge --sync" (e.g. no wrapping script, etc.); and it runs sometime between midnight and 6 AM. (Can't recall off hand.)
But - per gentoo portage protocol - it only runs once a day, no more.

And I am only referring to the time specified in the "calculating dependencies" stage when one first runs "emerge world -vuDNp".

I'll see if I have the space available to move portage on/off the partition it is on and see if that improves things; it is an ext3 partition that it is on.

Ben



----- Original Message ----
From: Daniel Pielmeier <daniel.pielmeier@googlemail.com>
To: gentoo-user@lists.gentoo.org
Sent: Thursday, September 18, 2008 9:36:23 AM
Subject: Re: [gentoo-user] Emerge takes a long time calculating dependencies...

2008/9/18 BRM <bm_witness@yahoo.com>:
> And syncing portage is done nightly using a cron job to do "emerge --sync"; however, I run "emerge world -vuDNp" manually.

How often does this cron job run and what are the contents of your
cron script. Do you use "emerge --sync" or probably "eix-sync". I am
asking this because if you sync to often and you use eix-sync the
portage cache is deleted in advantage of the sync and is not recreated
if it fails because of identical timestamps between the local machine
and the rsync server.

So when calling emerge after this the cache has to be created before
and thus it seems dependency calculation takes very long.

> (The system also hosts portage via rsync for my other gentoo system on my network too, which I manually sync against it; I might be adding another system too in the same manner, not sure yet.)
>
> Ben

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[Ricardo Saffi Marques]

Ricardo Saffi Marques wrote:
> In my case, my cron runs "eix-sync" every night at 00h and I just did an "time emerge -DNuvp world"
> here, getting:
> 
> Total: 129 packages (103 upgrades, 18 new, 8 reinstalls), Size of downloads: 187,761 kB
> 
> real	0m21.074s
> user	0m18.913s
> sys	0m0.724s
> 
> So I assume once a day (let's say, every night) is a good choice for syncing, right? :-)
> Never ever had a problem with slow dependency calculation on emerge.

Just out of curiosity, the sync entry on my cron:

00 00 * * * /usr/bin/eix-sync -q > /dev/null 2>&1 || echo -e "$(date)\n\n\n$(hostname -f) was NOT
sync'ed.\n\n" | mail -s "harley.las sync fail" saffi@las.ic.unicamp.br

And you can use "genlop -r" (from "app-portage/genlop") to check your sync history.

-- 
Ricardo Saffi Marques
http://www.las.ic.unicamp.br/~saffi/
======================================================
Laboratory of System Administration and Security - LAS
Institute of Computing - IC
P.O. Box: 6176
University of Campinas - UNICAMP
13083-852, Campinas, SP, Brazil
======================================================

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[Daniel Pielmeier]

2008/9/18 Ricardo Saffi Marques <saffi@las.ic.unicamp.br>:
>
> In my case, my cron runs "eix-sync" every night at 00h and I just did an "time emerge -DNuvp world"
> here, getting:
>
> Total: 129 packages (103 upgrades, 18 new, 8 reinstalls), Size of downloads: 187,761 kB
>
> real    0m21.074s
> user    0m18.913s
> sys     0m0.724s
>
> So I assume once a day (let's say, every night) is a good choice for syncing, right? :-)
> Never ever had a problem with slow dependency calculation on emerge.

As far as I know the rsync servers are syncing against the master
server(s) about every half an hour which obviously also updates the
timestamp. So what I described above only happens if you sync within a
shorter period. Running eix-sync every day shouldn't be a problem.

Syncing twice an hour is also considered bad practise because of the
heavy traffic involved. It just happens to me sometimes if I test
stuff and had to sync within a shorter period.

I just asked this because it could be possible that the script calls
eix-sync twice or after an emerge --sync which is not necessary or for
some other reason it is called twice within a too short period.

Regarding the fragmentation. I use ext3 and reiserfs for the portage
tree and did not recognize a significant slowdown. There is a slowdown
but not from 4 to 20 minutes. At least in my case.

Regards,

Daniel

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[Daniel Pielmeier]

2008/9/18 BRM <bm_witness@yahoo.com>:

> And I am only referring to the time specified in the "calculating dependencies" stage when one first runs "emerge world -vuDNp".

Is the second run significantly faster, if yes chances are high that
there is a problem with the portage cache.

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[BRM]

Not sure. Will check though. If so, how would I resolve it?

Ben



----- Original Message ----
From: Daniel Pielmeier <daniel.pielmeier@googlemail.com>
To: gentoo-user@lists.gentoo.org
Sent: Thursday, September 18, 2008 10:04:11 AM
Subject: Re: [gentoo-user] Emerge takes a long time calculating dependencies...

2008/9/18 BRM <bm_witness@yahoo.com>:

> And I am only referring to the time specified in the "calculating dependencies" stage when one first runs "emerge world -vuDNp".

Is the second run significantly faster, if yes chances are high that
there is a problem with the portage cache.

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[Alan McKinnon]

On Thursday 18 September 2008 16:43:31 BRM wrote:
> Not sure. Will check though. If so, how would I resolve it?

You can rebuild the cache:

rm -rf /var/cache/edb/dep/*
emerge --metadata

Changing to an sqlite cache can also help:
http://gentoo-wiki.com/TIP_speed_up_portage_with_sqlite

alan


>
> Ben
>
>
>
> ----- Original Message ----
> From: Daniel Pielmeier <daniel.pielmeier@googlemail.com>
> To: gentoo-user@lists.gentoo.org
> Sent: Thursday, September 18, 2008 10:04:11 AM
> Subject: Re: [gentoo-user] Emerge takes a long time calculating
> dependencies...
>
> 2008/9/18 BRM <bm_witness@yahoo.com>:
> > And I am only referring to the time specified in the "calculating
> > dependencies" stage when one first runs "emerge world -vuDNp".
>
> Is the second run significantly faster, if yes chances are high that
> there is a problem with the portage cache.



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[BRM]

Well...I got to trying the methods out tonight...and first started with a 'time' of my standard procedure...

$ time emerge world -vuDNp
real    1m41.658s
user    1m8.664s
sys     0m16.921s

$ time emerge world -vuDNp
real    0m36.697s
user    0m35.026s
sys     0m1.124s


That was two runs right in a row...but no where near the time level I saw the other day - which was well over half an hour. (I went to bed before it stopped the other day...)

I am going to guess that it was a portage cache issue and the daily update fixed it, or something.

Any how...will keep the ideas in mind should it happen again.

Thanks All!

Ben


----- Original Message ----
From: Daniel Pielmeier <daniel.pielmeier@googlemail.com>
To: gentoo-user@lists.gentoo.org
Sent: Thursday, September 18, 2008 10:04:11 AM
Subject: Re: [gentoo-user] Emerge takes a long time calculating dependencies...

2008/9/18 BRM <bm_witness@yahoo.com>:

> And I am only referring to the time specified in the "calculating dependencies" stage when one first runs "emerge world -vuDNp".

Is the second run significantly faster, if yes chances are high that
there is a problem with the portage cache.

Re: [gentoo-user] Emerge takes a long time calculating dependencies...

[BRM]

Sorry...replying to myself...any how...

Yes, I had been seeing it for a while, primarily over SSH. Why it fixed itself? I don't know, but I would guess the portage cache regen on the nightly had something to do with it too. Any how...hopefully it's fixed...but I would guess it'll happen again, so I'll test out more when it does!

Ben



----- Original Message ----
From: BRM <bm_witness@yahoo.com>
To: gentoo-user@lists.gentoo.org
Sent: Friday, September 19, 2008 10:32:30 PM
Subject: Re: [gentoo-user] Emerge takes a long time calculating dependencies...

Well...I got to trying the methods out tonight...and first started with a 'time' of my standard procedure...

$ time emerge world -vuDNp
real    1m41.658s
user    1m8.664s
sys     0m16.921s

$ time emerge world -vuDNp
real    0m36.697s
user    0m35.026s
sys     0m1.124s


That was two runs right in a row...but no where near the time level I saw the other day - which was well over half an hour. (I went to bed before it stopped the other day...)

I am going to guess that it was a portage cache issue and the daily update fixed it, or something.

Any how...will keep the ideas in mind should it happen again.

Thanks All!

Ben


----- Original Message ----
From: Daniel Pielmeier <daniel.pielmeier@googlemail.com>
To: gentoo-user@lists.gentoo.org
Sent: Thursday, September 18, 2008 10:04:11 AM
Subject: Re: [gentoo-user] Emerge takes a long time calculating dependencies...

2008/9/18 BRM <bm_witness@yahoo.com>:

> And I am only referring to the time specified in the "calculating dependencies" stage when one first runs "emerge world -vuDNp".

Is the second run significantly faster, if yes chances are high that
there is a problem with the portage cache.