[gentoo-user] Update portage cache ... horribly slow

[gentoo-user] Update portage cache ... horribly slow

[Paweł Madej]

Hello,

I want to know for what reason there is sth like updating portage cache 
after rsync in emerge sync.

It's very very slow and I dont know why. So my question is could I some 
way turn off this cache?

Greets
Paweł
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Allan Spagnol Comar]

This cache is used to resolve all packages you want to update,install
or remove from your box; if you turn off this cache you would need to
do the same action to every emerge option; I believe that is better to
let it do just when updating portage tree .... :)

Holpe it helps, Allan

On 9/28/05, Paweł Madej <linux@quanteam.info> wrote:
> Hello,
>
> I want to know for what reason there is sth like updating portage cache
> after rsync in emerge sync.
>
> It's very very slow and I dont know why. So my question is could I some
> way turn off this cache?
>
> Greets
> Paweł
> --
> gentoo-user@gentoo.org mailing list
>
>

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Paweł Madej]

Allan Spagnol Comar wrote:
> This cache is used to resolve all packages you want to update,install
> or remove from your box; if you turn off this cache you would need to
> do the same action to every emerge option; I believe that is better to
> let it do just when updating portage tree .... :)
> 
> Holpe it helps, Allan

Thx for fast answer.

Ok I'll leave it as is, but maybe is there any way too speedup it? It 
runs on my P3 800 / HDD 5400 rpm more that 10 minutes.

Or it is working on every computer so slowly?

Greets
Paweł

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Dunc]

Paweł Madej wrote:

> Allan Spagnol Comar wrote:
>
>> This cache is used to resolve all packages you want to update,install
>> or remove from your box; if you turn off this cache you would need to
>> do the same action to every emerge option; I believe that is better to
>> let it do just when updating portage tree .... :)
>>
>> Holpe it helps, Allan
>
>
> Thx for fast answer.
>
> Ok I'll leave it as is, but maybe is there any way too speedup it? It
> runs on my P3 800 / HDD 5400 rpm more that 10 minutes.
>
> Or it is working on every computer so slowly?
>
> Greets
> Paweł
>
That sounds a bit slow, but not massively so, it takes a long time.

You could CRON your emerge sync to happen in the middle of the night,
and then you won't have to watch it. :-)

For something you should only do once a day, 10 minutes isn't that bad
though.

Cheers,

Dunc



-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Update portage cache ... horribly slow

[Michael Kintzios]

> -----Original Message-----
> From: Pawe³ Madej [mailto:linux@quanteam.info] 
> Sent: 28 September 2005 12:25
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] Update portage cache ... horribly slow
> 
> 
> Allan Spagnol Comar wrote:
> > This cache is used to resolve all packages you want to 
> update,install
> > or remove from your box; if you turn off this cache you 
> would need to
> > do the same action to every emerge option; I believe that 
> is better to
> > let it do just when updating portage tree .... :)
> > 
> > Holpe it helps, Allan
> 
> Thx for fast answer.
> 
> Ok I'll leave it as is, but maybe is there any way too speedup it? It 
> runs on my P3 800 / HDD 5400 rpm more that 10 minutes.
> 
> Or it is working on every computer so slowly?

Be grateful you're not running my PIII 600MHz.  If you also are running
X with a browser, xmms, or mplayer and updatedb decides to join in, then
20 min to 1/2 hour is a possibility!  I'll make a mental note to measure
how long it takes next time I sync.
-- 
Regards,
Mick


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Paweł Madej]

Dunc wrote:
> For something you should only do once a day, 10 minutes isn't that bad
> though.

I sync not day by day but 2-3 times a week but when I sync I want to run 
just after it emerge of updates (I follow new ebuilds on [1] site and 
run emerge if sth interesting for me  appears there).

My other question is if there is some script which could follow rrs from 
[1] and run emerge sync and emerge -uND world after there is for example 
10 ebuild updated comparing to my system, or other way that it will 
email me that there is 10 ebuilds new and i should run sync manually.

Greets
Paweł
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Paweł Madej]

I forgot link [1] is 
http://packages.gentoo.org/archs/x86/testing/gentoo_simple.rss

Greets
Paweł
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Holly Bostick]

Paweł Madej schreef:
> 
> My other question is if there is some script which could follow rrs 
> from [1] and run emerge sync and emerge -uND world after there is for
>  example 10 ebuild updated comparing to my system, or other way that 
> it will email me that there is 10 ebuilds new and i should run sync 
> manually.

Well, esync might be what you want-- it's part of gentoolkit, and while
it won't do any emerges for you, it will display (or mail you, which is
how I do it) the list of updated packages for the day, with a note as to
whether the package is an upgrade for your system, new to your system,
or the like. Here's an example of the daily mail I get when I run esync
as a cron job and mail myself the output:* Importing old portage tree

 * Doing '/usr/bin/emerge sync' now

 * Doing 'eupdatedb' now


 * esearch-index generated in 4 minute(s) and 23 second(s)
 * indexed 10357 ebuilds
 * size of esearch-index: 1618 kB

 * Importing new portage tree
 * Preparing databases
 * Searching for changes

[ N] dev-ruby/activesupport (1.1.1-r1):  Utility Classes and Extension
to the Standard Library
[MN] net-libs/aqbanking (1.6.0_beta):  Generic Online Banking Interface
[MN] media-sound/ardour (0.99):  multi-track hard disk recording software
[ N] mail-filter/clamsmtp (1.5):  ClamSMTP is an SMTP filter that allows
you to check for viruses using the ClamAV anti-virus software.
[ N] dev-util/cmake (2.0.6-r1):  Cross platform Make
[ N] net-libs/cvm (0.32-r1):  Credential Validation Modules by Bruce Guenter
[ N] dev-java/dom4j (1.6.1):  Easy to use, open source library for
working with XML, XPath and XSLT on the Java platform using the Java
Collections Framework and with full support for DOM, SAX and JAXP.
[ N] net-mail/fetchyahoo (2.9.0):  Perl script that downloads mail from
a Yahoo! webmail account to a local mail spool, an mbox file, or to
procmail.
[ N] net-libs/libmonetra (4.2.2):  library for connecting to a MCVE
Credit Card Processing Daemon via SSL, TCP/IP, and drop-files.
[ N] net-analyzer/libnasl (2.2.5):  A remote security scanner for Linux
(libnasl)
[ U] www-client/mozilla-firefox (1.0.7-r1):  Firefox Web Browser
[ N] net-analyzer/nagios-core (1.2-r3):  Nagios Core - Check daemon,
CGIs, docs
[ N] net-dns/ndu (0.4-r2):  DNS serial number incrementer and reverse
zone builder
[ N] net-analyzer/nessus (2.2.5):  A remote security scanner for Linux
[ N] net-analyzer/nessus-core (2.2.5):  A remote security scanner for
Linux (nessus-core)
[ N] net-analyzer/nessus-libraries (2.2.5):  A remote security scanner
for Linux (nessus-libraries)
[ N] net-analyzer/nessus-plugins (2.2.5):  A remote security scanner for
Linux (nessus-plugins)
[ N] mail-client/nmh (1.1-r1):  New MH mail reader
[MN] x11-misc/openclipart (0.17-r1):  Open Clip Art Library
(openclipart.org)
[MN] app-office/qbankmanager (0.9.29):  Onlinebanking frontend for aqbanking
[MN] x11-misc/service-discovery-applet (0.1):  Service Discovery Applet
[MN] sys-block/unieject (5):  Multiplatform command to eject and load
CD-Rom drives
[MN] media-libs/urt (3.1b):  the Utah Raster Toolkit is a library for
dealing with raster images
[MN] sys-cluster/vzctl (2.7.0.21):  OpenVZ VPS control utility
[ N] x11-plugins/wmail (2.0-r2):  Window Maker dock application showing
incoming mail
[ N] x11-plugins/wmlpq (0.2.1-r1):  Windowmaker dockapp which monitors
up to 5 printqueues
[ N] x11-plugins/wmmenu (1.2-r1):  WindowMaker DockApp: Provides a popup
menu of icons like in AfterStep, as a dockable application.
[ N] x11-plugins/wmnetload (1.3-r2):  Network interface monitor dockapp


The only upgrade (this is from the 27th), is firefox, but I might take a
look at wmail. Nothing else interesting to me on this particular list.

I suppose if you really felt that you wanted to have an emerge of the
new packages done automatically, you could always create a script to run
esync and mail you the output, then run emerge -uD world after esync
completed successfully (doesn't seem much point in using --newuse if
you're not going to be there to look at the output), but I prefer to do
my emerges manually.

Hope this helps,
Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Michael Crute]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On 9/28/05, Holly Bostick <motub@planet.nl> wrote:
>
> I suppose if you really felt that you wanted to have an emerge of the
> new packages done automatically, you could always create a script to run
> esync and mail you the output, then run emerge -uD world after esync
> completed successfully (doesn't seem much point in using --newuse if
> you're not going to be there to look at the output), but I prefer to do
> my emerges manually.
>
I'm not really sure automating your updates like that is a really good idea
especially in light of a few new package layouts that break things. Wouldn't
that just bite, wake up and, oops the system's broken and I have no idea
why.
 -Mike

--
________________________________
Michael E. Crute
Software Developer
SoftGroup Development Corporation

Linux, because reboots are for installing hardware.
"In a world without walls and fences, who needs windows and gates?"

[-- Attachment #2: Type: text/html, Size: 1259 bytes --]

RE: [gentoo-user] Update portage cache ... horribly slow

[Dave Nebinger]

> It's very very slow and I dont know why. So my question is could I some
> way turn off this cache?

I've had a lot of luck with the cdb patch for portage.  It's mentioned in
the gentoo wiki.  I haven't measured to see how syncs are impacted but
regular portage stuff seems faster.


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 705 bytes --]

On Wed, 28 Sep 2005 15:04:43 +0200, Holly Bostick wrote:

> Well, esync might be what you want-- it's part of gentoolkit,

esync is part or esearch. If you prefer eix, it has an equivalent now,
diff-eix.

This is the script I run from cron.daily

emerge world --update --deep --newuse --pretend --verbose | Mail -s "Updated packages for $(hostname)" neil 
glsa-check 2>/dev/null --test all | Mail -s "GLSA check on $(hostname)" neil
cp -f /var/cache/eix /var/cache/eix.old update-eix
diff-eix /var/cache/eix.old | Mail -s "New packages on $(hostname)" neil
emerge world --update --deep --newuse --fetchonly


-- 
Neil Bothwick

Anything worth fighting for is worth fighting dirty for.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Update portage cache ... horribly slow

[Paweł Madej]

Dave Nebinger wrote:
> 
> 
> I've had a lot of luck with the cdb patch for portage.  It's mentioned in
> the gentoo wiki.  I haven't measured to see how syncs are impacted but
> regular portage stuff seems faster.
> 
> 
That's what I was looking for ... great speedup.

Thx Dave

Greets
Paweł
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Holly Bostick]

Dave Nebinger schreef:
>> It's very very slow and I dont know why. So my question is could I
>> some way turn off this cache?
> 
> 
> I've had a lot of luck with the cdb patch for portage.  It's
> mentioned in the gentoo wiki.  I haven't measured to see how syncs
> are impacted but regular portage stuff seems faster.
> 
> 

This sounds quite interesting, but I can't find any mention of this
patch on the Wiki, even after two searches on the Wiki and 3 on Google.
I feel pretty dumb, since Paweł clearly found it easily, but I can't.

Help...?

Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Hemmann, Volker Armin]

On Wednesday 28 September 2005 17:10, Holly Bostick wrote:

>
> This sounds quite interesting, but I can't find any mention of this
> patch on the Wiki, even after two searches on the Wiki and 3 on Google.
> I feel pretty dumb, since Paweł clearly found it easily, but I can't.
>
> Help...?
>

here:

http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb

gentoo wiki cdb were the search terms ;)

-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] Update portage cache ... horribly slow

[Dave Nebinger]

> Dave Nebinger schreef:
> >> It's very very slow and I dont know why. So my question is could I
> >> some way turn off this cache?
> >
> >
> > I've had a lot of luck with the cdb patch for portage.  It's
> > mentioned in the gentoo wiki.  I haven't measured to see how syncs
> > are impacted but regular portage stuff seems faster.
> >
> >
> 
> This sounds quite interesting, but I can't find any mention of this
> patch on the Wiki, even after two searches on the Wiki and 3 on Google.
> I feel pretty dumb, since Paweł clearly found it easily, but I can't.

http://gentoo-wiki.com/TIP_speed_up_portage_with_cdb

Dave



-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Tony Davison]

On Wednesday 28 September 2005 16:10, Holly Bostick wrote:
> Dave Nebinger schreef:
> >> It's very very slow and I dont know why. So my question is could I
> >> some way turn off this cache?
> >
> > I've had a lot of luck with the cdb patch for portage.  It's
> > mentioned in the gentoo wiki.  I haven't measured to see how syncs
> > are impacted but regular portage stuff seems faster.
>
> This sounds quite interesting, but I can't find any mention of this
> patch on the Wiki, even after two searches on the Wiki and 3 on
> Google. I feel pretty dumb, since Paweł clearly found it easily, but
> I can't.
>
> Help...?
Its under tips and tricks portage.
Just found it myself.
HTH

PS 
Hows the cold?
-- 
Tony Davison
tony.davison2@ntlworld.com
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Holly Bostick]

Tony Davison schreef:
> 
> Its under tips and tricks portage. Just found it myself. HTH

I was looking under How-Tos (and searching in the wiki/google for cdb
patch). Weird that the wiki search engine didn't find it, though. Maybe
I just didn't go through enough of the hits.

> 
> PS Hows the cold?

Better (I'm on the downslope from peak 'I can't sleep because I can't
breathe laying down, do we have any more Vicks VapoRub?' miserableness),
but clearly not back up to par :) . My bf wants me to go back to bed,
but I'm too hardheaded to listen (since I do feel better, and I can get
some minor maintenance done), sitting here with a pot of tea and a lot
of tissues and not doing anything too stressful (meaning all my projects
are backed up, including one 'official' and public one, dammit).

Thanks for asking :) .

And thanks for the link, everybody.

Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Glenn Enright]

On Wed, 28 Sep 2005 12:12, Paweł Madej wrote:
> Dunc wrote:
> > For something you should only do once a day, 10 minutes isn't that bad
> > though.
>
> I sync not day by day but 2-3 times a week but when I sync I want to run
> just after it emerge of updates (I follow new ebuilds on [1] site and
> run emerge if sth interesting for me  appears there).
>
> My other question is if there is some script which could follow rrs from
> [1] and run emerge sync and emerge -uND world after there is for example
> 10 ebuild updated comparing to my system, or other way that it will
> email me that there is 10 ebuilds new and i should run sync manually.
>
> Greets
> Paweł
I second esync. It is a nice script that runs a little faster than 'emerge 
sync'. I have a little script in my daily cron list that follows...

#! /bin/sh
# /etc/cron.daily/esync.automated
# ============================================================================
# Based on rkhunter script by Aaron Walker <ka0ttic@gentoo.org>
# Written by Glenn Enright on 14-Oct-2004
#
# Purpose
#  To provide a facility to regularly update the portage tree via cron so we
#  don't have to remember to do it manually, and send us a list of updates to
#  consider.
#
# Requires
#  portage to be properly installed. see 'man emerge' for details
#  esearch: do 'emerge esearch'
#
# CHANGELOG:
#  19-Oct-2004 - updated to use tool esearch. Improved speed and
#              feedback.
#  29-Sep-2005 - Reviewed code and did some tidying up. Changed file 
#              group to wheel to allow easier administration.
#

# ----------------------------------------------------------------------------
#     ENABLEMENT OPTIONS

# set to no if you don't want to run this script
#
ENABLE=yes

# set to 'yes' if you wish the output to be mailed to you
#
SEND_EMAIL=yes

# ----------------------------------------------------------------------------
#     OPERATIONAL COMMANDS

# main command
#	fixme: is nice needed with 2.6 kernel?
CMD="nice esync --nocolor --nospinner"

# NOTE: the rest of these options are only relevant
# if you set SEND_EMAIL to 'yes'
#
EMAIL_SUBJECT="${HOSTNAME}: 'esync' output"
EMAIL_RECIPIENT=root
EMAIL_CMD="| mail -s \"${EMAIL_SUBJECT}\" ${EMAIL_RECIPIENT}"

# ============================================================================

if [ "${ENABLE}" = "yes" ]; then

   if [ -x /usr/bin/emerge ]; then
      #-------------------------------------------------------
      # test to see if we want to send success/failure email
      #
      if [ "${SEND_EMAIL}" = "yes" ]; then
         CMD="${CMD} ${EMAIL_CMD}"
      else
         CMD="${CMD} &>/dev/null"
      fi
      # ------------------------------------------------------


      # ------------------------------------------------------
      # check to see if emerge is already running so
      # we dont interupt it. fail gracefully if it is
      #
      if [ ! "ps -ax | grep -c emerge" ]; then
         # NB: the following text is formated from the margin.
         MAIL_WARN_MESSAGE="echo -e ' ** WARNING ** : The program 'emerge' is 
currently running. Please wait for the next scheduled update or run esync 
manually.'"
         eval exec "${MAIL_WARN_MESSAGE} ${EMAIL_CMD}"
      else
         # actually do the thing
         eval exec "${CMD}"
      fi
      # -----------------------------------------------------

   fi
fi
# EOF

-- 

	The programmers of old were mysterious and profound.  We cannot fathom
their thoughts, so all we do is describe their appearance.
	Aware, like a fox crossing the water.  Alert, like a general on the
battlefield.  Kind, like a hostess greeting her guests. Simple, like uncarved
blocks of wood.  Opaque, like black pools in darkened caves.
	Who can tell the secrets of their hearts and minds?
	The answer exists only in the Tao.
		-- Geoffrey James, "The Tao of Programming"

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Update portage cache ... horribly slow

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 483 bytes --]

On Thu, 29 Sep 2005 04:58:12 +0000, Glenn Enright wrote:

> I second esync. It is a nice script that runs a little faster than
> 'emerge sync'. 

It calls emerge sync, so how can it run faster?

# This script imports the current esearch index,
# calls `emerge sync` and `eupdatedb` and then
# shows the packages which were updated or added
# during the sync.
#...
syncprogram =   "/usr/bin/emerge sync"


-- 
Neil Bothwick

What colour is a chameleon on a mirror?

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Update portage cache ... horribly slow

[Glenn Enright]

On Thu, 29 Sep 2005 08:22, Neil Bothwick wrote:
> On Thu, 29 Sep 2005 04:58:12 +0000, Glenn Enright wrote:
> > I second esync. It is a nice script that runs a little faster than
> > 'emerge sync'.
>
> It calls emerge sync, so how can it run faster?
>
> # This script imports the current esearch index,
> # calls `emerge sync` and `eupdatedb` and then
> # shows the packages which were updated or added
> # during the sync.
> #...
> syncprogram =   "/usr/bin/emerge sync"

Umm... i didn't realize that! :p Until now I hadn't actually looked at the 
esync script.

I guess while I was trying it out, subjectively it seemed to do its thing 
faster. Perhaps that was down to the fact it calls emerge with the verbose 
option off, which provides less overhead in the terminal. But obviously that 
is easy enough to do from the command line as well. Still I get the 
functionality of esearch in the bargain so it suits me.

-- 

"To IBM, 'open' means there is a modicum of interoperability among some of 
their equipment."
-- Harv Masterson
-- 
gentoo-user@gentoo.org mailing list