[gentoo-user] Portage 50-51% and emerge metadata timings

[gentoo-user] Portage 50-51% and emerge metadata timings

[Peter]

peter@mars ~ $ sudo emerge info
Portage 2.0.53 (default-linux/x86/2005.0, gcc-3.4.4, glibc-2.3.5-r2, 2.6.14-gentoo-r5 i686)
================================================================= 
System uname: 2.6.14-gentoo-r5 i686 AMD Athlon(tm) XP 2800+ 
Gentoo Base System version 1.6.14
dev-lang/python:     2.4.2
sys-apps/sandbox:    1.2.12
sys-devel/autoconf:  2.13, 2.59-r6
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1
sys-devel/binutils:  2.16.1
sys-devel/libtool:   1.5.20
virtual/os-headers:  2.6.11-r3

I'd seen all the threads on the problems with portage getting held up
during emerge metadata. Since around a month ago, I have suffered through
the holdup almost every day. Sometimes, it would take 5 or more minutes!

I read as much as I could on the various forums and ng and wanted to post
MY suggested workaround.

In a nutshell, I found that portage needed to be on its own partition and
had to be backed up and restored onto a clean partition. By doing this
alone, there was no more dreaded 50-51% holdup.

I also found that it did not make much difference whether I used ext2,3 or
reiserfs when running emerge metadata. In fact, reiser was slower by
around 30 seconds (57-ext3 vs 1:30-reiser). This was not surprising since
on the namesys website they note that programs like find won't perform as
well due to its internal organization. Interestingly, if I ran emerge
metadata a second time on reiser, the speed dropped to 45 seconds. I
imagine this is because of directory caching.

On my particular setup, I have one partition that includes portage,
portage overlay, distfiles, and portage tmp dir. I set all these in
make.conf.

Having portage in /usr is kind of a contradiction imho, since most files
there are supposed to be static, not changing all the time. Moving portage
and its related files elsewhere makes everything a log cleaner, and at
least in my case, faster.

>From my make.conf. Note, /mnt/src is my portage partition.

PORT_LOGDIR="/var/log/portage"
PORTDIR="/mnt/src/portage"
PORTDIR_OVERLAY="/mnt/src/local/portage"
PORTAGE_TMPDIR="/mnt/src/var/tmp"
DISTDIR="/mnt/src/distfiles"

For right now, I have reiser handling /mnt/src, and I'll leave this alone
for a while. Don't know if performance will improve. It _might_ since
the act of syncing will force a read through the entire portage tree
which might make metadata faster

HTH.

Suggestions:

0) You _are_ backed up, right? At least, save the directories you'll be
working on!
1) make a partition. 1+G for /usr/portage. Larger for additional files. 
I have 7 gig for my setup. Filesystem is up to you. Just make sure kernel
support is enabled (i.e. reiser) and appropriate support files loaded.
2) copy over existing portage using cp -Rp (preserve times and
permissions otherwise portage will download the entire tree), or tar it
and untar it. Whatever works. Just make sure that make.conf is adjusted
and /etc/make.profile is relinked to the new location. 

/etc/make.profile -> /mnt/src/portage/profiles/default-linux/x86/2005.0

This is important otherwise any emerge command will bomb. Your link may
differ depending on your arch. ls -l /etc/make.profile will show what you
need to do. Just substitute your new portage location and add the rest.

3) update make.conf. 
4) see if everything works, try emerge -s, emerge sync, metadata.
5) if it does, you can purge /usr/portage and any other directories you
moved over to your dedicated portage partition.


-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: Portage 50-51% and emerge metadata timings

[Peter]

On Wed, 28 Dec 2005 10:53:18 -0500, Peter wrote:

One more thing that saves time. Mount the portage partition with the
noatime option. For as many files as portage has, this will improve
performance too.

mount -t fstype /dev/hd? /mnt/portage_location -o noatime,....


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Portage 50-51% and emerge metadata timings

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On Wed, 28 Dec 2005 11:22:07 -0500 Jerry McBride
<mcbrides9@comcast.net> wrote:
| All the gentoo systems that I admin show the same slow down at about
| 51%. I dearly wish we'd get away from a file based database. And
| before everyone jumps on me about the various database backend
| patches that I can apply to get what I cry for... I've tried them...
| None of them put "all the data" into a real database... A shame
| too....

Wrong solution. You do realise that the "updating Portage cache" thing
is due to a Portage deficiency, and that the real cache is centrally
generated, right?

-- 
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Portage 50-51% and emerge metadata timings

[Jerry McBride]

On Wednesday 28 December 2005 11:33, Ciaran McCreesh wrote:
> On Wed, 28 Dec 2005 11:22:07 -0500 Jerry McBride
>
> <mcbrides9@comcast.net> wrote:
> | All the gentoo systems that I admin show the same slow down at about
> | 51%. I dearly wish we'd get away from a file based database. And
> | before everyone jumps on me about the various database backend
> | patches that I can apply to get what I cry for... I've tried them...
> | None of them put "all the data" into a real database... A shame
> | too....
>
> Wrong solution. You do realise that the "updating Portage cache" thing
> is due to a Portage deficiency, 

"Portage deficency"? You mean the fact that python scans some thousands of 
files in the file based database, writing as it goes?

> and that the real cache is centrally generated, right?

Yup, from thousands of files in the file based database...

Portage is a wonderful tool for package management, but the sheer size of the 
beast begs for movig it to C and a proper database. I remember in the early 
days of my gentoo experience that portage wasn't a bother. But as ebuilds are 
added to portage and my choice of installed ebuilds grows... portage has 
become quite a slug performance wise. I guess this is where the IT types step 
in and say it scales poorly.

Cheers all and Happy New Year to everyone.


Jerry

-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: Portage 50-51% and emerge metadata timings

[Peter]

On Wed, 28 Dec 2005 14:16:39 -0500, Jerry McBride wrote:
> 
> "Portage deficency"? You mean the fact that python scans some thousands of
> files in the file based database, writing as it goes?
> 
>> and that the real cache is centrally generated, right?
> 
> Yup, from thousands of files in the file based database...
> 
> Portage is a wonderful tool for package management, but the sheer size of
> the beast begs for movig it to C and a proper database. I remember in the
> early days of my gentoo experience that portage wasn't a bother. But as
> ebuilds are added to portage and my choice of installed ebuilds grows...
> portage has become quite a slug performance wise. I guess this is where
> the IT types step in and say it scales poorly.
> 
> Cheers all and Happy New Year to everyone.
> 
> 
> Jerry

http://bugs.gentoo.org/show_bug.cgi?id=108412

It _does_ suck, but looks like it will get better. I'm still going to keep
portage on a separate partition. After all, I went through all the trouble!



-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Portage 50-51% and emerge metadata timings

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]

On Wed, 28 Dec 2005 14:16:39 -0500 Jerry McBride
<mcbrides9@comcast.net> wrote:
| > Wrong solution. You do realise that the "updating Portage cache"
| > thing is due to a Portage deficiency, 
| 
| "Portage deficency"? You mean the fact that python scans some
| thousands of files in the file based database, writing as it goes?

Nope. The fact that Portage uses that second level of cache at all.

| > and that the real cache is centrally generated, right?
| 
| Yup, from thousands of files in the file based database...
| 
| Portage is a wonderful tool for package management, but the sheer
| size of the beast begs for movig it to C and a proper database. I
| remember in the early days of my gentoo experience that portage
| wasn't a bother. But as ebuilds are added to portage and my choice of
| installed ebuilds grows... portage has become quite a slug
| performance wise. I guess this is where the IT types step in and say
| it scales poorly.

The scalability issues have nothing to do with us using files.

-- 
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-user] Re: Portage 50-51% and emerge metadata timings

[Peter]

On Wed, 28 Dec 2005 10:53:18 -0500, Peter wrote:
> For right now, I have reiser handling /mnt/src, and I'll leave this
> alone for a while. Don't know if performance will improve. It _might_
> since the act of syncing will force a read through the entire portage
> tree which might make metadata faster
> 
> 
Things were going along great...Well, after 4 days, it's back to hanging
at 50-51% again. Hope this is better in the next version of portage.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: Portage 50-51% and emerge metadata timings

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter wrote:
| On Wed, 28 Dec 2005 10:53:18 -0500, Peter wrote:
|> For right now, I have reiser handling /mnt/src, and I'll leave this
|> alone for a while. Don't know if performance will improve. It _might_
|> since the act of syncing will force a read through the entire portage
|> tree which might make metadata faster
|>
|>
| Things were going along great...Well, after 4 days, it's back to hanging
| at 50-51% again. Hope this is better in the next version of portage.
| 

Actually, portage-2.1 has seen an improvement in this area.  In addition, I have written a patch that obsoletes the metadata transfer.  That's right, no metadata transfer necessary. :)

http://thread.gmane.org/gmane.linux.gentoo.portage.devel/1602

Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDuJdB/ejvha5XGaMRAmH8AJ9WeMgBwjSs3CTDuWWE4iElurRJigCeIRPS
gC89IkTzngJtTVMMv5uachk=
=iTs5
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: Re: Portage 50-51% and emerge metadata timings

[Peter]

On Sun, 01 Jan 2006 19:00:17 -0800, Zac Medico wrote:
> Actually, portage-2.1 has seen an improvement in this area.  In addition,
> I have written a patch that obsoletes the metadata transfer.  That's
> right, no metadata transfer necessary. :)
> 
> http://thread.gmane.org/gmane.linux.gentoo.portage.devel/1602
> 
> Zac

Thank you. I will take a look. It sure _looks_ simple and logical enough.
I don't know enough about portage though to understand! Will it work with
< pre?. I have x86 portage installed, not ~x86.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: Re: Portage 50-51% and emerge metadata timings

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter wrote:
| On Sun, 01 Jan 2006 19:00:17 -0800, Zac Medico wrote:
|> Actually, portage-2.1 has seen an improvement in this area.  In addition,
|> I have written a patch that obsoletes the metadata transfer.  That's
|> right, no metadata transfer necessary. :)
|>
|> http://thread.gmane.org/gmane.linux.gentoo.portage.devel/1602
|>
|> Zac
| 
| Thank you. I will take a look. It sure _looks_ simple and logical enough.
| I don't know enough about portage though to understand! Will it work with
| < pre?. I have x86 portage installed, not ~x86.
| 

No, you need a 2.1_pre release due to cache changes that my patch relies on (a backport would not really be worth the trouble IMO).  Hopefully the 2.1 branch will be stable soon enough though.

Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDuXbx/ejvha5XGaMRAsVUAKCfDlOg8aloPHwgxeG2tQIYt0MnuACfe8zH
ApZjRceyt6uL/g64T7heGmM=
=KERa
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list