[gentoo-portage-dev] portage doc & cache

[gentoo-portage-dev] portage doc & cache

[Philippe Lafoucrière]

Hi all

I'm wondering where I can find some doc about portage, and especially 
the cache part. I use debian at work, and get so sad when doing an 
emerge rsync at home (10 or 15 minutes to rsync, and almost 5 minutes to 
calculate world dependancies) on a bi-athlon 1.8GHz.

I just wanted to check that the cache system is really a cache :)

maybe ebuilds should be "compiled" into seriaziled objects or something 
similar. Only an emerge rsync (or something called "emerge update") 
would fetch new ebuilds, and compile new ones (we don't care anymore of 
hand modified files after sync).



On the other hand, I was thinkink of the "rsync way" to synchronise the 
portage tree. Why don't we use a portage tree ID (incremented each time 
an ebuild is commited to CVS). For exemple, I have the tree number 
12512. Current portage tree ID is 12514. I just need too fresh ebuilds.

I don't think it would be really difficult to implement a server that 
would handle requests like :

- "tell me which files where updated since 12512"
- "files are :
foo.ebuild
bar.ebuild"
[emerge is download foo.ebuild, bar.ebuild here]

Moreover, do I REALLY need all ebuilds content ???? Why can't portage 
just look at some headers (with arch, desc, etc.) and left the install 
part on other files. Rsync local tree would be lighter, and faster to 
sync. When you need to emerge some program, emerge would download the 
_real_ ebuild file, the digests, the distfiles, etc.

The headers can also be contained in one large tar.gz file, to reduce 
bandwith again (I know, like debian).

I know this stuff might let you think I'm crazy to rebuild all portage, 
but here are some ideas to dig :D


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Nicholas Jones]

[-- Attachment #1: Type: text/plain, Size: 2857 bytes --]

> I'm wondering where I can find some doc about portage, and especially 
> the cache part. I use debian at work, and get so sad when doing an 
> emerge rsync at home (10 or 15 minutes to rsync, and almost 5 minutes to 
> calculate world dependancies) on a bi-athlon 1.8GHz.

This can be highly dependent on portage version. If you are using
2.0.51, there is no cache available on the servers as of yet. The
format has changed.

Using 2.0.51, with an EMPTY cache the performace should still
be greater than what you are stating. It takes me 2m59.628 on
my P4-2.2 Laptop (5400 RPM) to 'emerge -puD world'. If you're
seeing 5 minutes, there may be an issue on your system with IO.

After caching, it takes 7.867 seconds on my laptop.

2.0.50 should be fairly similar to the rates of 2.0.51 with
a populated cache. The generation times can vary. A hidden
feature, 'the modules file', might help as well.

carpaski@fenchurch carpaski $ cat /etc/portage/modules
portdbapi.auxdbmodule="portage_db_anydbm.database"
eclass_cache.dbmodule="portage_db_anydbm.database"

You'll have to sync after you make those changes as the cache
is essentially invalidated/not-present.

> maybe ebuilds should be "compiled" into seriaziled objects
> or something similar. Only an emerge rsync (or something
> called "emerge update") would fetch new ebuilds, and compile
> new ones (we don't care anymore of hand modified files after sync).

There are a lot of things you could take for granted with the
cache. There are problems inherent with assumptions though; they
tend to break things. Portage forces a complete update of the
cache to ensure that nothing strange happens. The cache entries
_CAN_ and _DO_ change randomly sometimes. Fixing a portage bug
here and there might cause a variable to change and without the
regeneration you wouldn't see that. We've had a lot of issues in
the past where data in the tree was corrupt and/or wrong. This
way expends a couple more minutes on the user's part and can free
up many more on our dev's part.

> I don't think it would be really difficult to implement a server that 
> would handle requests like :

You would have to implement it. The problem is available time and
interfacing with existing systems. If it's not as easy, efficient,
and solid it's not readily acceptable.

> Moreover, do I REALLY need all ebuilds content ???? Why can't portage 
> just look at some headers (with arch, desc, etc.) and left the install 
> part on other files. Rsync local tree would be lighter, and faster to 
> sync. When you need to emerge some program, emerge would download the 
> _real_ ebuild file, the digests, the distfiles, etc.

It wouldn't be any faster. Rsync uses stat values to do the transfer
determination. The values would be exactly the same for headers-only.

--NJ


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage doc & cache

[John Nilsson]

[-- Attachment #1: Type: text/plain, Size: 2625 bytes --]

On Fri, 2004-06-11 at 23:15, Philippe Lafoucrière wrote:
> Hi all
> 
> I'm wondering where I can find some doc about portage, and especially 
> the cache part. I use debian at work, and get so sad when doing an 
> emerge rsync at home (10 or 15 minutes to rsync, and almost 5 minutes to 
> calculate world dependancies) on a bi-athlon 1.8GHz.
> 
> I just wanted to check that the cache system is really a cache :)
> 
> maybe ebuilds should be "compiled" into seriaziled objects or something 
> similar. Only an emerge rsync (or something called "emerge update") 
> would fetch new ebuilds, and compile new ones (we don't care anymore of 
> hand modified files after sync).

Mabey rfc2616 can be of some nice inspiration about caching.
As always, I refer to Roy T. Fielding's "Architectural Styles and
the Design of Network-based Software Architectures" as a must read.

> 
> 
> On the other hand, I was thinkink of the "rsync way" to synchronise the 
> portage tree. Why don't we use a portage tree ID (incremented each time 
> an ebuild is commited to CVS). For exemple, I have the tree number 
> 12512. Current portage tree ID is 12514. I just need too fresh ebuilds.

You have just described subversion...
Anyway you might want to experiment with cvsup vs. rsync.

> Moreover, do I REALLY need all ebuilds content ???? Why can't portage 
> just look at some headers (with arch, desc, etc.) and left the install 
> part on other files. Rsync local tree would be lighter, and faster to 
> sync. When you need to emerge some program, emerge would download the 
> _real_ ebuild file, the digests, the distfiles, etc.

Do some experiments and benchmarking. What if only the cache was
downloaded at sync? Say you have a sql backend for portage, just sync
that and download the ebuilds on demand.


> The headers can also be contained in one large tar.gz file, to reduce 
> bandwith again (I know, like debian).
> 
> I know this stuff might let you think I'm crazy to rebuild all portage, 
> but here are some ideas to dig :D

Personally I like crazy =) I think you should have a better
understanding of the current system before you criticise it though. 

<quote>rsync is a file transfer program for Unix systems. rsync uses the
"rsync algorithm" which provides a very fast method for bringing remote
files into sync. It does this by sending just the differences in the
files across the link, without requiring that both sets of files are
present at one of the ends of the link beforehand.</quote>

rsync allready does it's best to only download new information.

-John

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage doc & cache

[Philippe Lafoucrière]

> Mabey rfc2616 can be of some nice inspiration about caching.
> As always, I refer to Roy T. Fielding's "Architectural Styles and
> the Design of Network-based Software Architectures" as a must read.

I'll take a look at this. As you guys, I don't have enough spare time :p

> You have just described subversion...
> Anyway you might want to experiment with cvsup vs. rsync.

Freebsd is using cvsup. That's the reason for sure.

> Do some experiments and benchmarking. What if only the cache was
> downloaded at sync? Say you have a sql backend for portage, just sync
> that and download the ebuilds on demand.

That's the idea. If you look at debian package system (I know, it's far 
different, but ideas can be used on both projects), an update will
only fetch the "cache", and then, package en demand (using apt-get install)



> Personally I like crazy =) I think you should have a better
> understanding of the current system before you criticise it though. 

Of course, I don't criticise anything, or anyone. I know the maxim "If 
you want it, do it" (Copyright Linus ;p ).
I'll spend sometime if I can to help you guys.

> <quote>rsync is a file transfer program for Unix systems. rsync uses the
> "rsync algorithm" which provides a very fast method for bringing remote
> files into sync. It does this by sending just the differences in the
> files across the link, without requiring that both sets of files are
> present at one of the ends of the link beforehand.</quote>
> 
> rsync allready does it's best to only download new information.

There are too many files to consider now in portage. At start, portage 
had just a few hundred of files, and rsync was doing its job quite well. 
Now portage has more than 80 000 files. Rsync has to consider all of 
them, and it's *REALLY* too long + painfull (my laptop is almost 
unusable during rsyncing). Sound like a "select * from table" on a test 
server with 15 test clients => will break in production environnement 
with thousands of clients.

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Joseph Booker]

Philippe Lafoucrière said:
> That's the idea. If you look at debian package system (I know, it's far
> different, but ideas can be used on both projects), an update will
> only fetch the "cache", and then, package en demand (using apt-get
> install)

For someone like me using mysql-based cache rather then portage's default
cache, how will that figure?

> There are too many files to consider now in portage. At start, portage
> had just a few hundred of files, and rsync was doing its job quite well.
> Now portage has more than 80 000 files. Rsync has to consider all of
> them, and it's *REALLY* too long + painfull (my laptop is almost
> unusable during rsyncing). Sound like a "select * from table" on a test
> server with 15 test clients => will break in production environnement
> with thousands of clients.

I have a 900mhz celeron (copermine), and maybe its the love-sources and
~x86 packages, but ive still been able to do crazy things like talk on
irc, have firefox open, update some programs, all while 'emerge sync'-ing
without too much preformance lost* (now if i forget to turn off
seti@home.....then my computer will simply lockup untill i can get to a
tty). You might want to check out if theres something with your laptop, if
these extremly long times happen with other equpiment.

* I should mention i emerge sync every one or two days.
-- 
 Joe Booker

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Philippe Lafoucrière]

> For someone like me using mysql-based cache rather then portage's default
> cache, how will that figure?

I have a laptop with 256 MB RAM. I can't consider installing a full 
database system to handle package managment. Maybe SQLlite, but not mysql :(


>>There are too many files to consider now in portage. At start, portage
>>had just a few hundred of files, and rsync was doing its job quite well.
>>Now portage has more than 80 000 files. Rsync has to consider all of
>>them, and it's *REALLY* too long + painfull (my laptop is almost
>>unusable during rsyncing). Sound like a "select * from table" on a test
>>server with 15 test clients => will break in production environnement
>>with thousands of clients.
> 
> 
> I have a 900mhz celeron (copermine), and maybe its the love-sources and
> ~x86 packages, but ive still been able to do crazy things like talk on
> irc, have firefox open, update some programs, all while 'emerge sync'-ing
> without too much preformance lost* (now if i forget to turn off
> seti@home.....then my computer will simply lockup untill i can get to a
> tty). You might want to check out if theres something with your laptop, if
> these extremly long times happen with other equpiment.
> 
> * I should mention i emerge sync every one or two days.

Rsync doesn't work with the corporate proxy + firewall. I have to use 
emerge-webrsync (ev. 2 or 3 days).

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Nicholas Jones]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

> Rsync doesn't work with the corporate proxy + firewall. I have to use 
> emerge-webrsync (ev. 2 or 3 days).

This was a fairly important piece of info, btw...

After you update using webrsync you should run
# emerge-webrsync
# emerge metadata

Which will update your cache. I'll update emerge-webrsync
to mention this as well. 'emerge metadata' is a regen, but
it uses the metadata-cache to bypass the bash calls and is
what a normal 'emerge sync' does when it completes the update.

--NJ


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage doc & cache

[Philippe Lafoucrière]

> This was a fairly important piece of info, btw...
> 
> After you update using webrsync you should run
> # emerge-webrsync
> # emerge metadata
> 
> Which will update your cache. I'll update emerge-webrsync
> to mention this as well. 'emerge metadata' is a regen, but
> it uses the metadata-cache to bypass the bash calls and is
> what a normal 'emerge sync' does when it completes the update.

Thanx for the tip !!! Why this is not automatic ?

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Hasan Khalil]

Perhaps using a mysql-based cache would give more benefits than meets 
the eye? I'm thinking particularly about, if it were possible, keeping a 
mysql-based cache on a central server on my LAN, and having all clients 
use that rather than having to keep a cache on each and every one of 
them. This would also speed up cache times on slower machines, assuming 
the network is faster than local access (which is sometimes the case in 
my environment). Just a thought.

	-Hasan

Philippe Lafoucrière wrote:
> 
>> For someone like me using mysql-based cache rather then portage's default
>> cache, how will that figure?
> 
> 
> I have a laptop with 256 MB RAM. I can't consider installing a full 
> database system to handle package managment. Maybe SQLlite, but not 
> mysql :(
> 
> 
>>> There are too many files to consider now in portage. At start, portage
>>> had just a few hundred of files, and rsync was doing its job quite well.
>>> Now portage has more than 80 000 files. Rsync has to consider all of
>>> them, and it's *REALLY* too long + painfull (my laptop is almost
>>> unusable during rsyncing). Sound like a "select * from table" on a test
>>> server with 15 test clients => will break in production environnement
>>> with thousands of clients.
>>
>>
>>
>> I have a 900mhz celeron (copermine), and maybe its the love-sources and
>> ~x86 packages, but ive still been able to do crazy things like talk on
>> irc, have firefox open, update some programs, all while 'emerge sync'-ing
>> without too much preformance lost* (now if i forget to turn off
>> seti@home.....then my computer will simply lockup untill i can get to a
>> tty). You might want to check out if theres something with your 
>> laptop, if
>> these extremly long times happen with other equpiment.
>>
>> * I should mention i emerge sync every one or two days.
> 
> 
> Rsync doesn't work with the corporate proxy + firewall. I have to use 
> emerge-webrsync (ev. 2 or 3 days).
> 
> -- 
> gentoo-portage-dev@gentoo.org mailing list
> 

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Joseph Booker]

Hasan Khalil said:
> Perhaps using a mysql-based cache would give more benefits than meets
> the eye? I'm thinking particularly about, if it were possible, keeping a
> mysql-based cache on a central server on my LAN, and having all clients
> use that rather than having to keep a cache on each and every one of
> them. This would also speed up cache times on slower machines, assuming
> the network is faster than local access (which is sometimes the case in
> my environment). Just a thought.
>
> 	-Hasan

Try modifing OneOfOne's mysql cache modules at
http://forums.gentoo.org/viewtopic.php?p=1156177&sid=365314fc99c8112c75f84a13428b2c88#1156177

-- 
 Joe Booker

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Jeff Smelser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 13 June 2004 11:17 am, Hasan Khalil wrote:
> Perhaps using a mysql-based cache would give more benefits than meets
> the eye? I'm thinking particularly about, if it were possible, keeping a
> mysql-based cache on a central server on my LAN, and having all clients
> use that rather than having to keep a cache on each and every one of
> them. This would also speed up cache times on slower machines, assuming
> the network is faster than local access (which is sometimes the case in
> my environment). Just a thought.

We have been testing this out. This would be a great addition..

http://oneofone.limitlessfx.com/phpwiki/index.php/temp/portage_db_mysql.py

- -- 
	     If God can't help you, how about Mr Coffee?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFAzITzld4MRA3gEwYRAiEbAJ4+knjH4UhMrlaSbg/u2C1es/GS6ACfSfwJ
2InlgOg+QX2Vg/8mD3kw3ZQ=
=VJQT
-----END PGP SIGNATURE-----

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[John Nilsson]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

On Sun, 2004-06-13 at 18:46, Jeff Smelser wrote:
> We have been testing this out. This would be a great addition..
> 
> http://oneofone.limitlessfx.com/phpwiki/index.php/temp/portage_db_mysql.py

As mentioned before, sqlite seems like it would be a better fit.

Selected features (from their site):
      * Zero-configuration - no setup or administration needed.
      * A complete database is stored in a single disk file.
      * Database files can be freely shared between machines with
        different byte orders.
      * Supports databases up to 2 terabytes (2^41 bytes) in size.
      * Small memory footprint: less than 30K lines of C code, less than
        250KB code space (gcc on i486)
      * Faster than other popular database engines for most common
        operations.
      * Self-contained: no external dependencies.

-John

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] portage doc & cache

[Philippe Lafoucrière]

John Nilsson wrote:
> On Sun, 2004-06-13 at 18:46, Jeff Smelser wrote:
> 
>>We have been testing this out. This would be a great addition..
>>
>>http://oneofone.limitlessfx.com/phpwiki/index.php/temp/portage_db_mysql.py
> 
> 
> As mentioned before, sqlite seems like it would be a better fit.


SQLite version 3 is out, check it out :

http://www.sqlite.org/version3.html


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Philippe Lafoucrière]

$ time emerge rsync

real	4m19.561s
user	0m49.026s
sys	0m15.069s

It's a little better than 10 minutes, because I'm using a local rsync 
server (100 Mbps). This time comes from my bi-athlon 1.8 GHz.

$ time emerge -UDp world

real	0m24.430s
user	0m20.513s
sys	0m2.871s

hmm, it's quite fast today. If have a lot of package to upgrade, it 
takes much time






--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] portage doc & cache

[Philippe Lafoucrière]

Thanks a lot ! Portage is really an amazing tool :)

I'm so sad I can't use gentoo on servers at work. My boss and other 
workers are huge fans of debian. Thank god, they let me install a gentoo 
on my box :p

I discovered something interesting in debian : debbootstrap. A tool to 
manage chrooted environments quite nicely. I wonder if there are similar 
tools in gentoo. You just choose what kind of debian you want in the 
chroot (stable, testing, ...) and debbootstrap manage the whole creation.
Moreover, it creates some init.d scripts to restart services inside the 
chroot.. Usefull when restarting the server.

(I know, this is surely not the right place to speak about that)


John Nilsson wrote:
> On Sat, 2004-06-12 at 11:10, Philippe Lafoucrière wrote:
> 
>>$ time emerge rsync
>>$ time emerge -UDp world
> 
> 
> I see it's been a while since you updated you portage skills =)
> 
> http://forums.gentoo.org/viewtopic.php?t=148415 a nice thread on the
> topic.
> 
> summary:
> emerge rsync = emerge sync
> -U is evil, use /etc/portage/* (man portage) instead.
> -p can be replaced with -a to speed things up a bit.
> 
> Just a friendly advice
> -John

--
gentoo-portage-dev@gentoo.org mailing list