Re: [gentoo-dev] Idea for the portage maintainers - personal experiences with a .zip-db

[Karl Trygve Kalleberg <karltk@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]

On Mon, Apr 12, 2004 at 01:03:18PM -0400, Todd Berman wrote:
>
> With compression I could understand, but without I don't think any speed
> difference would be noticeable. However, even with compression the
> operation should be fairly fast. zip is not like a tar.gz or tar.bz2,
> ie, you can read a file out of it and search through it fairly fast, and
> you don't have to uncompress the entire archive to get a single file.

About 1.5 years ago I hacked portage to store the /usr/portage (portdir)
in one .zip-file. At the time, du -h reported ~116MB for the portdir
tree. Zip, compressed with -9, resulted in a 16MB file.

I did two prinicpal tests
1) emerge --emptytree -up world  with /var/cache/edb/dep 
and
2) emerge --emptytree -up world without /var/cache/edb/dep

I also did a few emerges of selected packages.

All testing was done on a 100MHz K5 with 96MBs of RAM. There was no
noticeable speed drop in case (1) compared to a non-zipped version, most
probably thanks to the cache.

Case (2) was of course noticeably slower. In some cases, the >100MB
space gain my very well be worth it, on firewalls or older systems
(where we want to use a distcc).

Actual emerges were not a problem either. Uncompressing the relevant
ebuild file and its auxiliary files (the contents in files/) is too
quick to be measurable, even on this very slow test box, since
.zip-files are indexed.

I didn't get so far as to implement an alternative sync mechanism;
obviously rsync isn't ideal for this sort of stuff, perhaps xdeltas
would be better?

I did have a scheme in mind however, based on regular incremental
backups, that would insure only downloading the ebuilds that have
changed since last time (and the new ones), called a changeset.

Each changeset would be compressed in a .zip-file and served statically
over http, thus allowing one server to serve thousands instead of tens
of users at any one time (rsync requires a lot of server cpu power
compared to static file serving over http).

However, as this work was geared against gentoo-embedded, which is not a
bit dormant, and I haven't had any time to keep it current against
portage, it never fell into use:)


Kind regards,

Karl T

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]