Re: [gentoo-dev] Idea for the portage maintainers

[Chris Bainbridge <C.J.Bainbridge@ed.ac.uk>]

On Monday 12 April 2004 13:03, Tom St Denis wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Oh, ok so I'll just format my disk, reinstall Gentoo from scratch so that I
> can not waste 200M of space on 80k small files.
>
> That's not really user friendly.  Could have done a JAR like setup for each
> dir of the tree.  e.g. all of app-text be one huge ZIP file [with no
> compression].  Such a setup might be a little slower to add/remove files
> but would waste less space.
>
> The idea would make a little sense though in practice.  When I do "emerge
> sync" instead of fetching 1000s of small files I just check the timestamp
> on the directory zips and download them wholesale.  [Ok so maybe
> compression makes sense here].

I once posted a test of using rsync with a single large file versus lots of 
small ones on the gentoo forums 
(http://forums.gentoo.org/viewtopic.php?t=10108). I was more concerned with 
bandwidth than speed, but I guess they're connected. Using one big file gave 
a 35% improvement over lots of small files. The problem then was how to 
generate one big file for syncing. In the end I got bored and decided it was 
rsyncs problem for not being as efficient with the small files ;-)

The test of rsyncing uncompressed distfile sources rather than downloading 
them gave an 88% improvement in bandwidth. Problem here is that we would need 
lots of disk space for the uncompressed files.. so its a tradeoff, if your 
bandwidth is limited or expensive and computation and disk space are cheap  
(eg. dialup), then it seems like a good idea.

--
gentoo-dev@gentoo.org mailing list