Re: [gentoo-dev] Idea for the portage maintainers

[Troy Dack <troy@tkdack.com>]

On Mon, 2004-04-12 at 22:36, Tom St Denis wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On April 12, 2004 08:23 am, Georgi Georgiev wrote:
> > maillog: 12/04/2004-08:03:13(-0400): Tom St Denis types
> >
> > > Oh, ok so I'll just format my disk, reinstall Gentoo from scratch so that
> > > I can not waste 200M of space on 80k small files.
> >
> > You can always use a loopback device with a filesystem of your choice.
> > http://forums.gentoo.org/viewtopic.php?t=68215
> 
> That's a potential solution but a bit out of the way for the user don't you 
> think?  Isn't the point of quality software to attract users simply by having 
> merit?
> 
> > > That's not really user friendly.  Could have done a JAR like setup for
> > > each dir of the tree.  e.g. all of app-text be one huge ZIP file [with no
> > > compression].  Such a setup might be a little slower to add/remove files
> > > but would waste less space.
> > >
> > > The idea would make a little sense though in practice.  When I do "emerge
> > > sync" instead of fetching 1000s of small files I just check the timestamp
> > > on the directory zips and download them wholesale.  [Ok so maybe
> > > compression makes sense here].
> >
> > How is this method faster or in any way better than rsync?
> 
> Less metadata?  If I want to download 1000 files off your site I have to say 
> 
> +GET /file1
> - -HTTP/1.1 200 OK
> ... data
> +GET /file2
> - -HTTP/1.1 200 OK
> ... data
> +GET /file3
> - -HTTP/1.1 200 OK
> ... data
> ETC
>
> The overhead translates on both sides too.  E.g. My FS now has to find room 
> for and enter in 1000s of files.  The server side has to locate 1000s of 
> files.
> 
> Also the portage files are not compressed.  If you take an entire directory 
> [again say app-text] and compress it you save bandwidth.  A quick check of 
> app-text on my box [updated last night] gives a 850K zip file [290K tar.bz2] 
> compared to 2.7M of raw data.  Obviously sending the entire zip would be 
> wasting more bandwidth when only small changes occur.

Rsync compresses the information it sends, additionally (and most
importantly) rsync only sends the differences, not the whole file, so it
is even more efficient. Visit rsync.samba.org and have a read about how
it all works.

Another point against a monolithic zip containing all the ebuilds (or
even per directory zips) is the performance hit that slow machines would
take, not everybody runs gentoo on a 2GHz plus machine (eg: my little
PII-400 in the corner)

-- 
Troy Dack					http://linux.tkdack.com
<troy@tkdack.com>				http://webportage.sf.net

Public Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x4D90BE3C
Key fingerprint = 1F3D 6C15 16AA 09D5 0C96  92E5 FD89 16F9 4D90 BE3C


--
gentoo-dev@gentoo.org mailing list