Re: [gentoo-dev] Idea for the portage maintainers

[Todd Berman <tberman@gentoo.org>]

On Mon, 2004-12-04 at 12:17 -0500, Andrew Gaffney wrote:
> Todd Berman wrote:
> > On Mon, 2004-12-04 at 11:59 -0500, Andrew Gaffney wrote:
> > 
> >>Todd Berman wrote:
> >>
> >>>On Mon, 2004-12-04 at 11:22 -0500, Andrew Gaffney wrote:
> >>>
> >>>
> >>>>Troy Dack wrote:
> >>>>
> >>>>
> >>>>>Another point against a monolithic zip containing all the ebuilds (or
> >>>>>even per directory zips) is the performance hit that slow machines would
> >>>>>take, not everybody runs gentoo on a 2GHz plus machine (eg: my little
> >>>>>PII-400 in the corner)
> >>>>
> >>>>Or my little P233 Thinkpad...
> >>>>
> >>>
> >>>
> >>>And with the current setup of writing thousands of 1K files that little
> >>>p233 thinkpad really flys i bet...
> >>
> >>I'm not sure if that was supposed to be sarcastic, but yes, it does fly. It only takes 
> >>slightly longer to sync than my Athlon 1.3GHz desktop. The only part that takes forever is 
> >>updating the portage cache. That's why I just use a NFS shared portage tree from my 
> >>desktop machine now.
> > 
> > In a way it was and in a way it wasn't. I honestly don't understand how
> > you can explain to me that a compression-less zip file be any slower
> > than the current setup.
> > 
> > With compression I could understand, but without I don't think any speed
> > difference would be noticeable. However, even with compression the
> > operation should be fairly fast. zip is not like a tar.gz or tar.bz2,
> > ie, you can read a file out of it and search through it fairly fast, and
> > you don't have to uncompress the entire archive to get a single file.
> 
> Even without compression, there is still a little bit of overhead with having ebuilds and 
> such contained in ZIP files. The overhead comes into play both during sync and emerge. It 
> wouldn't be noticable on a fast machine (e.g. my Athlon 1.3GHz desktop) but it would make 
> an operation like 'emerge -uDpv world' take quite a bit longer on my Thinkpad.
> 

This is going to be a slow operation on that thinkpad regardless.

> As someone else pointed out, the ZIP files wouldn't make syncs faster either. Rsync 
> transfers only the differences in files and not the entire app-text directory or whatever 
> category you're working with.
> 

Portage does not use this as it takes more cpu server and client side to
figure out what changed with 80 thousand 1K files than it does to just
transfer the files across.

> As another person pointed out, it would be more difficult to modify ebuilds in the tree by 
> hand when they're contained in ZIP files.
> 

This could easily be dealt with. the cvs->rsync process could create zip
files, and portage would use the zip stuff by default for /usr/portage/
and use the old directory structure code for your overlays.

> It was a good idea, but I don't think it is practical.

It is absolutely practical, and I think would cut down on the time taken
to sync (because now you could turn compression back on, and actually
transfer just changes because you wouldnt be syncing 80 thousand files).
It might potentially have small speed loses when running emerge -vuDp
world, but as that process takes a long time regardless, I don't think
it would be noticeable. The time it takes to read a single file out of
an uncompressed zip is not noticeably longer than the time it takes to
read a single file off the hard drive.

Note, as of writing this the portage tree takes up 333MB of space on my
hard drive and a compression-less zipfile of the tree takes up 74MB. To
me, the potential savings are incredible.

--Todd


--
gentoo-dev@gentoo.org mailing list