[gentoo-dev] emerge-webrsync bandwidth improvement

[gentoo-dev] emerge-webrsync bandwidth improvement

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Hola all.

Just sending out a notice for those who have low bandwidth 
connections- app-portage/emerge-delta-webrsync was added to the tree 
earlier this week, and I'd like to get some user feedback on it.

Short version of it: it pulls down patches instead of pulling a full 
snapshot.  Figure ~100kB per day.  (if you were using emerge-webrsync 
already, per day is 19mB normally, for rsync figure 3mB).

Long version of it: 
http://dev.gentoo.org/~ferringb/blog/archives/2005-03.html#e2005-03-29T17_14_42.txt
and
http://dev.gentoo.org/~ferringb/blog/archives/2005-04.html#e2005-04-04T04_47_41.txt 

It's a work in progress, but it's been functional without any major 
bugs for a few weeks now, so if you're interested, take it for a spin.

~brian
PS: if responding to me via gentoo-user, kindly cc me- I'm not on that 
ml (too much mail already).

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: emerge-webrsync bandwidth improvement

[sf]

Brian Harring wrote:
...
> It's a work in progress, but it's been functional without any major 
> bugs for a few weeks now, so if you're interested, take it for a spin.

Thanks in advance for your effort.

On first invocation emerge-delta-webrsync pulled a complete snapshot (as
described in your blog). On second invocation I got:

Looking for available base versions for a delta
fetching patches
failed fetching snapshot-20050503-20050504.patch.bz2.md5sum
no patches found? up to date?  syncing
Syncing local tree...
building file list ...
114391 files to consider

Number of files: 114391
Number of files transferred: 0
Total file size: 89923713 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 2701117
Total bytes written: 2701170
Total bytes read: 20

wrote 2701170 bytes  read 20 bytes  125636.74 bytes/sec
total size is 89923713  speedup is 33.29
cleaning up
transferring metadata/cache
skipping sync

>>> Updating Portage cache:  100%


Is there any way to skip syncing if no patches are found? emerge sync
uses a timestamp for that purpose, doesn't it?

Regards
	Stephan

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge-webrsync bandwidth improvement

[Brian Harring]

On Wed, May 04, 2005 at 02:27:02PM +0200, sf wrote:
> Is there any way to skip syncing if no patches are found? emerge sync
> uses a timestamp for that purpose, doesn't it?

Could, yes.  Few more pressing things re: emerge-delta-webrsync, 
namely (order of what I'm hacking on now)

A) bzip2 v1.03 is stabled now.  MD5 compressor issue _will_ bite  
   emerge-delta-webrsync in the ass _soon_ .  Should have a version 
   bump that handles this tomorrow or friday at the latest.
B) Permenant solution needed for when snapshots upstream aren't 
   generated, as occured 04/29.  This probably will have to be 
   serverside- easiest route, otherwise have to implement version 
   jumping logic in bash, which is ugly
C) cleansing of old snapshots.  Current 'handling' of it (read: not 
   doing a damn thing) is ugly.  :)

So... re: your request, emerge-webrsync default behaviour actually is 
to sync, regardless if an update was pulled or not.  I'd rather not 
deviate from the default behaviour of emerge-webrsync, but tacking in 
support for a command opt (say -u for 'update only'?) can be slipped 
out at some point.

Thoughts?
~brian
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: emerge-webrsync bandwidth improvement

[sf]

Brian Harring wrote:
...
> B) Permenant solution needed for when snapshots upstream aren't 
>    generated, as occured 04/29.  This probably will have to be 
>    serverside- easiest route, otherwise have to implement version 
>    jumping logic in bash, which is ugly

Generate an empty delta when no snapshot exists?

> C) cleansing of old snapshots.  Current 'handling' of it (read: not 
>    doing a damn thing) is ugly.  :)

Always create the most recent snapshot and delete old snapshots and all
deltas?

Regards
	Stephan

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: emerge-webrsync bandwidth improvement

[Brian Harring]

On Mon, May 09, 2005 at 09:09:57AM +0200, sf wrote:
> Brian Harring wrote:
> ...
> > B) Permenant solution needed for when snapshots upstream aren't 
> >    generated, as occured 04/29.  This probably will have to be 
> >    serverside- easiest route, otherwise have to implement version 
> >    jumping logic in bash, which is ugly
> 
> Generate an empty delta when no snapshot exists?
Doesn't really work that way; each patch basically is a list of 
commands, multiple patches == pulling bits/pieces of commands from 
the the referencing version for that patch.
Basically it's overlaying commands; an empty patch would break the 
chain.  While it doesn't sound perfect/pretty, generating a patch that 
is just a copy of the reference (the command list is just a copy of 
all of the reference file to the versioned file).  It's actually 
easier/cleaner to go this route, then modify the client- the client's 
approach is simply, I get patch from day N-1 to N, wash rinse repeat.  

Modifying it to handle arbitrary jumps basically means the client has 
to inspect the remote server for what's available, rather then 
just making an attempt for a fetch.  It's easiest, and is only around 
60 bytes per day upstream snapshots go out... so it's minor overhead, 
and shouldn't occur all that often :)

> > C) cleansing of old snapshots.  Current 'handling' of it (read: not 
> >    doing a damn thing) is ugly.  :)
> 
> Always create the most recent snapshot and delete old snapshots and all
> deltas?
Pretty much.  Pushed emerge-delta-webrsync-2 into the tree last 
friday, should address all of the issues I mentioned.
~brian
-- 
gentoo-dev@gentoo.org mailing list