[gentoo-dev] GLEP for Deltup package update system

[gentoo-dev] GLEP for Deltup package update system

[John Whitney]

Hi everyone,
   I'd like to call your attention to GLEP 9 (http://glep.gentoo.org) which describes a non-intrusive system for patching old tarballs when updates are available.  This would result in great bandwidth savings and requires no changes in the portage code.  Users can simply emerge deltup and edit their FETCHCOMMAND in make.conf to enable patching.  Package updates would greatly reduce mirror loads and are a must for anyone with a slow net connection.  All that is required is to make patches available and make a file in the portage tree with the list of available patches.  Deltup was written especially for Gentoo and is designed to patch tarballs while still retaining the correct md5sum.  
   You can try my demo (only a few patches available) by following Lisa's "Tiny-HOWTO: Deltup for Gentoo":  http://www.thedoh.com/linux/HOWTO/deltup
   All this project needs is the green light.  Isn't it about time we make Gentoo more available to the bandwidth deprived?
   ---JJW
-- 
______________________________________________
http://www.linuxmail.org/
Now with e-mail forwarding for only US$5.95/yr

Powered by Outblaze

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP for Deltup package update system

[bdharring]

On Monday, July 28, 2003, at 05:09 PM, John Whitney wrote:

>    All this project needs is the green light.  Isn't it about time we 
> make Gentoo more available to the bandwidth deprived?
Taking a quick look at deltup-0.4 (most recent available via 
sourceforge), it is still reliant on having an older version of bzip2 
installed to handle cases where the target distfile was compressed w/ 
the older version of bzip2.  That strikes me as side stepping the 
problem- say down the line, either the bzip2 or gzip implementation was 
tweaked again to produce a slightly better compression, you'd run into 
the same problem again of maintaining md5 correctness.
As I had mentioned in the past on this ml, I'd think some solution 
allowing for an md5 checksum of the data rather then the compressed 
form of said data would be a better solution.  On the plus side, it 
could be extended to allow the user better control over what 
compression is used on their distfiles, but that's ancillary to the 
problem I mentioned above.

Other then that point of contention, kudos on bdelta's performance.
~bdh

>    ---JJW
> -- 
> ______________________________________________
> http://www.linuxmail.org/
> Now with e-mail forwarding for only US$5.95/yr
>
> Powered by Outblaze
>
> --
> gentoo-dev@gentoo.org mailing list
>


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP for Deltup package update system

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

On Mon, Jul 28, 2003 at 04:09:24PM -0600 or thereabouts, John Whitney wrote:

>All that is required is to make patches available and make a file in the portage tree with the list of available patches.  

Making those files available on our mirroring system is not as easy as you
might think.   It is not a big-bang tranformation, meaning we will need to
continue to support people using the old way.  Therefore, deltup will
require additional space on our mirrors.  Before we can commit to
supporting this on our already-bulging mirror system, someone needs to
identify *exactly* what sort of space requirements this system will
require. (ok, not exactly, but at least within a 10% margin of error)

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] GLEP for Deltup package update system

[John Whitney]

The space requirements are dynamic - they can be scaled to any level desired.  We don't need patches for every package.  This is accomplished by knowing which patches are available before trying to download them.  You can provide a significant number of patches with just 100MB of space.  Take a look at my demo repository: ftp://sunsite.dk/projects/deltup/patchfiles.  All these patches take under 50MB and are still not fully optimized (newer patches will be about 33% more efficient).  It'd be great to start out with patches for kernel, kde, gnome, gcc, mozilla, and some other big and necessary packages.  Remember, there are a lot of dial-up users out there (some with only one phone line! :O :) )
  A quota based system would be fine... 
   ---JJW

> Making those files available on our mirroring system is not as easy as you
> might think.   It is not a big-bang tranformation, meaning we will need to
> continue to support people using the old way.  Therefore, deltup will
> require additional space on our mirrors.  Before we can commit to
> supporting this on our already-bulging mirror system, someone needs to
> identify *exactly* what sort of space requirements this system will
> require. (ok, not exactly, but at least within a 10% margin of error)

> --kurt
-- 
______________________________________________
http://www.linuxmail.org/
Now with e-mail forwarding for only US$5.95/yr

Powered by Outblaze

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] GLEP for Deltup package update system

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1194 bytes --]

On Tuesday 29 July 2003 18:45, John Whitney wrote:
> The space requirements are dynamic - they can be scaled to any level
> desired.  We don't need patches for every package.  This is accomplished by
> knowing which patches are available before trying to download them.  You
> can provide a significant number of patches with just 100MB of space.  Take
> a look at my demo repository: ftp://sunsite.dk/projects/deltup/patchfiles. 
> All these patches take under 50MB and are still not fully optimized (newer
> patches will be about 33% more efficient).  It'd be great to start out with
> patches for kernel, kde, gnome, gcc, mozilla, and some other big and
> necessary packages.  Remember, there are a lot of dial-up users out there
> (some with only one phone line! :O :) ) A quota based system would be
> fine...

You also might want to use download figures from a used mirror (not the 
failsafe that is last in the list) to see which patches are actually wanted 
by the userbase. If no-one downloads a certain patch that would signal the 
time has come to remove it.

Paul

-- 
Paul de Vrieze
Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.cs.kun.nl/~pauldv

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]