[gentoo-portage-dev] downloading kernel sources with portage

[gentoo-portage-dev] downloading kernel sources with portage

[Bart Van Kerkhove]

Hi there,

I’ve been using Gentoo Linux for about a year now and one thing that has
been bothering me is that every time a new kernel comes out portage
downloads the full version of the new kernel instead of a patch.  Since
every kernel gets stored in distfiles it would be very easy to use the
older kernel and only download the patch.

For example the 2.6.x series:
linux-2.6.0.tar.bz2	32476 KB	32 MB
linux-2.6.1.tar.bz2  	32461 KB	32 MB
linux-2.6.2.tar.bz2  	33106 KB	32 MB
linux-2.6.3.tar.bz2  	33469 KB	33 MB
linux-2.6.4.tar.bz2  	33581 KB	33 MB

linux-2.6.*.tar.bz2	165093 KB	161 MB

Since portage has downloaded all 5 full kernels for me I’ve downloaded
160MB of kernel sources.

If we had simply patched the allready stored 2.6.0 kernel it would have
been only 40MB:
linux-2.6.0.tar.bz2	32476 KB	32 MB
patch-2.6.1.bz2  	  760 KB	 1 MB
patch-2.6.2.bz2  	 2268 KB	 2 MB
patch-2.6.3.bz2  	 2006 KB	 2 MB
patch-2.6.4.bz2  	 2210 KB	 2 MB
linux.*+patch.*		39720 KB	39 MB

I’ve tried this myself and it’s as easy as:
tar -xvjpf linux-2.6.0.tar.bz2 && cd linux-2.6.0
patch -p1 < ../patch-2.6.1
patch -p1 < ../patch-2.6.2
patch -p1 < ../patch-2.6.3
patch -p1 < ../patch-2.6.4
cd .. && mv linux-2.6.0 linux-2.6.4

Since bandwith doesn’t grow on trees and the patching doesn’t take long,
saving far over a gigabyte every 10 users it seems to me this is well
worth it. I understand we can’t patch forever, but every 5 kernels or so
is a big saving in bandwith.

Comments? What do you think?

FrostFire

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] downloading kernel sources with portage

[Lisa Seelye]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

On Sun, 2004-03-28 at 13:02, Bart Van Kerkhove wrote:
> Hi there,
> 
> Ive been using Gentoo Linux for about a year now and one thing that has
> been bothering me is that every time a new kernel comes out portage
> downloads the full version of the new kernel instead of a patch.  Since
> every kernel gets stored in distfiles it would be very easy to use the
> older kernel and only download the patch.

On a related topic, see Deltup project at deltup.sf.net.
-- 
Regards,
-Lisa
<Quidquid latine dictum sit, altum viditur>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] downloading kernel sources with portage

[Jeremy Maitin-Shepard]

This would be a possible portage-ng feature -- specifying alternate
download paths; portage could determine how to minimize downloading.
-- 
Jeremy Maitin-Shepard

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] downloading kernel sources with portage

[Michael Kohl]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]

On Sun, 28 Mar 2004 17:21:04 -0500
Lisa Seelye <lisa@gentoo.org> wrote:

> On a related topic, see Deltup project at deltup.sf.net.

Am I completely mistaken or has there been a patch for portage once that
did make use of delta updates? I believe to remember I've read something
about it on gentoo-dev a while ago. Anyone out there who can remember
thins better than I do?

Regards,
Michael

-- 
www.cargal.org 
GnuPG-key-ID: 0x90CA09E3
Jabber-ID: citizen428 [at] cargal [dot] org
Registered Linux User #278726

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] downloading kernel sources with portage

[Bart Van Kerkhove]

I've tried delta but i couldn't get it to work:

-------------------------------------
Nemo root # ls /usr/distfiles/linux*
/usr/distfiles/linux-2.6.0.fdtu /usr/distfiles/linux-2.6.0.tar.bz2

Nemo root # emerge development-sources
Calculating dependencies ...done!
>>> emerge (1 of 1) sys-kernel/development-sources-2.6.4 to /
>>> Downloading ftp://10.97.1.57/distfiles/linux-2.6.4.tar.bz2
need to obtain linux-2.6.4.tar.bz2
Couldn't find old package to update.
Trying full download
--18:51:28--  ftp://10.97.1.57/distfiles/linux-2.6.4.tar.bz2
           => `/usr/distfiles/linux-2.6.4.tar.bz2'
-------------------------------------
and then it starts to download the full sources...

FrostFire

> On Sun, 28 Mar 2004 17:21:04 -0500
> Lisa Seelye <lisa@gentoo.org> wrote:
>
>> On a related topic, see Deltup project at deltup.sf.net.
>
> Am I completely mistaken or has there been a patch for portage once that
> did make use of delta updates? I believe to remember I've read something
> about it on gentoo-dev a while ago. Anyone out there who can remember
> thins better than I do?
>
> Regards,
> Michael

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] downloading kernel sources with portage

[Bart Van Kerkhove]

I have been reading http://forums.gentoo.org/viewtopic.php?t=53406 with
great interest.  The thread started in may, the GLEP was written in July,
I guess the project just misses "critical mass" to lift of...

I think it's a really great program that would reduce download sizes
immensely.  This is not only good for dial-up users but also for the
download mirrors.

FrostFire

> On Sun, 28 Mar 2004 17:21:04 -0500
> Lisa Seelye <lisa@gentoo.org> wrote:
>
>> On a related topic, see Deltup project at deltup.sf.net.
>
> Am I completely mistaken or has there been a patch for portage once that
> did make use of delta updates? I believe to remember I've read something
> about it on gentoo-dev a while ago. Anyone out there who can remember
> thins better than I do?
>
> Regards,
> Michael
>
> --
> www.cargal.org
> GnuPG-key-ID: 0x90CA09E3
> Jabber-ID: citizen428 [at] cargal [dot] org
> Registered Linux User #278726
>


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] downloading kernel sources with portage

[Bart Van Kerkhove]

Is there a place to submit portage-ng feature requests?
I think modifying "kernel-2.eclass" to download the official patches is a
better choice than using deltup for this.

FrostFire

> This would be a possible portage-ng feature -- specifying alternate
> download paths; portage could determine how to minimize downloading.
> --
> Jeremy Maitin-Shepard
>
> --
> gentoo-portage-dev@gentoo.org mailing list

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] downloading kernel sources with portage

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1564 bytes --]

On Tue, 2004-03-30 at 05:02, Bart Van Kerkhove wrote:
> I have been reading http://forums.gentoo.org/viewtopic.php?t=53406 with
> great interest.  The thread started in may, the GLEP was written in July,
> I guess the project just misses "critical mass" to lift of...
More so the developer split, that and a couple of outstanding technical
issues.  Pretty much deltup is developmentally dead, and the patch
repository that it used is quite out of date.
As it stands, deltup I'd like to get out of the tree as soon as
possible- I've already added a warning to the existing ebuild to this
effect.

> 
> I think it's a really great program that would reduce download sizes
> immensely.  This is not only good for dial-up users but also for the
> download mirrors.
Well, it's a double edged sword- it reduces bandwidth usage (if the user
isn't wiping their DISTDIR often), but it increases the mirror size- the
latter seems to be one of the stickier points of getting patching
support implemented.

> > Am I completely mistaken or has there been a patch for portage once that
> > did make use of delta updates? I believe to remember I've read something
> > about it on gentoo-dev a while ago. Anyone out there who can remember
> > thins better than I do?
Deltup's only tweaks to portage was the addition of dtu.list.aux (if I
recall the filename correctly) to profiles/, and setting FETCHCOMMAND to
call deltup prior to the normal fetchcommand- akin to how distcc/ccache
are called w/ the normal gcc command passed as an arguement.
~brian

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] downloading kernel sources with portage

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

On Sat, 2004-04-03 at 15:50, Bart Van Kerkhove wrote:
> Is there a place to submit portage-ng feature requests?
> I think modifying "kernel-2.eclass" to download the official patches
is a
> better choice than using deltup for this.
Couple of issues w/ such an approach, which I've gone over in this
thread -> http://forums.gentoo.org/viewtopic.php?t=122443

The main problem in my books is the $A kludgery that would have to be
implemented for one specific set of packages- ebuild's/eclass's futzing
w/ $A ($S would be saner, although still problematic) is utterly
intrusive, and would require modification of the existing ebuilds.

That's not to say supporting gnu diff patches isn't a bad idea- I'm just
trying to point out that the implementation issues have some rather
thorny problems to address.
~brian


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]