[gentoo-dev] portage question

[gentoo-dev] portage question

[Leon Chiver]

I've installed gentoo and I must say that I'm very pleased whith it. 
Marvelous job.
But there's still one feature I'm missing and which regards the update 
procedure: wouldn't it be nice if we could upgrade from one version of a 
package to another by only downloading the diff? A few days before I've 
installed mozilla-1.0, today I emerged 1.0.1. But I had to download the 
entire 1.0.1 archive, instead of only a diff file. For people with a 
limited internet access this would be a really great feature.
Does anyone know, is such an improvement planned?

Anyway, marvelous job, a great 10x.
Leon

Re: [gentoo-dev] portage question

[Henti Smith]

On Tue, 01 Oct 2002 23:09:14 +0000
Leon Chiver <leon@level7.ro> wrote:

> I've installed gentoo and I must say that I'm very pleased whith it. 
> Marvelous job.
> But there's still one feature I'm missing and which regards the update 
> procedure: wouldn't it be nice if we could upgrade from one version of a 
> package to another by only downloading the diff? A few days before I've 
> installed mozilla-1.0, today I emerged 1.0.1. But I had to download the 
> entire 1.0.1 archive, instead of only a diff file. For people with a 
> limited internet access this would be a really great feature.
> Does anyone know, is such an improvement planned?

I had a look on the bugzilla page and did a few searches .. didn't se anything that pertains to this, but I have to admit .. 
In the image of gentoo this should have been in portage already. 

being able to update packages just with patches would be really nice for us low bandwidth users .. 

Henti Smith 

Re: [gentoo-dev] portage question

[Evan Read]

On Wed, Oct 02, 2002 at 09:22:06AM +0200, Henti Smith wrote: > On Tue, 01
Oct 2002 23:09:14 +0000 > Leon Chiver <leon@level7.ro> wrote: > > > I've
installed gentoo and I must say that I'm very pleased whith it.  > >
Marvelous job. > > But there's still one feature I'm missing and which
regards the update > > procedure: wouldn't it be nice if we could upgrade
from one version of a > > package to another by only downloading the diff?
A few days before I've > > installed mozilla-1.0, today I emerged 1.0.1.
But I had to download the > > entire 1.0.1 archive, instead of only a diff
file. For people with a > > limited internet access this would be a really
great feature. > > Does anyone know, is such an improvement planned? > > I
had a look on the bugzilla page and did a few searches .. didn't se
anything that pertains to this, but I have to admit ..  > In the image of
gentoo this should have been in portage already.  > > being able to update
packages just with patches would be really nice for us low bandwidth users
..  >

There was a thread about why the kernel stuff isn't done that way.  It was 
discussed that it is very hard to allow for people that sit at very 
different kernel versions to be able to upgrade with one ebuild.  Yes, one 
can do all the checks to see what is needed to go from 2.4.x to 2.4.19 (or 
even 2.2.x to 2.4.19) and just fetch the right diff, but that is much more 
difficult than "get latest source, unpack into /usr/src and patch".

There is nothing stopping anyone from creating thes ebuild themselves and 
posting them on the web say "diff2.4.16to2.4.19.ebuild" and giving people 
the ability to use them , even if they don't get commited.

Back to Mozilla, mozilla is a less complicated beast.  If someone has a 
diff, then numberous ebuilds can be created to cover all the bases, or 
build the logic into one ebuild.  Maybe email the maintainer to ask "why 
not" and if he says "because", then write one yourself ;)

-- 
Evan Read
http://eread.freeshell.org

"The future comes 60 minutes an hour no matter who you are or what you 
do." 
	The Screwtape Letters - C.S. Lewis

Re: [gentoo-dev] portage question

[Alexander Gretencord]

On Wednesday 02 October 2002 09:39, Evan Read wrote:
> On Wed, Oct 02, 2002 at 09:22:06AM +0200, Henti Smith wrote: > On Tue, 01
> Oct 2002 23:09:14 +0000 > Leon Chiver <leon@level7.ro> wrote: > > > I've
> installed gentoo and I must say that I'm very pleased whith it.  > >
> Marvelous job. > > But there's still one feature I'm missing and which
> regards the update > > procedure: wouldn't it be nice if we could upgrade
> from one version of a > > package to another by only downloading the diff?
> A few days before I've > > installed mozilla-1.0, today I emerged 1.0.1.
> But I had to download the > > entire 1.0.1 archive, instead of only a diff
> file. For people with a > > limited internet access this would be a really
> great feature. > > Does anyone know, is such an improvement planned? > > I
> had a look on the bugzilla page and did a few searches .. didn't se
> anything that pertains to this, but I have to admit ..  > In the image of
> gentoo this should have been in portage already.  > > being able to update
> packages just with patches would be really nice for us low bandwidth users
> ..  >

Sorry for that but you _had_ to see that mess. Didn't think mutt would do such 
stuff or do you fake your User-Agent Header and are using Outlook? :)

> There was a thread about why the kernel stuff isn't done that way.  It was
> discussed that it is very hard to allow for people that sit at very
> different kernel versions to be able to upgrade with one ebuild.  Yes, one
> can do all the checks to see what is needed to go from 2.4.x to 2.4.19 (or
> even 2.2.x to 2.4.19) and just fetch the right diff, but that is much more
> difficult than "get latest source, unpack into /usr/src and patch".

Huh? What's so difficult about unpack source from version x, unpack from 
version y run a diff over the two directories and put that into 
patch-source-package z.

Now lets say we want to update our kernel from gentoo-sources-2.4.18 to 
gentoo-sources-2.4.19-r7. Portage sees there's already a linux-2.4.18.tar.bz2 
so we don't download the linux-2.4.19.tar.bz2 from kernel.org but instead 
download the patch from 2.4.18 to 2.4.19, unpack the 2.4.18 into our new 
directory, patch it to 2.4.19 and then patch in the stuff that makes a 
gentoo-sources-r7 out of it.

I don't see where this would be a problem either with the kernel or any other 
package. Of course you'd need quite some patches on the servers for that 
thats true :)

> There is nothing stopping anyone from creating thes ebuild themselves and
> posting them on the web say "diff2.4.16to2.4.19.ebuild" and giving people
> the ability to use them , even if they don't get commited.

No, don't build it into the ebuilds, let portage do it. The ebuild has nothing 
to do with it!


Alex
-- 
"They that can give up essential liberty to obtain a little temporary safety 
deserve neither liberty nor safety."
Benjamin Franklin

Re: [gentoo-dev] portage question

[Paul de Vrieze]

On Wednesday 02 October 2002 11:02, Alexander Gretencord wrote:
>
> Now lets say we want to update our kernel from gentoo-sources-2.4.18 to
> gentoo-sources-2.4.19-r7. Portage sees there's already a
> linux-2.4.18.tar.bz2 so we don't download the linux-2.4.19.tar.bz2 from
> kernel.org but instead download the patch from 2.4.18 to 2.4.19, unpack the
> 2.4.18 into our new directory, patch it to 2.4.19 and then patch in the
> stuff that makes a gentoo-sources-r7 out of it.
>
> I don't see where this would be a problem either with the kernel or any
> other package. Of course you'd need quite some patches on the servers for
> that thats true :)
>
> No, don't build it into the ebuilds, let portage do it. The ebuild has
> nothing to do with it!
>

I wish it was so simple. The only way that portage is going to know about 
patches is when we tell portage about them. This would involve some file that 
describes how to get from source 1 to source 2. It also only would work when 
the ebuild doesn't explicitly unpack the source file. 

In short this is very complex and not really supported by portages original 
design. It would be nice though, but at the moment there are more important 
things to do for portage.

Paul

-- 
Paul de Vrieze
Junior Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net

Re: [gentoo-dev] portage question

[Henti Smith]

> Huh? What's so difficult about unpack source from version x, unpack from 
> version y run a diff over the two directories and put that into 
> patch-source-package z.
> 
> Now lets say we want to update our kernel from gentoo-sources-2.4.18 to 
> gentoo-sources-2.4.19-r7. Portage sees there's already a linux-2.4.18.tar.bz2 
> so we don't download the linux-2.4.19.tar.bz2 from kernel.org but instead 
> download the patch from 2.4.18 to 2.4.19, unpack the 2.4.18 into our new 
> directory, patch it to 2.4.19 and then patch in the stuff that makes a 
> gentoo-sources-r7 out of it.

This should be fairly easy to implement .. 
patches have a naming standard as well .. or should have anyway .. 

having a look at kernel stuff .. the naming is : 

patch-2.4.19.bz2

So .. if you have linux-2.4.15.tar.bz2 installed portage know to grab 

patch-2.4.16.bz2
patch-2.4.17.bz2
patch-2.4.18.bz2
patch-2.4.19.bz2

and applying them in order to linux-2.4.15.tar.bz2

Seems pretty simple and straight forward to me ... 

> I don't see where this would be a problem either with the kernel or any other 
> package. Of course you'd need quite some patches on the servers for that 
> thats true :)
> 
> No, don't build it into the ebuilds, let portage do it. The ebuild has nothing 
> to do with it!

I agree ... this should be handled in portage .. not the ebuild .. 
yes there should be a PATCH_SRC_URI setting for the patches but thats about it 

Henti 

Re: [gentoo-dev] portage question

[Christian Skarby]

I kind of like the idea of this feature, but AFAIK are diffs most usefull
when there are little difference from the source-files we already have.
I.e. as soon as all the code is changed a diff-file will include both the
old and the new source. If we implement this feature I think it would be
nice to put some logic into it so that it can (f.ex. by looking at
file-sizes) decide if one should download a patch or two or rather
download the full source.

Christian

>> Huh? What's so difficult about unpack source from version x, unpack
>> from  version y run a diff over the two directories and put that into
>> patch-source-package z.
>>
>> Now lets say we want to update our kernel from gentoo-sources-2.4.18
>> to  gentoo-sources-2.4.19-r7. Portage sees there's already a
>> linux-2.4.18.tar.bz2  so we don't download the linux-2.4.19.tar.bz2
>> from kernel.org but instead  download the patch from 2.4.18 to 2.4.19,
>> unpack the 2.4.18 into our new  directory, patch it to 2.4.19 and then
>> patch in the stuff that makes a  gentoo-sources-r7 out of it.
>
> This should be fairly easy to implement ..
> patches have a naming standard as well .. or should have anyway ..
>
> having a look at kernel stuff .. the naming is :
>
> patch-2.4.19.bz2
>
> So .. if you have linux-2.4.15.tar.bz2 installed portage know to grab
>
> patch-2.4.16.bz2
> patch-2.4.17.bz2
> patch-2.4.18.bz2
> patch-2.4.19.bz2
>
> and applying them in order to linux-2.4.15.tar.bz2
>
> Seems pretty simple and straight forward to me ...
>
>> I don't see where this would be a problem either with the kernel or
>> any other  package. Of course you'd need quite some patches on the
>> servers for that  thats true :)
>>
>> No, don't build it into the ebuilds, let portage do it. The ebuild has
>> nothing  to do with it!
>
> I agree ... this should be handled in portage .. not the ebuild ..  yes
> there should be a PATCH_SRC_URI setting for the patches but thats about
> it
>
> Henti
> _______________________________________________
> gentoo-dev mailing list
> gentoo-dev@gentoo.org
> http://lists.gentoo.org/mailman/listinfo/gentoo-dev

Re: [gentoo-dev] portage question

[Henti Smith]

On Wed, 2 Oct 2002 13:53:39 +0200 (CEST)
"Christian Skarby" <christian@skarby.no> wrote:

> I kind of like the idea of this feature, but AFAIK are diffs most usefull
> when there are little difference from the source-files we already have.
> I.e. as soon as all the code is changed a diff-file will include both the
> old and the new source. If we implement this feature I think it would be
> nice to put some logic into it so that it can (f.ex. by looking at
> file-sizes) decide if one should download a patch or two or rather
> download the full source.

the whole point of a diff/patch file is to make only changes made to the source .. not carry the original .. 

Henti Smith 

Re: [gentoo-dev] portage question

[Christian Skarby]

> On Wed, 2 Oct 2002 13:53:39 +0200 (CEST)
> "Christian Skarby" <christian@skarby.no> wrote:
>
>> I kind of like the idea of this feature, but AFAIK are diffs most
>> usefull when there are little difference from the source-files we
>> already have. I.e. as soon as all the code is changed a diff-file will
>> include both the old and the new source. If we implement this feature
>> I think it would be nice to put some logic into it so that it can
>> (f.ex. by looking at file-sizes) decide if one should download a patch
>> or two or rather download the full source.
>
> the whole point of a diff/patch file is to make only changes made to the
> source .. not carry the original ..
>
> Henti Smith

:) I think I'll have to be a bit more spesific on this

If we have a source-1.0 that reads
/*
 This is a lovely comment
*/

and then a source-1.1 that reads
/*
 This comment make more sense
*/

a diff would look something like
/*
- This is a lovely comment
+ This comment make more sense
*/

As we see the diff is larger than both the sources, and this will happen
as soon as the all the source is fully replaced. This is actually a worst
case scenario and hence not a good example, nevertheless I believe that
fetching source-1.0.tar.gz and patch-1.1.tar.gz often will result in
downloading more than just source-1.1.tar.gz, thus it will not be cost
effective to get the patch unless one already have the sources that the
patch applies to. Therefor I believe that if we should implement this into
portage it would be nice to have some checks looking at what relevant
source-files we already have, how large the patches are and how large the
full source download is. Then it should be quite easy to consider what
will be the least time consuming download.

Taking in consideration that many (most/all?) gentoo-users keep their
systems up to date at all times with emerge -u world it probably would be
a great ideá just with patches, but I think that new installs and install
of packages with huge rewrites will benefit from having clean full source
downloads. Thus I suggest these checks.

Please arrest me if I am wrong ..

All the best,
Christian

Re: [gentoo-dev] portage question

[Henti Smith]

> :) I think I'll have to be a bit more spesific on this
> 
> If we have a source-1.0 that reads
> /*
>  This is a lovely comment
> */
> 
> and then a source-1.1 that reads
> /*
>  This comment make more sense
> */
> 
> a diff would look something like
> /*
> - This is a lovely comment
> + This comment make more sense
> */


this is verey understandable ...... but 

If you as a developer is going to change he's whole codebase and only 
bump the version from 1.0 to 1.0.1 ... peopl are going to shoot you !!! 
That is also another reason why people MUST learn to read changelogs before 
doing anything as a whole codebase shift should be commented in thw changelog.

> As we see the diff is larger than both the sources, and this will happen
> as soon as the all the source is fully replaced. This is actually a worst
> case scenario and hence not a good example, nevertheless I believe that
> fetching source-1.0.tar.gz and patch-1.1.tar.gz often will result in
> downloading more than just source-1.1.tar.gz, thus it will not be cost
> effective to get the patch unless one already have the sources that the
> patch applies to. Therefor I believe that if we should implement this into
> portage it would be nice to have some checks looking at what relevant
> source-files we already have, how large the patches are and how large the
> full source download is. Then it should be quite easy to consider what
> will be the least time consuming download.

99,9% of the time .. a patch will be smaller then the whole tarball .. 
yes there are duplicate lines in a patch .. but no changes will be THAT drastic as 
to make the patch bigger then the original source. 

Even if it is ... the developer who releases a patch bigger then the target complete source 
is not worth he's salt in this comunity

> Taking in consideration that many (most/all?) gentoo-users keep their
> systems up to date at all times with emerge -u world it probably would be
> a great ideá just with patches, but I think that new installs and install
> of packages with huge rewrites will benefit from having clean full source
> downloads. Thus I suggest these checks.

I think size is the only consern here .. as long as the md5 checks out only size should be considered.
It's smaller to download patches from verion x to y then to download y completely.

therefor get patches ... 

If there are people that HAVE to use source ... look at something like USE to pass the variable in 

USE="-x -gtk -patch" will no use X, GTK, or patches to make ... 
USE="x gtk patch" will use X, GTK and patches not the source ...

> Please arrest me if I am wrong ..

woulnd't dream of it ... freedom is what this is all about :) 

Henti Smith 

Re: [gentoo-dev] portage question

[Toby Dickenson]

(resend)

On Wednesday 02 Oct 2002 10:12 am, Paul de Vrieze wrote:
> On Wednesday 02 October 2002 11:02, Alexander Gretencord wrote:

> > No, don't build it into the ebuilds, let portage do it. The ebuild has
> > nothing to do with it!

> I wish it was so simple. The only way that portage is going to know about
> patches is when we tell portage about them. 

It could be made simple, with no changes to portage, if this bandwidth 
optimisation was encapsulated inside the download process....

If the uncompressed tar archives were made available for download from an 
rsync server then we could replace wget with a wrapper script that chooses a 
previously downloaded archive of an older version of the same package, and 
uses rsync to transfer the differences.


A few years ago this project <http://rproxy.samba.org/> tried to integrate 
exactly this process directly into the http protocol. It seems to be dead now 
:-(

Re: [gentoo-dev] portage question

[Toby Dickenson]

(re-re-send)


On Wednesday 02 Oct 2002 10:12 am, Paul de Vrieze wrote:
> On Wednesday 02 October 2002 11:02, Alexander Gretencord wrote:

> I wish it was so simple. The only way that portage is going to know about
> patches is when we tell portage about them.


It could be made simple, with no changes to portage, if this bandwidth 
optimisation was encapsulated inside the download process....

If the uncompressed tar archives were made available for download from an 
rsync server then we could replace wget with a wrapper script that chooses a 
previously downloaded archive of an older version of the same package, and 
uses rsync to transfer the differences.


A few years ago this project <http://rproxy.samba.org/> tried to integrate 
exactly this process directly into the http protocol. It seems to be dead now 
:-(

[gentoo-dev] Re: portage question

[Leon Chiver]

> This should be fairly easy to implement ..
> patches have a naming standard as well .. or should have anyway ..
....
> I agree ... this should be handled in portage .. not the ebuild ..
> yes there should be a PATCH_SRC_URI setting for the patches but thats

Of course, it shouldn't be too complicated to handle patches.

Let's say you want to install foo-100.ebuild. The ebuild could contain
some information about how to get the sources based on the previous
version of the package:

fo-100.ebuild:
--------------
foo-100 = foo-99 + pach_99-100.tar.bz2.

If you haven't installed foo-99 but only foo-98
on your computer, foo-100 can be downloaded completely or can be build
by applying a patch to foo-98 :

fo-99.ebuild:
-------------
foo-99 = foo-98 + patch_98-99.tar.bz2

Does this sound too weird?

Re: [gentoo-dev] portage question

[Alexander Gretencord]

On Wednesday 02 October 2002 14:23, Christian Skarby wrote:

> Taking in consideration that many (most/all?) gentoo-users keep their
> systems up to date at all times with emerge -u world it probably would be
> a great ideá just with patches, but I think that new installs and install
> of packages with huge rewrites will benefit from having clean full source
> downloads. Thus I suggest these checks.

I don't. I only upgrade if I need to or am feeling like it. I like gentoo for 
it's system not for the bleeding stuff. Of course I like current software 
(running my desktop on newest kernel, newest nvidia drivers etc.) but I don't 
really wanna bleed :)

And there's things like the new ghostscript 7.05 ebuild which just sucks! 
(Already bugreported lets see). I won't install the whole of gimp just 
because some ebuild writer was too lazy or using gimp anyway ... :)


Alex

-- 
"They that can give up essential liberty to obtain a little temporary safety 
deserve neither liberty nor safety."
Benjamin Franklin

Re: [gentoo-dev] Re: portage question

[Paul de Vrieze]

On Wednesday 02 October 2002 16:11, Leon Chiver wrote:

> Does this sound too weird?

No, just complicated.

Paul

-- 
Paul de Vrieze
Junior Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net