[gentoo-soc] GLEP 25 + GLEP 9

[gentoo-soc] GLEP 25 + GLEP 9

[mmacleod]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

Hello,

I am interested in proposing picking up where GLEP 9 and GLEP 25 left off and 
finally getting difference patches for portage distfiles working properly and 
implemented in such a way that they can become part of the official tree.
The proposal would address actual implementation of the above as well as 
modifying the various other tools that must use them such as "distfiles-clean" 
and portage itself of course.

Would this be worth proposing? I know that I am not the only one who would be 
very happy to finally see this make its way properly into Gentoo.
Which source repository should I submit a patch for if I want to propose this?



[-- Attachment #2: Type: text/html, Size: 1662 bytes --]

Re: [gentoo-soc] GLEP 25 + GLEP 9

[Jeremy Olexa]

mmacleod@webmail.co.za wrote:
> Hello,
> 
> 
> I am interested in proposing picking up where GLEP 9 and GLEP 25 left 
> off and finally getting difference patches for portage distfiles working 
> properly and implemented in such a way that they can become part of the 
> official tree.
> The proposal would address actual implementation of the above as well as 
> modifying the various other tools that must use them such as 
> "distfiles-clean" and portage itself of course.
> 
> 
> Would this be worth proposing? I know that I am not the only one who 
> would be very happy to finally see this make its way properly into Gentoo.
> Which source repository should I submit a patch for if I want to propose 
> this?

I just re-read the GLEPs. The first thing that comes to my mind is: Is 
there still a big use case for smaller downloads with the advance of 
broadband internet? However, I accept that not everyone has alot of 
bandwidth available to them and ignore my initial reaction ;)

There is a utility created by Brian Ferring (former Gentoo dev, author 
of GLEP 25) called dev-util/diffball that is used by infra to create the 
portage daily snapshot patches. 
http://distfiles.gentoo.org/snapshots/deltas/

Can you elaborate on where you envision this idea going? and perhaps 
some implementation (high level) details that you are interested in 
pursuing?

Thanks,
Jeremy

Re: [gentoo-soc] GLEP 25 + GLEP 9

[mmacleod]

> I just re-read the GLEPs. The first thing that comes to my mind is: Is
> there still a big use case for smaller downloads with the advance of
> broadband internet? However, I accept that not everyone has alot of
> bandwidth available to them and ignore my initial reaction ;)
I would just like to comment on possible motivations for this as well.

Some parts of the world have very low threshold for what they consider broad 
band, where I am from(South Africa) I have an ADSL line that is crippled to 
384k and am only allowed to do 3 gigabytes of traffic(combined 
upload/download) a month, after which my connection is completely cut off even 
for local browsing.
It is possible to buy more bandwidth and a faster line but the pricing is 
really prohibitive.
At these speeds the downloads do not take an insignificant time especially for 
a large package like open office(+-400mb)[1], it is not uncommon for me to 
have to delay updating packages to a new month in order to save money.
Sadly when it comes to Internet I am better off then many, it is not uncommon 
for people to use the internet through 56k/ISDN/GPRS here, I can only imagine 
there are other countries where this might be even worse.

The extreme large cases like open office are likely to benefit even people 
with fairly decent bandwidth/caps, it is an example of how designing software 
with the lowest target system in mind can benefit even those on faster 
systems.

Lower bandwidth load on mirrors, if a large amount of users make use of this 
the bandwidth load on mirrors would be substantially reduced. This in turn can 
lower the requirements needed to host a mirror and therefore make donated 
mirrors more likely.

Social responsibility. As internet users it is our responsibility to not waste 
large quantities of bandwidth unnecessarily, as it is eventually a finite 
resource that we all  have to pay for, as human beings it is also our 
responsibility in general to try and be less wasteful.
While the bandwidth usage of Gentoo might be a small drop in the ocean I am of 
the opinion that every small bit helps.
> There is a utility created by Brian Ferring (former Gentoo dev, author
> of GLEP 25) called dev-util/diffball that is used by infra to create the
> portage daily snapshot patches.
> http://distfiles.gentoo.org/snapshots/deltas/
There is also deltup created by the author of GLEP 9.
> Can you elaborate on where you envision this idea going? and perhaps
> some implementation (high level) details that you are interested in
> pursuing?
I would like to first reevaluate all the outcomes and assumptions of GLEP 9/25 
and see if there are areas where improvements could be made, I suspect there 
are one or two areas where this might be the case.
I also reevaluate the various options for which difference program should be 
used, starting with diffball and deltup, and then looking at a few others as 
well. I would want to examine each to see if there is room for improvement as 
well, and document these areas if possible.
I would then like to publish a new GLEP like document or even a GLEP itself 
and allow others a small period to comment on the new outcomes, or add any new 
ideas of there own.
I would then begin work on implementing the new "GLEP", starting with a server 
daemon in charge of generating the initial patch directory for the existing 
distfiles and new patches for new distfiles as they are added.
Finally I would do modifications to portage itself so that it can make use of 
the difference information where necessary and apply the patches etc. I would 
also look into other portage tools that should be aware of this, disfiles-
clean should take care of getting rid of patches that are no longer needed as 
well, knowledge of patches should be taken into account when chosing which 
distfiles to delete as well.
There would obviously be fairly large amounts of testing required for each of 
these parts as well.


[1] I realize open office might not be the best example as open office 
tarballs must be fetched from sun directly and are not stored on distfiles 
server. I don't think patches should be generated for externally hosted 
packages yet although it could be done if people want it. Manual adding of 
patches would also be possible so open office maintainers could potentially 
"manually" do differences for it as a special case.

Re: [gentoo-soc] GLEP 25 + GLEP 9

[Jeremy Olexa]

mmacleod@webmail.co.za wrote:
>> I just re-read the GLEPs. The first thing that comes to my mind is: Is
>> there still a big use case for smaller downloads with the advance of
>> broadband internet? However, I accept that not everyone has alot of
>> bandwidth available to them and ignore my initial reaction ;)
> I would just like to comment on possible motivations for this as well.
> 
> Some parts of the world have very low threshold for what they consider broad 
> band, where I am from(South Africa) I have an ADSL line that is crippled to 
> 384k and am only allowed to do 3 gigabytes of traffic(combined 
> upload/download) a month, after which my connection is completely cut off even 
> for local browsing.
> It is possible to buy more bandwidth and a faster line but the pricing is 
> really prohibitive.
> At these speeds the downloads do not take an insignificant time especially for 
> a large package like open office(+-400mb)[1], it is not uncommon for me to 
> have to delay updating packages to a new month in order to save money.
> Sadly when it comes to Internet I am better off then many, it is not uncommon 
> for people to use the internet through 56k/ISDN/GPRS here, I can only imagine 
> there are other countries where this might be even worse.
> 
<snip other good points>

I like it.
-Jeremy

Re: [gentoo-soc] GLEP 25 + GLEP 9

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 502 bytes --]

On 15:30 Sun 22 Mar     , Jeremy Olexa wrote:
> mmacleod@webmail.co.za wrote: There is a utility created by Brian 
> Ferring (former Gentoo dev, author of GLEP 25) called 
> dev-util/diffball that is used by infra to create the portage daily 
> snapshot patches.  http://distfiles.gentoo.org/snapshots/deltas/

You really need to catch ferringb on IRC to talk about this idea.

-- 
Thanks,
Donnie

Donnie Berkholz
Admin, Summer of Code
Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-soc] GLEP 25 + GLEP 9

[mmacleod]

On Thursday 02 April 2009 07:21:28 Donnie Berkholz wrote:
> On 15:30 Sun 22 Mar     , Jeremy Olexa wrote:
> > mmacleod@webmail.co.za wrote: There is a utility created by Brian
> > Ferring (former Gentoo dev, author of GLEP 25) called
> > dev-util/diffball that is used by infra to create the portage daily
> > snapshot patches.  http://distfiles.gentoo.org/snapshots/deltas/
>
> You really need to catch ferringb on IRC to talk about this idea.
Unfortunately I must leave for the next few days and will only return after 
the proposal deadline, so I won't have time to talk to ferringb on IRC before 
proposal deadline.(I have read a lot of threads/bugs etc.. where he has talked 
about this though)
I have submitted my proposal in the meantime.