[gentoo-dev] Reducing the size of distfiles for our mirrors

[gentoo-dev] Reducing the size of distfiles for our mirrors

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

Folks --

Please avoid placing files in our distfile mirroring system that are not
absolutely necessary.  Yesterday, a 1.1GB file was dropped into it which
caused some of our mirrors to complain, especially when they discovered it
was a game file. 

In the last few months, our distfiles mirror has grown nearly 100% in terms
of space requirements.  Gentoo's increasing popularity also places greater
demands on our mirrors in terms of bandwidth requirements as well.  We need
to do our share to be respectful of these limited resources and use them
only when absolutely necessary.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Tue, Jul 22, 2003 at 08:58:51AM -0400 or thereabouts, Kurt Lieber wrote:
> In the last few months, our distfiles mirror has grown nearly 100% in terms
> of space requirements.  Gentoo's increasing popularity also places greater
> demands on our mirrors in terms of bandwidth requirements as well.  

To follow up to this message, I'm going to be working with avenj to come up
wtih a draft policy for changing the way we use our mirrors.  Any developer
or source mirror provider who is interested in participating, please drop
me an email.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Rainer Groesslinger]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1511 bytes --]

On Tuesday 22 July 2003 15:03, Kurt Lieber wrote:
> On Tue, Jul 22, 2003 at 08:58:51AM -0400 or thereabouts, Kurt Lieber wrote:
> > In the last few months, our distfiles mirror has grown nearly 100% in
> > terms of space requirements.  Gentoo's increasing popularity also places
> > greater demands on our mirrors in terms of bandwidth requirements as
> > well.
>
> To follow up to this message, I'm going to be working with avenj to come up
> wtih a draft policy for changing the way we use our mirrors.  Any developer
> or source mirror provider who is interested in participating, please drop
> me an email.
>
> --kurt

Perhaps you already thought about this but IMHO part of that policy should be 
that files which are stored on a network with a good mirroring infrastructure 
(sourceforge, kernel.org etc.) are never placed on our distfiles because it 
is very unlikely all 10 mirrors (or how many sourceforge ever has) are down 
and since the gentoo distfile mirrors are used before the 
mirror://sourceforge in the ebuild it causes more traffic to the mirrors 
(plus the space it takes) because nobody really gets it from sourceforge...

I think we could save a lot of bandwidth _and_ space if such files aren't 
placed on the gentoo mirrors if not absolutly neccessary, I don't see a real 
reason why we need to put hundreds of files on the gentoo mirrors which are 
already stored on a very good mirroring system.

-- 
Rainer Groesslinger
http://dev.gentoo.org/~scandium/

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Don Seiler]

My thoughts exactly.  Also IIRC portage will go through the list of
sf.net mirrors in the same pre-determined order every time.  Perhaps to
do some quasi-load-balancing we can have it randomly go through the
mirrors and/or allow the user to specify their preferred mirror in an
/etc/make.conf var like:

SOURCEFORGE_MIRROR=twtelecom

Perhaps this won't work but I thought I'd throw it out.

Don.

On Tue, Jul 22, 2003 at 03:24:09PM +0200, Rainer Groesslinger wrote:
> Perhaps you already thought about this but IMHO part of that policy should be 
> that files which are stored on a network with a good mirroring infrastructure 
> (sourceforge, kernel.org etc.) are never placed on our distfiles because it 
> is very unlikely all 10 mirrors (or how many sourceforge ever has) are down 
> and since the gentoo distfile mirrors are used before the 
> mirror://sourceforge in the ebuild it causes more traffic to the mirrors 
> (plus the space it takes) because nobody really gets it from sourceforge...
> 
> I think we could save a lot of bandwidth _and_ space if such files aren't 
> placed on the gentoo mirrors if not absolutly neccessary, I don't see a real 
> reason why we need to put hundreds of files on the gentoo mirrors which are 
> already stored on a very good mirroring system.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Rainer Groesslinger]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1011 bytes --]

On Tuesday 22 July 2003 15:35, Don Seiler wrote:
> My thoughts exactly.  Also IIRC portage will go through the list of
> sf.net mirrors in the same pre-determined order every time.  Perhaps to
> do some quasi-load-balancing we can have it randomly go through the
> mirrors and/or allow the user to specify their preferred mirror in an
> /etc/make.conf var like:
>
> SOURCEFORGE_MIRROR=twtelecom
>
> Perhaps this won't work but I thought I'd throw it out.
>
> Don.

That would be interesting, too.
Perhaps a make.conf setting "RANDOM_MIRROR" or something which is "yes" by 
default and then randomly picks a mirror from the list if a keyword from 
thirdpartymirrors is used...That's somewhat pseudo-balancing but better than 
now.

Very popular packages where you can expect a "rush" should still be mirrored 
to gentoo distfiles so that the thirdpartymirrors aren't overloaded by gentoo 
users just because we don't mirror it.

-- 
Rainer Groesslinger
http://dev.gentoo.org/~scandium/

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1425 bytes --]

On Tuesday 22 July 2003 15:24, Rainer Groesslinger wrote:
>
> Perhaps you already thought about this but IMHO part of that policy should
> be that files which are stored on a network with a good mirroring
> infrastructure (sourceforge, kernel.org etc.) are never placed on our
> distfiles because it is very unlikely all 10 mirrors (or how many
> sourceforge ever has) are down and since the gentoo distfile mirrors are
> used before the
> mirror://sourceforge in the ebuild it causes more traffic to the mirrors
> (plus the space it takes) because nobody really gets it from sourceforge...
>
> I think we could save a lot of bandwidth _and_ space if such files aren't
> placed on the gentoo mirrors if not absolutly neccessary, I don't see a
> real reason why we need to put hundreds of files on the gentoo mirrors
> which are already stored on a very good mirroring system.

Although I privately like to have everything on the distfile mirror as that 
WILL give me a higher bandwidth than sourceforge does, I agree with you.

I think we could start out by having portage not mirroring the mirror:/ url's 
and in the mirror list provide an extensive list of mirrors for the various 
packages. Then we can update the ebuilds to use the mirror url  scheme where 
aplicable. (For example for kde)

Paul

-- 
Paul de Vrieze
Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.cs.kun.nl/~pauldv

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Jon Portnoy]

On Tue, Jul 22, 2003 at 03:24:09PM +0200, Rainer Groesslinger wrote:
Content-Description: signed data
> On Tuesday 22 July 2003 15:03, Kurt Lieber wrote:
> > On Tue, Jul 22, 2003 at 08:58:51AM -0400 or thereabouts, Kurt Lieber wrote:
> > > In the last few months, our distfiles mirror has grown nearly 100% in
> > > terms of space requirements.  Gentoo's increasing popularity also places
> > > greater demands on our mirrors in terms of bandwidth requirements as
> > > well.
> >
> > To follow up to this message, I'm going to be working with avenj to come up
> > wtih a draft policy for changing the way we use our mirrors.  Any developer
> > or source mirror provider who is interested in participating, please drop
> > me an email.
> >
> > --kurt
> 
> Perhaps you already thought about this but IMHO part of that policy should be 
> that files which are stored on a network with a good mirroring infrastructure 
> (sourceforge, kernel.org etc.) are never placed on our distfiles because it 
> is very unlikely all 10 mirrors (or how many sourceforge ever has) are down 
> and since the gentoo distfile mirrors are used before the 
> mirror://sourceforge in the ebuild it causes more traffic to the mirrors 
> (plus the space it takes) because nobody really gets it from sourceforge...
> 
> I think we could save a lot of bandwidth _and_ space if such files aren't 
> placed on the gentoo mirrors if not absolutly neccessary, I don't see a real 
> reason why we need to put hundreds of files on the gentoo mirrors which are 
> already stored on a very good mirroring system.
> 
> -- 
> Rainer Groesslinger
> http://dev.gentoo.org/~scandium/


The idea is that if files move or go missing, users won't get hit with 
404s (and you'd be surprised as how often they move or go missing, 
too...)

-- 
Jon Portnoy
avenj/irc.freenode.net

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Fred Van Andel]

Jon Portnoy <avenj@gentoo.org> wrote:
(07/22/2003 13:16)

>The idea is that if files move or go missing, users won't get hit with 
>404s (and you'd be surprised as how often they move or go missing, 
>too...)

How about primary/secondary mirrors being implemented in portage?

If a file is available via sf.net then great, if it cannot be found then portage falls back on to the standard gentoo mirrors.

---
Fred Van Andel
fava@gentoo.org
GPG KeyID: 76526AD599455482  
GPG fingerprint: 64E4 4BAB 9C99 D565 3E3C F5D0 7652 6AD5 9945 5482


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Fred Van Andel]

Rainer Groesslinger <scandium@gentoo.org> wrote:
(07/22/2003 06:24)

>I think we could save a lot of bandwidth _and_ space if such files aren't 
>placed on the gentoo mirrors if not absolutly neccessary, I don't see a real 
>reason why we need to put hundreds of files on the gentoo mirrors which are 
>already stored on a very good mirroring system.

I am working on a p2p system specifically for distrubuting the contents of our  distfile mirrors. I hope to have proof of concept code ready for review in a couple of weeks.

Fred Van Andel
fava@gentoo.org
GPG KeyID: 76526AD599455482  
GPG fingerprint: 64E4 4BAB 9C99 D565 3E3C F5D0 7652 6AD5 9945 5482


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[John Davis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Fred Van Andel wrote:

|Rainer Groesslinger <scandium@gentoo.org> wrote:
|(07/22/2003 06:24)
|
|>I think we could save a lot of bandwidth _and_ space if such files aren't
|>placed on the gentoo mirrors if not absolutly neccessary, I don't see 
a real
|>reason why we need to put hundreds of files on the gentoo mirrors 
which are
|>already stored on a very good mirroring system.
|
|
|I am working on a p2p system specifically for distrubuting the contents 
of our  distfile mirrors. I hope to have proof of concept code ready for 
review in a couple of weeks.
|
|Fred Van Andel
|fava@gentoo.org
|GPG KeyID: 76526AD599455482  
|GPG fingerprint: 64E4 4BAB 9C99 D565 3E3C F5D0 7652 6AD5 9945 5482
|
|
|--
|gentoo-dev@gentoo.org mailing list
|
If I may ask, what p2p system are you going to use as the underlying 
structure?

Cheers,
//zhen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE/HaZpZlASNRlGLUcRAoA8AKDW1vLVUT/SyePC32ILAYP65sLorgCdFdia
VlrY8cm0VgP+aIcsqXPTPiY=
=v138
-----END PGP SIGNATURE-----



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Fred Van Andel]

John Davis <zhen@gentoo.org> wrote:
(07/22/2003 14:02)

>If I may ask, what p2p system are you going to use as the underlying 
>structure?

Mine, because it will allow better controll over what and how things are distrubuted.


Fred Van Andel
fava@gentoo.org
GPG KeyID: 76526AD599455482  
GPG fingerprint: 64E4 4BAB 9C99 D565 3E3C F5D0 7652 6AD5 9945 5482


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[John Davis]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


|
|Mine, because it will allow better controll over what and how things 
are distrubuted.
|
|
|Fred Van Andel
|fava@gentoo.org
|GPG KeyID: 76526AD599455482  
|GPG fingerprint: 64E4 4BAB 9C99 D565 3E3C F5D0 7652 6AD5 9945 5482
|
|
|--
|gentoo-dev@gentoo.org mailing list
|
Cool :)

Cheers,
//zhen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE/HdbSZlASNRlGLUcRAswfAKCxAADlzrj0x84PvEzLnNUxAxrjpQCeMuqA
CCgHerHWHlmp01JlbQ9DprA=
=fzut
-----END PGP SIGNATURE-----



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Stuart Herbert]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1984 bytes --]

On Tuesday 22 July 2003 2:24 pm, Rainer Groesslinger wrote:
> Perhaps you already thought about this but IMHO part of that policy should
> be that files which are stored on a network with a good mirroring
> infrastructure (sourceforge, kernel.org etc.) are never placed on our
> distfiles because it is very unlikely all 10 mirrors (or how many
> sourceforge ever has) are down and since the gentoo distfile mirrors are
> used before the
> mirror://sourceforge in the ebuild it causes more traffic to the mirrors
> (plus the space it takes) because nobody really gets it from sourceforge...
>
> I think we could save a lot of bandwidth _and_ space if such files aren't
> placed on the gentoo mirrors if not absolutly neccessary, I don't see a
> real reason why we need to put hundreds of files on the gentoo mirrors
> which are already stored on a very good mirroring system.

I've just thought of a problem with this.  Not necessarily about Sourceforge 
(although it needs to be thought about) but one in general.

If the source code's in our distfiles mirror system, we're immune from any 
changes / withdrawals made elsewhere.  Our ebuilds will continue to work 
until the package itself breaks, or we choose to replace it.

But if we're not mirroring the source code, then the ebuild breaks as soon as 
the source code tarball/rpm/whatever is withdrawn from the author's master 
site.  It shouldn't be a big problem, but it's worth thinking about.

(Just to clarify - I believe we shouldn't be mirroring commercial packages 
without very good reason ;-)

Best regards,
Stu
-- 
Stuart Herbert                                              stuart@gentoo.org
Gentoo Developer                                       http://www.gentoo.org/
Upcoming packages list                http://dev.gentoo.org/~stuart/packages/

GnuGP key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319  C549 0C2F 80BA F9AF C57C
--

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 799 bytes --]

On Friday 25 July 2003 22:04, Stuart Herbert wrote:
>
> If the source code's in our distfiles mirror system, we're immune from any
> changes / withdrawals made elsewhere.  Our ebuilds will continue to work
> until the package itself breaks, or we choose to replace it.
>
> But if we're not mirroring the source code, then the ebuild breaks as soon
> as the source code tarball/rpm/whatever is withdrawn from the author's
> master site.  It shouldn't be a big problem, but it's worth thinking about.
>

To battle such an event we could choose to automatically mirror the original 
location, and have a fill in file to come into place as soon as a file is 
removed from such a mirror.

Paul

-- 
Paul de Vrieze
Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

On Tue, Jul 22, 2003 at 11:42:18PM -0400 or thereabouts, Mike Frysinger wrote:
> yep, nwn, it was a beast to d/l myself.
> is the correct answer to this putting RESTRICT=nomirror ?  i thought that was 
> client side only ... it hadnt occured to me that it would affect how the 
> gentoo distfile mirrors would be created ...
> 
> if that is so i'll add RESTRICT=nomirror to a bunch of games ... many are a 
> few hundred megs (mostly q3 mods) ...

We're actually in the process of working out a policy to address these
issues.  The current *suggested* proposal is the following:

Per-file size limit:

commercial products > 100MB
non-commercial products > 500MB

The reasoning behind this is that commercial products typically have
resources and funding behind them to offer a fairly robust downloading
experience to their users meaning we can avoid mirroring them ourselves
without impacting our user base to a significant degree.  non-commercial
products often don't have the same resources available to them, so we would
continue to mirror all but the largest files in that case. (currently,
there are no non-commercial products in our tree that would be affected by
this cap limit)

Thoughts?

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Don Seiler]

I agree with your limits and your reasoning.

Don.

On Wed, Jul 23, 2003 at 08:56:12AM -0400, Kurt Lieber wrote:
> Per-file size limit:
> 
> commercial products > 100MB
> non-commercial products > 500MB
> 
> The reasoning behind this is that commercial products typically have
> resources and funding behind them to offer a fairly robust downloading
> experience to their users meaning we can avoid mirroring them ourselves
> without impacting our user base to a significant degree.  non-commercial
> products often don't have the same resources available to them, so we would
> continue to mirror all but the largest files in that case. (currently,
> there are no non-commercial products in our tree that would be affected by
> this cap limit)

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Chris Gianelloni]

While I tend to agree with you on your reasoning, what would you
consider a commercial product?

For example, many Linux games are volunteer efforts, even though they
may have a commercial entity behind the Windows version.  Here's an
example: America's Army.  It weighs in at something like 380MB, but the
Linux port is not offered at all from the official mirrors, but rather
from volunteer mirrors that Ryan Gordon (icculus) rounded up himself. 
Would it be in our best interest to take the load off the volunteer
servers?  I think so.

I guess my main concern is that we make a clear distinction between what
is commercial and what is not.

As for Bioware and Neverwinter Nights, it took them long enough to get a
client out, grab it from their servers... ;p

On Wed, 2003-07-23 at 08:56, Kurt Lieber wrote:
> On Tue, Jul 22, 2003 at 11:42:18PM -0400 or thereabouts, Mike Frysinger wrote:
> > yep, nwn, it was a beast to d/l myself.
> > is the correct answer to this putting RESTRICT=nomirror ?  i thought that was 
> > client side only ... it hadnt occured to me that it would affect how the 
> > gentoo distfile mirrors would be created ...
> > 
> > if that is so i'll add RESTRICT=nomirror to a bunch of games ... many are a 
> > few hundred megs (mostly q3 mods) ...
> 
> We're actually in the process of working out a policy to address these
> issues.  The current *suggested* proposal is the following:
> 
> Per-file size limit:
> 
> commercial products > 100MB
> non-commercial products > 500MB
> 
> The reasoning behind this is that commercial products typically have
> resources and funding behind them to offer a fairly robust downloading
> experience to their users meaning we can avoid mirroring them ourselves
> without impacting our user base to a significant degree.  non-commercial
> products often don't have the same resources available to them, so we would
> continue to mirror all but the largest files in that case. (currently,
> there are no non-commercial products in our tree that would be affected by
> this cap limit)
> 
> Thoughts?
> 
> --kurt


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Reducing the size of distfiles for our mirrors

[Kurt Lieber]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

On Wed, Jul 23, 2003 at 09:10:01AM -0400 or thereabouts, Chris Gianelloni wrote:
> While I tend to agree with you on your reasoning, what would you
> consider a commercial product?

Well, even if we decide on a policy, it doesn't mean we cannot have
exceptions.  Personally speaking, I'd say the AA example you gave would be
a valid execption.

My $.02.

--kurt

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]