[gentoo-user] Trying SRC_URI first during fetching

[gentoo-user] Trying SRC_URI first during fetching

[Nikos Chantziaras]

Normally, portage will try fetching from GENTOO_MIRRORS during an emerge 
and SRC_URI comes last.  I want to reverse this; try SRC_URI first and 
if that fails, only then proceed to fetch from GENTOO_MIRRORS.  Doable?

Re: [gentoo-user] Trying SRC_URI first during fetching

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

On Tue, 09 Jun 2009 07:16:32 +0300, Nikos Chantziaras wrote:

> Normally, portage will try fetching from GENTOO_MIRRORS during an
> emerge and SRC_URI comes last.  I want to reverse this; try SRC_URI
> first and if that fails, only then proceed to fetch from
> GENTOO_MIRRORS.  Doable?

You can use RESTRICT=nomirror to prevent portage trying the mirrors at
all. I don't know of a way of reversing the order in which things are
tried.


-- 
Neil Bothwick

Those are my principles. If you don't like them I have others.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Trying SRC_URI first during fetching

[Stroller]

On 9 Jun 2009, at 05:16, Nikos Chantziaras wrote:

> Normally, portage will try fetching from GENTOO_MIRRORS during an  
> emerge and SRC_URI comes last.  I want to reverse this; try SRC_URI  
> first and if that fails, only then proceed to fetch from  
> GENTOO_MIRRORS.  Doable?

This is undesirable behaviour - te mirrors exist because SRC_URI may  
often have limited bandwidth.

The package's hosting may be donated to the software's author, for  
instance, by a 3rd party, so when you go directly to SRC_URI, avoiding  
mirrors, you wear out the author's welcome.

Mirrors are hosted by people with gallons & gallons of bandwidth to  
spare, who expect you to use it.

It makes sense to use the mirrors FIRST.

Stroller.

Re: [gentoo-user] Trying SRC_URI first during fetching

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

On Tue, 9 Jun 2009 14:04:54 +0100, Stroller wrote:

> The package's hosting may be donated to the software's author, for  
> instance, by a 3rd party, so when you go directly to SRC_URI, avoiding  
> mirrors, you wear out the author's welcome.

There's another factor that some projects will change the contents of a 
tarball without bumping the name. If you get the altered file from
SRC_URI, emerge will bail out with a checksum failure.


-- 
Neil Bothwick

WITLAG: The delay between delivery and comprehension of a joke.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Trying SRC_URI first during fetching

[Alan McKinnon]

On Tuesday 09 June 2009 15:04:54 Stroller wrote:
> On 9 Jun 2009, at 05:16, Nikos Chantziaras wrote:
> > Normally, portage will try fetching from GENTOO_MIRRORS during an
> > emerge and SRC_URI comes last.  I want to reverse this; try SRC_URI
> > first and if that fails, only then proceed to fetch from
> > GENTOO_MIRRORS.  Doable?
>
> This is undesirable behaviour - te mirrors exist because SRC_URI may
> often have limited bandwidth.
>
> The package's hosting may be donated to the software's author, for
> instance, by a 3rd party, so when you go directly to SRC_URI, avoiding
> mirrors, you wear out the author's welcome.
>
> Mirrors are hosted by people with gallons & gallons of bandwidth to
> spare, who expect you to use it.
>
> It makes sense to use the mirrors FIRST.

Definitely.

I have to beat users over the head (metaphorically) with a stick to get them 
to use my mirror. They somehow have the idea that SRC_URI has better quality 
bits than my ftp server...

By going to SRC_URI every time, they use up precious international bandwidth 
instead of local (of which there is heaps). Every six months, when Fedora or 
Ubuntu does a release, those users can saturate the entire pipe into this 
*country* - just to get isos that I already have publicly available and am 
begging them to use. 


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Trying SRC_URI first during fetching

[Volker Armin Hemmann]

On Dienstag 09 Juni 2009, Alan McKinnon wrote:
> On Tuesday 09 June 2009 15:04:54 Stroller wrote:
> > On 9 Jun 2009, at 05:16, Nikos Chantziaras wrote:
> > > Normally, portage will try fetching from GENTOO_MIRRORS during an
> > > emerge and SRC_URI comes last.  I want to reverse this; try SRC_URI
> > > first and if that fails, only then proceed to fetch from
> > > GENTOO_MIRRORS.  Doable?
> >
> > This is undesirable behaviour - te mirrors exist because SRC_URI may
> > often have limited bandwidth.
> >
> > The package's hosting may be donated to the software's author, for
> > instance, by a 3rd party, so when you go directly to SRC_URI, avoiding
> > mirrors, you wear out the author's welcome.
> >
> > Mirrors are hosted by people with gallons & gallons of bandwidth to
> > spare, who expect you to use it.
> >
> > It makes sense to use the mirrors FIRST.
>
> Definitely.
>
> I have to beat users over the head (metaphorically) with a stick to get
> them to use my mirror. They somehow have the idea that SRC_URI has better
> quality bits than my ftp server...
>
> By going to SRC_URI every time, they use up precious international
> bandwidth instead of local (of which there is heaps). Every six months,
> when Fedora or Ubuntu does a release, those users can saturate the entire
> pipe into this *country* - just to get isos that I already have publicly
> available and am begging them to use.

my university has a nice volume cap for all students. But everything 
downloaded from its own network - including the ftp servers is 'free' - only 
outside traffic counts.

Luckily, my university hosts a major gentoo mirror. Not rsync, but distfiles. 
They also have ubuntu, suse, fedora stuff. Windows updates.. and still people 
don't use it. Annoying. 

Re: [gentoo-user] Trying SRC_URI first during fetching

[Alan McKinnon]

On Tuesday 09 June 2009 15:48:26 Volker Armin Hemmann wrote:
> On Dienstag 09 Juni 2009, Alan McKinnon wrote:

> > Definitely.
> >
> > I have to beat users over the head (metaphorically) with a stick to get
> > them to use my mirror. They somehow have the idea that SRC_URI has better
> > quality bits than my ftp server...
> >
> > By going to SRC_URI every time, they use up precious international
> > bandwidth instead of local (of which there is heaps). Every six months,
> > when Fedora or Ubuntu does a release, those users can saturate the entire
> > pipe into this *country* - just to get isos that I already have publicly
> > available and am begging them to use.
>
> my university has a nice volume cap for all students. But everything
> downloaded from its own network - including the ftp servers is 'free' -
> only outside traffic counts.
>
> Luckily, my university hosts a major gentoo mirror. Not rsync, but
> distfiles. They also have ubuntu, suse, fedora stuff. Windows updates.. and
> still people don't use it. Annoying.

I'm six hours behind every mirror I sync - all major distros, every free BSD I 
can find and every major project out there; the only thing that lags is Ubuntu 
and Fedora at release time. And give it away at local prices over ftp, http, 
rsync

Why? The company has 2000 employees. The international bandwidth bill is 
larger than the salary bill - including bonuses, expense claims, subsidies...

So management is very very happy that I have a way to reduce that, and rather 
unhappy that users don't use it more

-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: Trying SRC_URI first during fetching

[Nikos Chantziaras]

Stroller wrote:
> 
> On 9 Jun 2009, at 05:16, Nikos Chantziaras wrote:
> 
>> Normally, portage will try fetching from GENTOO_MIRRORS during an 
>> emerge and SRC_URI comes last.  I want to reverse this; try SRC_URI 
>> first and if that fails, only then proceed to fetch from 
>> GENTOO_MIRRORS.  Doable?
> 
> This is undesirable behaviour - te mirrors exist because SRC_URI may 
> often have limited bandwidth.

Understandable, but my problem lies in "ebuild digest" of my own ebuilds 
and ebuilds of third-party overlays.  Along the URLs tried by portage is 
one that seems to need over a minute to reply with "401 not found". 
This is highly annoying.  I would want to try SRC_URI in those cases first.

Of course that doesn't mean SRC_URI should be tried by default first.  I 
never suggested that and look what happened to this thread :P

Re: [gentoo-user] Re: Trying SRC_URI first during fetching

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

On Wed, 10 Jun 2009 02:36:04 +0300, Nikos Chantziaras wrote:

> Understandable, but my problem lies in "ebuild digest" of my own
> ebuilds and ebuilds of third-party overlays.  Along the URLs tried by
> portage is one that seems to need over a minute to reply with "401 not
> found". This is highly annoying.  I would want to try SRC_URI in those
> cases first.

With your own ebuilds there's no point in them trying to contact the
mirrors, so put RESTRICT="nomirror" in the ebuild.


-- 
Neil Bothwick

"Criminal Lawyer" is a redundancy.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Trying SRC_URI first during fetching

[Nikos Chantziaras]

On 06/10/2009 10:11 AM, Neil Bothwick wrote:
> On Wed, 10 Jun 2009 02:36:04 +0300, Nikos Chantziaras wrote:
>
>> Understandable, but my problem lies in "ebuild digest" of my own
>> ebuilds and ebuilds of third-party overlays.  Along the URLs tried by
>> portage is one that seems to need over a minute to reply with "401 not
>> found". This is highly annoying.  I would want to try SRC_URI in those
>> cases first.
>
> With your own ebuilds there's no point in them trying to contact the
> mirrors, so put RESTRICT="nomirror" in the ebuild.

Thanks.  That's pretty much what I was looking for.

Re: [gentoo-user] Re: Trying SRC_URI first during fetching

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

On Wed, 10 Jun 2009 10:19:11 +0300, Nikos Chantziaras wrote:

> > With your own ebuilds there's no point in them trying to contact the
> > mirrors, so put RESTRICT="nomirror" in the ebuild.  
> 
> Thanks.  That's pretty much what I was looking for.

That's what I thought, which is why I posted it when you first asked ;-)


-- 
Neil Bothwick

Top Oxymorons Number 47: Act naturally

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Trying SRC_URI first during fetching

[Arttu V.]

On 6/9/09, Nikos Chantziaras <realnc@arcor.de> wrote:
> Normally, portage will try fetching from GENTOO_MIRRORS during an emerge
> and SRC_URI comes last.  I want to reverse this; try SRC_URI first and
> if that fails, only then proceed to fetch from GENTOO_MIRRORS.  Doable?

As others have pointed out, caches, proxies et cetera are often Your
Friends. Thus, I wonder if there is something specific you are trying
to achieve here? Maybe these wise men from odd countries (like Mr
McKinnon's weird country on an unknown planet, with companies with
actual clueful management! ;) ) can help you even more if you reveal a
bit of the *why* behind the question.

-- 
Arttu V.

Re: [gentoo-user] Trying SRC_URI first during fetching

[Alan McKinnon]

On Tuesday 09 June 2009 17:56:41 Arttu V. wrote:
> On 6/9/09, Nikos Chantziaras <realnc@arcor.de> wrote:
> > Normally, portage will try fetching from GENTOO_MIRRORS during an emerge
> > and SRC_URI comes last.  I want to reverse this; try SRC_URI first and
> > if that fails, only then proceed to fetch from GENTOO_MIRRORS.  Doable?
>
> As others have pointed out, caches, proxies et cetera are often Your
> Friends. Thus, I wonder if there is something specific you are trying
> to achieve here? Maybe these wise men from odd countries (like Mr
> McKinnon's weird country on an unknown planet, with companies with
> actual clueful management! ;) )

We're a carrier-grade Telco/ISP in a third world country that pretends to be 
first world.

Techies rule here :-)



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Trying SRC_URI first during fetching

[Volker Armin Hemmann]

On Dienstag 09 Juni 2009, Alan McKinnon wrote:
> On Tuesday 09 June 2009 17:56:41 Arttu V. wrote:
> > On 6/9/09, Nikos Chantziaras <realnc@arcor.de> wrote:
> > > Normally, portage will try fetching from GENTOO_MIRRORS during an
> > > emerge and SRC_URI comes last.  I want to reverse this; try SRC_URI
> > > first and if that fails, only then proceed to fetch from
> > > GENTOO_MIRRORS.  Doable?
> >
> > As others have pointed out, caches, proxies et cetera are often Your
> > Friends. Thus, I wonder if there is something specific you are trying
> > to achieve here? Maybe these wise men from odd countries (like Mr
> > McKinnon's weird country on an unknown planet, with companies with
> > actual clueful management! ;) )
>
> We're a carrier-grade Telco/ISP in a third world country that pretends to
> be first world.
>
> Techies rule here :-)

let me guess: 
South Africa?

Re: [gentoo-user] Trying SRC_URI first during fetching

[Alan McKinnon]

On Tuesday 09 June 2009 20:26:29 Volker Armin Hemmann wrote:
> On Dienstag 09 Juni 2009, Alan McKinnon wrote:
> > On Tuesday 09 June 2009 17:56:41 Arttu V. wrote:
> > > On 6/9/09, Nikos Chantziaras <realnc@arcor.de> wrote:
> > > > Normally, portage will try fetching from GENTOO_MIRRORS during an
> > > > emerge and SRC_URI comes last.  I want to reverse this; try SRC_URI
> > > > first and if that fails, only then proceed to fetch from
> > > > GENTOO_MIRRORS.  Doable?
> > >
> > > As others have pointed out, caches, proxies et cetera are often Your
> > > Friends. Thus, I wonder if there is something specific you are trying
> > > to achieve here? Maybe these wise men from odd countries (like Mr
> > > McKinnon's weird country on an unknown planet, with companies with
> > > actual clueful management! ;) )
> >
> > We're a carrier-grade Telco/ISP in a third world country that pretends to
> > be first world.
> >
> > Techies rule here :-)
>
> let me guess:
> South Africa?

Correct first time :-)

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Trying SRC_URI first during fetching

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]

On Tue, 9 Jun 2009 20:49:43 +0200, Alan McKinnon wrote:

> > let me guess:
> > South Africa?  
> 
> Correct first time :-)

It's not hard to work out ;-)

>Received: from nazgul.localnet (196-210-153-123-rrdg-esr-2.dynamic.isadsl.co.za


-- 
Neil Bothwick

Remember the good old days, when CPU was singular?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Trying SRC_URI first during fetching

[Alan McKinnon]

On Tuesday 09 June 2009 23:57:54 Neil Bothwick wrote:
> On Tue, 9 Jun 2009 20:49:43 +0200, Alan McKinnon wrote:
> > > let me guess:
> > > South Africa?
> >
> > Correct first time :-)
>
> It's not hard to work out ;-)
>
> >Received: from nazgul.localnet
> > (196-210-153-123-rrdg-esr-2.dynamic.isadsl.co.za

Drat. Horrors. Now my secret is out :-)

And on a completely different but related topic, herewith a puzzle:

How long does it take to sync the brand new Fedora 11 release?
How long should it take?

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Trying SRC_URI first during fetching

[Volker Armin Hemmann]

On Mittwoch 10 Juni 2009, Alan McKinnon wrote:
> On Tuesday 09 June 2009 23:57:54 Neil Bothwick wrote:
> > On Tue, 9 Jun 2009 20:49:43 +0200, Alan McKinnon wrote:
> > > > let me guess:
> > > > South Africa?
> > >
> > > Correct first time :-)
> >
> > It's not hard to work out ;-)
> >
> > >Received: from nazgul.localnet
> > > (196-210-153-123-rrdg-esr-2.dynamic.isadsl.co.za
>
> Drat. Horrors. Now my secret is out :-)
>
> And on a completely different but related topic, herewith a puzzle:
>
> How long does it take to sync the brand new Fedora 11 release?

two hours?

> How long should it take?

1 second?

Re: [gentoo-user] Trying SRC_URI first during fetching

[Alan McKinnon]

On Wednesday 10 June 2009 00:17:43 Volker Armin Hemmann wrote:
> On Mittwoch 10 Juni 2009, Alan McKinnon wrote:
> > On Tuesday 09 June 2009 23:57:54 Neil Bothwick wrote:
> > > On Tue, 9 Jun 2009 20:49:43 +0200, Alan McKinnon wrote:
> > > > > let me guess:
> > > > > South Africa?
> > > >
> > > > Correct first time :-)
> > >
> > > It's not hard to work out ;-)
> > >
> > > >Received: from nazgul.localnet
> > > > (196-210-153-123-rrdg-esr-2.dynamic.isadsl.co.za
> >
> > Drat. Horrors. Now my secret is out :-)
> >
> > And on a completely different but related topic, herewith a puzzle:
> >
> > How long does it take to sync the brand new Fedora 11 release?
>
> two hours?

5 days, 9 hours, 27 minutes and counting

> > How long should it take?
>
> 1 second?

200G shouldn't take more than a day.

Part of that is a booboo on the Fedora master mirror (content was available, 
it went away, it came back).

That's bandwidth constraints for you. Into Africa it gets even worse. Total 
bandwidth to Kenya is not even 1M. International companies get their mail over 
dialup with fetchmail. And let's not even mention Zimbabwe...



-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] Trying SRC_URI first during fetching

[Volker Armin Hemmann]

On Mittwoch 10 Juni 2009, Alan McKinnon wrote:
> On Wednesday 10 June 2009 00:17:43 Volker Armin Hemmann wrote:
> > On Mittwoch 10 Juni 2009, Alan McKinnon wrote:
> > > On Tuesday 09 June 2009 23:57:54 Neil Bothwick wrote:
> > > > On Tue, 9 Jun 2009 20:49:43 +0200, Alan McKinnon wrote:
> > > > > > let me guess:
> > > > > > South Africa?
> > > > >
> > > > > Correct first time :-)
> > > >
> > > > It's not hard to work out ;-)
> > > >
> > > > >Received: from nazgul.localnet
> > > > > (196-210-153-123-rrdg-esr-2.dynamic.isadsl.co.za
> > >
> > > Drat. Horrors. Now my secret is out :-)
> > >
> > > And on a completely different but related topic, herewith a puzzle:
> > >
> > > How long does it take to sync the brand new Fedora 11 release?
> >
> > two hours?
>
> 5 days, 9 hours, 27 minutes and counting

sweet. That is even worse than I imagined.


> > 1 second?
>
> 200G shouldn't take more than a day.
>

I was joking, but yeah, more than a day starts to stink.  Small pipes and bad 
upstream are no a loveable, cuddly cute situation.

> Part of that is a booboo on the Fedora master mirror (content was
> available, it went away, it came back).

sounds like real fun ... the fun you wish your enemy to have.
>
> That's bandwidth constraints for you. Into Africa it gets even worse. Total
> bandwidth to Kenya is not even 1M. International companies get their mail
> over dialup with fetchmail.

that is indeed horrible. Nobody should be forced to use fetchmail.

> And let's not even mention Zimbabwe...

I am surprised that Zimbabwe still exists to be honest. But for some reason 
that trainwreck still jerks around. Just like a headless chicken.

Re: [gentoo-user] Trying SRC_URI first during fetching

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

On Wed, 10 Jun 2009 00:10:09 +0200, Alan McKinnon wrote:

> How long does it take to sync the brand new Fedora 11 release?

Don't know.

> How long should it take?

Don't care :)


-- 
Neil Bothwick

NOTICE:
  --  THE ELEVATORS WILL BE OUT OF ORDER TODAY  --
  (The nearest working elevators are in the building
   across the street.)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]