[gentoo-dev] [RFC] New third party mirrors

[gentoo-dev] [RFC] New third party mirrors

[Corentin Chary]

[-- Attachment #1: Type: text/plain, Size: 4923 bytes --]

Hello,

As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor)
I created a tool that counts URIs prefixes and sort them.
Here is the result of this script on the gentoo-x86 tree:

$ ./mirrors.py --all --count
1262    http://dev.gentoo.org
483     http://oss.tresys.com
482     http://oss.tresys.com/files
482     http://oss.tresys.com/files/refpolicy
473     http://xorg.freedesktop.org
473     http://xorg.freedesktop.org/releases
473     http://xorg.freedesktop.org/releases/individual
313     http://dev.gentoo.org/~swift
313     http://dev.gentoo.org/~swift/patches
309     http://dev.gentoo.org/~swift/patches/selinux-base-policy
297     http://pear.php.net
297     http://pear.php.net/get
296     http://hackage.haskell.org/packages
296     http://hackage.haskell.org/packages/archive
296     http://hackage.haskell.org
209     http://launchpad.net
206     http://ftp.xemacs.org
197     https://github.com
196     http://ftp.xemacs.org/pub
196     http://ftp.xemacs.org/pub/xemacs
193     http://ftp.xemacs.org/pub/xemacs/packages
179     http://gstreamer.freedesktop.org
179     http://gstreamer.freedesktop.org/src
176     http://github.com
175     http://linuxgazette.net/ftpfiles
175     http://linuxgazette.net
139     http://xorg.freedesktop.org/releases/individual/app
130     http://pear.horde.org/get
130     http://pear.horde.org
126     http://xorg.freedesktop.org/releases/individual/driver
115     http://dev.gentoo.org/~vapier
112     http://dev.gentoo.org/~vapier/dist
96      ftp://sources.redhat.com/pub
96      ftp://sources.redhat.com
94      http://savannah.nongnu.org/download
94      http://savannah.nongnu.org
92      http://xorg.freedesktop.org/releases/individual/lib
88      http://pecl.php.net
88      http://download.gna.org
88      http://pecl.php.net/get
84      http://dev.gentoo.org/~caster/distfiles
84      http://dev.gentoo.org/~caster
80      https://fedorahosted.org
74      http://get.qt.nokia.com
72      http://components.ez.no/get
72      http://components.ez.no
71      http://get.qt.nokia.com/qt
70      http://get.qt.nokia.com/qt/source
67      http://www.phrack.org/archives
67      http://www.phrack.org/archives/tgz

And here are some mirrors that can be deduced from that (except freebsd-jp
which I found reading the linphone ebuild):

freebsd-jp      ftp://ports.jp.FreeBSD.org/pub/FreeBSD-jp/ ftp://ftp.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp1.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp2.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp3.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp4.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp5.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp6.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp7.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp8.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp9.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/mirrors/FreeBSD-jp/ ftp://ftp.allbsd.org/pub/FreeBSD-jp/ ftp://ftp.nagaokaut.ac.jp/pub/FreeBSD-jp/ ftp://ftp.nara.wide.ad.jp/pub/FreeBSD-jp/ ftp://ftp.jaist.ac.jp/pub/FreeBSD-jp/
pear            http://pear.php.net/ http://de.pear.php.net/ http://us.pear.php.net/
redhat          ftp://sources.redhat.com/pub/ ftp://mirrors.kernel.org/sources.redhat.com/ ftp://mirrors.kernel.org/sources.redhat.com/ http://mirrors.kernel.org/sources.redhat.com/ http://sources-redhat.mirrors.airband.net/ ftp://gd.tuwien.ac.at/gnu/sourceware/ http://gd.tuwien.ac.at/gnu/sourceware/ ftp://ftp.gwdg.de/pub/linux/sources.redhat.com/ http://ftp.gwdg.de/pub/linux/sources.redhat.com/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/ http://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/ http://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://ftp.mirrorservice.org/sites/sources.redhat.com/pub/ http://www.mirrorservice.org/sites/sources.redhat.com/pub/
xemacs          http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
xorg            http://xorg.freedesktop.org/pub/ http://ftp.x.org/pub/

That would affect ~1000 URIs.
Unfortunatly I didn't found any mirror for:
- pecl.php.net
- hackage.haskell.org
- get.qt.nokia.com
- launchpad

But maybe we should still add them to thirdpartymirrors ? Look at 3dgamers, beyondunreal, bitbucket, github, gnome, kernel, xfce, vdrfiles and vdr-developerorg: they all only have one mirror !

For those afraid to add these mirrors and patch all affected ebuilds: no worry, mirrors.py can do it for you, and I already generated splitted patches generated from various thirdpartymirrors here: http://xf.iksaif.net/dev/gentoo/

Note that as part of https://bugs.gentoo.org/show_bug.cgi?id=405533 mgorny already commited a massive amount of patches adding mirror:// when the URI prefix was already present in thirdpartymirrors.

Thanks,

-- 
Corentin Chary
http://xf.iksaif.net/

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] New third party mirrors

[Robin H. Johnson]

On Tue, Apr 24, 2012 at 03:10:00PM +0200, Corentin Chary wrote:
> Hello,
> 
> As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor)
> I created a tool that counts URIs prefixes and sort them.
> Here is the result of this script on the gentoo-x86 tree:
> 
> $ ./mirrors.py --all --count
> 297     http://pear.php.net
> 297     http://pear.php.net/get
> 88      http://pecl.php.net
> 88      http://pecl.php.net/get
These are already mirror bouncers. If you visit the above, you'll get
the closest mirror for downloading.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] [RFC] New third party mirrors

[Corentin Chary]

On Tue, Apr 24, 2012 at 4:35 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Tue, Apr 24, 2012 at 03:10:00PM +0200, Corentin Chary wrote:
>> Hello,
>>
>> As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor)
>> I created a tool that counts URIs prefixes and sort them.
>> Here is the result of this script on the gentoo-x86 tree:
>>
>> $ ./mirrors.py --all --count
>> 297     http://pear.php.net
>> 297     http://pear.php.net/get
>> 88      http://pecl.php.net
>> 88      http://pecl.php.net/get
> These are already mirror bouncers. If you visit the above, you'll get
> the closest mirror for downloading.

And since there is already ~10 "mirrors" with only one actual backend,
should they go to thirdpartymirrors or not ? If not, what about this
pseudo-mirrors already present in thirdpartymirrors ?

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] [RFC] New third party mirrors

[Robin H. Johnson]

On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
> >> $ ./mirrors.py --all --count
> >> 297 ?? ?? http://pear.php.net
> >> 297 ?? ?? http://pear.php.net/get
> >> 88 ?? ?? ??http://pecl.php.net
> >> 88 ?? ?? ??http://pecl.php.net/get
> > These are already mirror bouncers. If you visit the above, you'll get
> > the closest mirror for downloading.
> And since there is already ~10 "mirrors" with only one actual backend,
> should they go to thirdpartymirrors or not ? If not, what about this
> pseudo-mirrors already present in thirdpartymirrors ?
I think we should add the pseudo-mirrors, but explicitly mark them as
such in the file, so that they don't get duplicate entries added (eg
adding us.pear, de.pear and the pear bouncer is bad. Should have just
the bouncer).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] [RFC] New third party mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

On Tue, 24 Apr 2012 16:19:11 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
> > >> $ ./mirrors.py --all --count
> > >> 297 ?? ?? http://pear.php.net
> > >> 297 ?? ?? http://pear.php.net/get
> > >> 88 ?? ?? ??http://pecl.php.net
> > >> 88 ?? ?? ??http://pecl.php.net/get
> > > These are already mirror bouncers. If you visit the above, you'll
> > > get the closest mirror for downloading.
> > And since there is already ~10 "mirrors" with only one actual
> > backend, should they go to thirdpartymirrors or not ? If not, what
> > about this pseudo-mirrors already present in thirdpartymirrors ?
> I think we should add the pseudo-mirrors, but explicitly mark them as
> such in the file, so that they don't get duplicate entries added (eg
> adding us.pear, de.pear and the pear bouncer is bad. Should have just
> the bouncer).

It'd be great if we could add some kind of additional mirror entries,
which would be used by repoman to signal missing mirror:// entries but
won't be used for downloads.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [RFC] New third party mirrors

[Zac Medico]

On 04/24/2012 09:38 AM, Michał Górny wrote:
> On Tue, 24 Apr 2012 16:19:11 +0000
> "Robin H. Johnson"<robbat2@gentoo.org>  wrote:
>
>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>>>>> $ ./mirrors.py --all --count
>>>>> 297 ?? ?? http://pear.php.net
>>>>> 297 ?? ?? http://pear.php.net/get
>>>>> 88 ?? ?? ??http://pecl.php.net
>>>>> 88 ?? ?? ??http://pecl.php.net/get
>>>> These are already mirror bouncers. If you visit the above, you'll
>>>> get the closest mirror for downloading.
>>> And since there is already ~10 "mirrors" with only one actual
>>> backend, should they go to thirdpartymirrors or not ? If not, what
>>> about this pseudo-mirrors already present in thirdpartymirrors ?
>> I think we should add the pseudo-mirrors, but explicitly mark them as
>> such in the file, so that they don't get duplicate entries added (eg
>> adding us.pear, de.pear and the pear bouncer is bad. Should have just
>> the bouncer).
>
> It'd be great if we could add some kind of additional mirror entries,
> which would be used by repoman to signal missing mirror:// entries but
> won't be used for downloads.

We could just add them in thirpartymirrors under fake names that nobody 
will use.
-- 
Thanks,
Zac

Re: [gentoo-dev] [RFC] New third party mirrors

[Hans de Graaff]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

On Tue, 2012-04-24 at 15:10 +0200, Corentin Chary wrote:

> And here are some mirrors that can be deduced from that (except freebsd-jp
> which I found reading the linphone ebuild):
> 
> xemacs          http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/

I only consider the first site to be a reliable source for the packages,
and this is what the eclass provides (only 4 packages actually provide a
SRC_URI, all others are handled through the eclass).

Kind regards,

Hans

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] [RFC] New third party mirrors

[Corentin Chary]

On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org> wrote:
> On Tue, 24 Apr 2012 16:19:11 +0000
> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>
>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>> > >> $ ./mirrors.py --all --count
>> > >> 297 ?? ?? http://pear.php.net
>> > >> 297 ?? ?? http://pear.php.net/get
>> > >> 88 ?? ?? ??http://pecl.php.net
>> > >> 88 ?? ?? ??http://pecl.php.net/get
>> > > These are already mirror bouncers. If you visit the above, you'll
>> > > get the closest mirror for downloading.
>> > And since there is already ~10 "mirrors" with only one actual
>> > backend, should they go to thirdpartymirrors or not ? If not, what
>> > about this pseudo-mirrors already present in thirdpartymirrors ?
>> I think we should add the pseudo-mirrors, but explicitly mark them as
>> such in the file, so that they don't get duplicate entries added (eg
>> adding us.pear, de.pear and the pear bouncer is bad. Should have just
>> the bouncer).
>
> It'd be great if we could add some kind of additional mirror entries,
> which would be used by repoman to signal missing mirror:// entries but
> won't be used for downloads.

Yep, we could put that in it too:
github		http://github.com/downloads/ https://github.com/downloads/
nongnu		http://download.savannah.nongnu.org/releases/

(I'm not really sure for nongnu).


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] [RFC] New third party mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1815 bytes --]

On Wed, 25 Apr 2012 09:16:05 +0200
Corentin Chary <corentin.chary@gmail.com> wrote:

> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > On Tue, 24 Apr 2012 16:19:11 +0000
> > "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> >
> >> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
> >> > >> $ ./mirrors.py --all --count
> >> > >> 297 ?? ?? http://pear.php.net
> >> > >> 297 ?? ?? http://pear.php.net/get
> >> > >> 88 ?? ?? ??http://pecl.php.net
> >> > >> 88 ?? ?? ??http://pecl.php.net/get
> >> > > These are already mirror bouncers. If you visit the above,
> >> > > you'll get the closest mirror for downloading.
> >> > And since there is already ~10 "mirrors" with only one actual
> >> > backend, should they go to thirdpartymirrors or not ? If not,
> >> > what about this pseudo-mirrors already present in
> >> > thirdpartymirrors ?
> >> I think we should add the pseudo-mirrors, but explicitly mark them
> >> as such in the file, so that they don't get duplicate entries
> >> added (eg adding us.pear, de.pear and the pear bouncer is bad.
> >> Should have just the bouncer).
> >
> > It'd be great if we could add some kind of additional mirror
> > entries, which would be used by repoman to signal missing mirror://
> > entries but won't be used for downloads.
> 
> Yep, we could put that in it too:
> github		http://github.com/downloads/
> https://github.com/downloads/

Per spec, portage can choose a random mirror of the list. If we put
entries like that, these two will be equally possible as the preferred
cloud. URL -- while they redirect one to another.

We might decide on some common syntax like preceding all extra entries
with '-' but I don't want to be the one deciding here.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [RFC] New third party mirrors

[Corentin Chary]

On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
> On Wed, 25 Apr 2012 09:16:05 +0200
> Corentin Chary <corentin.chary@gmail.com> wrote:
>
>> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org>
>> wrote:
>> > On Tue, 24 Apr 2012 16:19:11 +0000
>> > "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>> >
>> >> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>> >> > >> $ ./mirrors.py --all --count
>> >> > >> 297 ?? ?? http://pear.php.net
>> >> > >> 297 ?? ?? http://pear.php.net/get
>> >> > >> 88 ?? ?? ??http://pecl.php.net
>> >> > >> 88 ?? ?? ??http://pecl.php.net/get
>> >> > > These are already mirror bouncers. If you visit the above,
>> >> > > you'll get the closest mirror for downloading.
>> >> > And since there is already ~10 "mirrors" with only one actual
>> >> > backend, should they go to thirdpartymirrors or not ? If not,
>> >> > what about this pseudo-mirrors already present in
>> >> > thirdpartymirrors ?
>> >> I think we should add the pseudo-mirrors, but explicitly mark them
>> >> as such in the file, so that they don't get duplicate entries
>> >> added (eg adding us.pear, de.pear and the pear bouncer is bad.
>> >> Should have just the bouncer).
>> >
>> > It'd be great if we could add some kind of additional mirror
>> > entries, which would be used by repoman to signal missing mirror://
>> > entries but won't be used for downloads.
>>
>> Yep, we could put that in it too:
>> github                http://github.com/downloads/
>> https://github.com/downloads/
>
> Per spec, portage can choose a random mirror of the list. If we put
> entries like that, these two will be equally possible as the preferred
> cloud. URL -- while they redirect one to another.
>
> We might decide on some common syntax like preceding all extra entries
> with '-' but I don't want to be the one deciding here.

I checked, and current portage code already handle entries starting
with a - gracefully thanks to stack_dictlist (removing them from the
list of mirrors).

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] [RFC] New third party mirrors

[Zac Medico]

On 04/26/2012 12:30 AM, Corentin Chary wrote:
> On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> On Wed, 25 Apr 2012 09:16:05 +0200
>> Corentin Chary <corentin.chary@gmail.com> wrote:
>>
>>> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org>
>>> wrote:
>>>> On Tue, 24 Apr 2012 16:19:11 +0000
>>>> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>>>>
>>>>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>>>>>>>> $ ./mirrors.py --all --count
>>>>>>>> 297 ?? ?? http://pear.php.net
>>>>>>>> 297 ?? ?? http://pear.php.net/get
>>>>>>>> 88 ?? ?? ??http://pecl.php.net
>>>>>>>> 88 ?? ?? ??http://pecl.php.net/get
>>>>>>> These are already mirror bouncers. If you visit the above,
>>>>>>> you'll get the closest mirror for downloading.
>>>>>> And since there is already ~10 "mirrors" with only one actual
>>>>>> backend, should they go to thirdpartymirrors or not ? If not,
>>>>>> what about this pseudo-mirrors already present in
>>>>>> thirdpartymirrors ?
>>>>> I think we should add the pseudo-mirrors, but explicitly mark them
>>>>> as such in the file, so that they don't get duplicate entries
>>>>> added (eg adding us.pear, de.pear and the pear bouncer is bad.
>>>>> Should have just the bouncer).
>>>>
>>>> It'd be great if we could add some kind of additional mirror
>>>> entries, which would be used by repoman to signal missing mirror://
>>>> entries but won't be used for downloads.
>>>
>>> Yep, we could put that in it too:
>>> github                http://github.com/downloads/
>>> https://github.com/downloads/
>>
>> Per spec, portage can choose a random mirror of the list. If we put
>> entries like that, these two will be equally possible as the preferred
>> cloud. URL -- while they redirect one to another.
>>
>> We might decide on some common syntax like preceding all extra entries
>> with '-' but I don't want to be the one deciding here.
> 
> I checked, and current portage code already handle entries starting
> with a - gracefully thanks to stack_dictlist (removing them from the
> list of mirrors).

That means repoman will ignore them too. If you want existing versions
of repoman to check for those paths in SRC_URI, you can add a line like
this to thirdpartymirrors:

github-bad-urls http://github.com/downloads/ https://github.com/downloads/
-- 
Thanks,
Zac

Re: [gentoo-dev] [RFC] New third party mirrors

[Corentin Chary]

On Thu, Apr 26, 2012 at 9:57 AM, Zac Medico <zmedico@gentoo.org> wrote:
> On 04/26/2012 12:30 AM, Corentin Chary wrote:
>> On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
>>> On Wed, 25 Apr 2012 09:16:05 +0200
>>> Corentin Chary <corentin.chary@gmail.com> wrote:
>>>
>>>> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org>
>>>> wrote:
>>>>> On Tue, 24 Apr 2012 16:19:11 +0000
>>>>> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>>>>>
>>>>>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>>>>>>>>> $ ./mirrors.py --all --count
>>>>>>>>> 297 ?? ?? http://pear.php.net
>>>>>>>>> 297 ?? ?? http://pear.php.net/get
>>>>>>>>> 88 ?? ?? ??http://pecl.php.net
>>>>>>>>> 88 ?? ?? ??http://pecl.php.net/get
>>>>>>>> These are already mirror bouncers. If you visit the above,
>>>>>>>> you'll get the closest mirror for downloading.
>>>>>>> And since there is already ~10 "mirrors" with only one actual
>>>>>>> backend, should they go to thirdpartymirrors or not ? If not,
>>>>>>> what about this pseudo-mirrors already present in
>>>>>>> thirdpartymirrors ?
>>>>>> I think we should add the pseudo-mirrors, but explicitly mark them
>>>>>> as such in the file, so that they don't get duplicate entries
>>>>>> added (eg adding us.pear, de.pear and the pear bouncer is bad.
>>>>>> Should have just the bouncer).
>>>>>
>>>>> It'd be great if we could add some kind of additional mirror
>>>>> entries, which would be used by repoman to signal missing mirror://
>>>>> entries but won't be used for downloads.
>>>>
>>>> Yep, we could put that in it too:
>>>> github                http://github.com/downloads/
>>>> https://github.com/downloads/
>>>
>>> Per spec, portage can choose a random mirror of the list. If we put
>>> entries like that, these two will be equally possible as the preferred
>>> cloud. URL -- while they redirect one to another.
>>>
>>> We might decide on some common syntax like preceding all extra entries
>>> with '-' but I don't want to be the one deciding here.
>>
>> I checked, and current portage code already handle entries starting
>> with a - gracefully thanks to stack_dictlist (removing them from the
>> list of mirrors).
>
> That means repoman will ignore them too. If you want existing versions
> of repoman to check for those paths in SRC_URI, you can add a line like
> this to thirdpartymirrors:
>
> github-bad-urls http://github.com/downloads/ https://github.com/downloads/

Hum, I checked repoman source code, and I didn't find where it checks
if SRC_URI matches something in thirdpartymirror. Any hint ?


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] [RFC] New third party mirrors

[Zac Medico]

On 04/26/2012 01:03 AM, Corentin Chary wrote:
> On Thu, Apr 26, 2012 at 9:57 AM, Zac Medico <zmedico@gentoo.org> wrote:
>> On 04/26/2012 12:30 AM, Corentin Chary wrote:
>>> On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
>>>> On Wed, 25 Apr 2012 09:16:05 +0200
>>>> Corentin Chary <corentin.chary@gmail.com> wrote:
>>>>
>>>>> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org>
>>>>> wrote:
>>>>>> On Tue, 24 Apr 2012 16:19:11 +0000
>>>>>> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>>>>>>
>>>>>>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>>>>>>>>>> $ ./mirrors.py --all --count
>>>>>>>>>> 297 ?? ?? http://pear.php.net
>>>>>>>>>> 297 ?? ?? http://pear.php.net/get
>>>>>>>>>> 88 ?? ?? ??http://pecl.php.net
>>>>>>>>>> 88 ?? ?? ??http://pecl.php.net/get
>>>>>>>>> These are already mirror bouncers. If you visit the above,
>>>>>>>>> you'll get the closest mirror for downloading.
>>>>>>>> And since there is already ~10 "mirrors" with only one actual
>>>>>>>> backend, should they go to thirdpartymirrors or not ? If not,
>>>>>>>> what about this pseudo-mirrors already present in
>>>>>>>> thirdpartymirrors ?
>>>>>>> I think we should add the pseudo-mirrors, but explicitly mark them
>>>>>>> as such in the file, so that they don't get duplicate entries
>>>>>>> added (eg adding us.pear, de.pear and the pear bouncer is bad.
>>>>>>> Should have just the bouncer).
>>>>>>
>>>>>> It'd be great if we could add some kind of additional mirror
>>>>>> entries, which would be used by repoman to signal missing mirror://
>>>>>> entries but won't be used for downloads.
>>>>>
>>>>> Yep, we could put that in it too:
>>>>> github                http://github.com/downloads/
>>>>> https://github.com/downloads/
>>>>
>>>> Per spec, portage can choose a random mirror of the list. If we put
>>>> entries like that, these two will be equally possible as the preferred
>>>> cloud. URL -- while they redirect one to another.
>>>>
>>>> We might decide on some common syntax like preceding all extra entries
>>>> with '-' but I don't want to be the one deciding here.
>>>
>>> I checked, and current portage code already handle entries starting
>>> with a - gracefully thanks to stack_dictlist (removing them from the
>>> list of mirrors).
>>
>> That means repoman will ignore them too. If you want existing versions
>> of repoman to check for those paths in SRC_URI, you can add a line like
>> this to thirdpartymirrors:
>>
>> github-bad-urls http://github.com/downloads/ https://github.com/downloads/
> 
> Hum, I checked repoman source code, and I didn't find where it checks
> if SRC_URI matches something in thirdpartymirror. Any hint ?

Search for SRC_URI.mirror in /usr/bin/repoman.
-- 
Thanks,
Zac

Re: [gentoo-dev] [RFC] New third party mirrors

[Corentin Chary]

On Thu, Apr 26, 2012 at 10:07 AM, Zac Medico <zmedico@gentoo.org> wrote:
> On 04/26/2012 01:03 AM, Corentin Chary wrote:
>> On Thu, Apr 26, 2012 at 9:57 AM, Zac Medico <zmedico@gentoo.org> wrote:
>>> On 04/26/2012 12:30 AM, Corentin Chary wrote:
>>>> On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
>>>>> On Wed, 25 Apr 2012 09:16:05 +0200
>>>>> Corentin Chary <corentin.chary@gmail.com> wrote:
>>>>>
>>>>>> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny <mgorny@gentoo.org>
>>>>>> wrote:
>>>>>>> On Tue, 24 Apr 2012 16:19:11 +0000
>>>>>>> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
>>>>>>>
>>>>>>>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote:
>>>>>>>>>>> $ ./mirrors.py --all --count
>>>>>>>>>>> 297 ?? ?? http://pear.php.net
>>>>>>>>>>> 297 ?? ?? http://pear.php.net/get
>>>>>>>>>>> 88 ?? ?? ??http://pecl.php.net
>>>>>>>>>>> 88 ?? ?? ??http://pecl.php.net/get
>>>>>>>>>> These are already mirror bouncers. If you visit the above,
>>>>>>>>>> you'll get the closest mirror for downloading.
>>>>>>>>> And since there is already ~10 "mirrors" with only one actual
>>>>>>>>> backend, should they go to thirdpartymirrors or not ? If not,
>>>>>>>>> what about this pseudo-mirrors already present in
>>>>>>>>> thirdpartymirrors ?
>>>>>>>> I think we should add the pseudo-mirrors, but explicitly mark them
>>>>>>>> as such in the file, so that they don't get duplicate entries
>>>>>>>> added (eg adding us.pear, de.pear and the pear bouncer is bad.
>>>>>>>> Should have just the bouncer).
>>>>>>>
>>>>>>> It'd be great if we could add some kind of additional mirror
>>>>>>> entries, which would be used by repoman to signal missing mirror://
>>>>>>> entries but won't be used for downloads.
>>>>>>
>>>>>> Yep, we could put that in it too:
>>>>>> github                http://github.com/downloads/
>>>>>> https://github.com/downloads/
>>>>>
>>>>> Per spec, portage can choose a random mirror of the list. If we put
>>>>> entries like that, these two will be equally possible as the preferred
>>>>> cloud. URL -- while they redirect one to another.
>>>>>
>>>>> We might decide on some common syntax like preceding all extra entries
>>>>> with '-' but I don't want to be the one deciding here.
>>>>
>>>> I checked, and current portage code already handle entries starting
>>>> with a - gracefully thanks to stack_dictlist (removing them from the
>>>> list of mirrors).
>>>
>>> That means repoman will ignore them too. If you want existing versions
>>> of repoman to check for those paths in SRC_URI, you can add a line like
>>> this to thirdpartymirrors:
>>>
>>> github-bad-urls http://github.com/downloads/ https://github.com/downloads/
>>
>> Hum, I checked repoman source code, and I didn't find where it checks
>> if SRC_URI matches something in thirdpartymirror. Any hint ?
>
> Search for SRC_URI.mirror in /usr/bin/repoman.

Arg.. ok, I only looked in pym/repoman/.

So two solutions here:

First one:
github http://cloud.github.com/downloads -http://github.com/downloads/
-https://github.com/downloads/
+ a small patch that would allow repoman to do something like
settings.thirdpartymirrors(keep_bad_uris=True) in order to keep uris
starting with a - in the list.

Second solution:
github http://cloud.github.com/downloads
github-bad-uris -http://github.com/downloads/ -https://github.com/downloads/

The good thing with the first one is that it would allow repoman to
outputs something like "you should use 'mirror://github'".


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] [RFC] New third party mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Thu, 26 Apr 2012 10:21:36 +0200
Corentin Chary <corentin.chary@gmail.com> wrote:

> Second solution:
> github http://cloud.github.com/downloads
> github-bad-uris -http://github.com/downloads/
-https://github.com/downloads/
> 
> The good thing with the first one is that it would allow repoman to
> outputs something like "you should use 'mirror://github'".

Well, we could decide on something common and special like:

github:bad-uris http://.....

And then let repoman suggest using mirror with ':bad-uris' stripped.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [RFC] New third party mirrors

[Corentin Chary]

On Thu, Apr 26, 2012 at 8:41 PM, Michał Górny <mgorny@gentoo.org> wrote:
> On Thu, 26 Apr 2012 10:21:36 +0200
> Corentin Chary <corentin.chary@gmail.com> wrote:
>
>> Second solution:
>> github http://cloud.github.com/downloads
>> github-bad-uris -http://github.com/downloads/
> -https://github.com/downloads/
>>
>> The good thing with the first one is that it would allow repoman to
>> outputs something like "you should use 'mirror://github'".
>
> Well, we could decide on something common and special like:
>
> github:bad-uris http://.....
>
> And then let repoman suggest using mirror with ':bad-uris' stripped.

Works for me. What would be the next step to push this ?


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] [RFC] New third party mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

On Fri, 27 Apr 2012 10:06:38 +0200
Corentin Chary <corentin.chary@gmail.com> wrote:

> On Thu, Apr 26, 2012 at 8:41 PM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > On Thu, 26 Apr 2012 10:21:36 +0200
> > Corentin Chary <corentin.chary@gmail.com> wrote:
> >
> >> Second solution:
> >> github http://cloud.github.com/downloads
> >> github-bad-uris -http://github.com/downloads/
> > -https://github.com/downloads/
> >>
> >> The good thing with the first one is that it would allow repoman to
> >> outputs something like "you should use 'mirror://github'".
> >
> > Well, we could decide on something common and special like:
> >
> > github:bad-uris http://.....
> >
> > And then let repoman suggest using mirror with ':bad-uris' stripped.
> 
> Works for me. What would be the next step to push this ?

Next step would be to find a good name :P. I suggest going the Perl way
and prepending '~' ;P.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]