[gentoo-dev] [RFC] New third party mirrors

[Corentin Chary <corentin.chary@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4923 bytes --]

Hello,

As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor)
I created a tool that counts URIs prefixes and sort them.
Here is the result of this script on the gentoo-x86 tree:

$ ./mirrors.py --all --count
1262    http://dev.gentoo.org
483     http://oss.tresys.com
482     http://oss.tresys.com/files
482     http://oss.tresys.com/files/refpolicy
473     http://xorg.freedesktop.org
473     http://xorg.freedesktop.org/releases
473     http://xorg.freedesktop.org/releases/individual
313     http://dev.gentoo.org/~swift
313     http://dev.gentoo.org/~swift/patches
309     http://dev.gentoo.org/~swift/patches/selinux-base-policy
297     http://pear.php.net
297     http://pear.php.net/get
296     http://hackage.haskell.org/packages
296     http://hackage.haskell.org/packages/archive
296     http://hackage.haskell.org
209     http://launchpad.net
206     http://ftp.xemacs.org
197     https://github.com
196     http://ftp.xemacs.org/pub
196     http://ftp.xemacs.org/pub/xemacs
193     http://ftp.xemacs.org/pub/xemacs/packages
179     http://gstreamer.freedesktop.org
179     http://gstreamer.freedesktop.org/src
176     http://github.com
175     http://linuxgazette.net/ftpfiles
175     http://linuxgazette.net
139     http://xorg.freedesktop.org/releases/individual/app
130     http://pear.horde.org/get
130     http://pear.horde.org
126     http://xorg.freedesktop.org/releases/individual/driver
115     http://dev.gentoo.org/~vapier
112     http://dev.gentoo.org/~vapier/dist
96      ftp://sources.redhat.com/pub
96      ftp://sources.redhat.com
94      http://savannah.nongnu.org/download
94      http://savannah.nongnu.org
92      http://xorg.freedesktop.org/releases/individual/lib
88      http://pecl.php.net
88      http://download.gna.org
88      http://pecl.php.net/get
84      http://dev.gentoo.org/~caster/distfiles
84      http://dev.gentoo.org/~caster
80      https://fedorahosted.org
74      http://get.qt.nokia.com
72      http://components.ez.no/get
72      http://components.ez.no
71      http://get.qt.nokia.com/qt
70      http://get.qt.nokia.com/qt/source
67      http://www.phrack.org/archives
67      http://www.phrack.org/archives/tgz

And here are some mirrors that can be deduced from that (except freebsd-jp
which I found reading the linphone ebuild):

freebsd-jp      ftp://ports.jp.FreeBSD.org/pub/FreeBSD-jp/ ftp://ftp.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp1.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp2.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp3.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp4.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp5.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp6.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp7.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp8.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp9.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/mirrors/FreeBSD-jp/ ftp://ftp.allbsd.org/pub/FreeBSD-jp/ ftp://ftp.nagaokaut.ac.jp/pub/FreeBSD-jp/ ftp://ftp.nara.wide.ad.jp/pub/FreeBSD-jp/ ftp://ftp.jaist.ac.jp/pub/FreeBSD-jp/
pear            http://pear.php.net/ http://de.pear.php.net/ http://us.pear.php.net/
redhat          ftp://sources.redhat.com/pub/ ftp://mirrors.kernel.org/sources.redhat.com/ ftp://mirrors.kernel.org/sources.redhat.com/ http://mirrors.kernel.org/sources.redhat.com/ http://sources-redhat.mirrors.airband.net/ ftp://gd.tuwien.ac.at/gnu/sourceware/ http://gd.tuwien.ac.at/gnu/sourceware/ ftp://ftp.gwdg.de/pub/linux/sources.redhat.com/ http://ftp.gwdg.de/pub/linux/sources.redhat.com/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/ http://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/ http://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://ftp.mirrorservice.org/sites/sources.redhat.com/pub/ http://www.mirrorservice.org/sites/sources.redhat.com/pub/
xemacs          http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/
xorg            http://xorg.freedesktop.org/pub/ http://ftp.x.org/pub/

That would affect ~1000 URIs.
Unfortunatly I didn't found any mirror for:
- pecl.php.net
- hackage.haskell.org
- get.qt.nokia.com
- launchpad

But maybe we should still add them to thirdpartymirrors ? Look at 3dgamers, beyondunreal, bitbucket, github, gnome, kernel, xfce, vdrfiles and vdr-developerorg: they all only have one mirror !

For those afraid to add these mirrors and patch all affected ebuilds: no worry, mirrors.py can do it for you, and I already generated splitted patches generated from various thirdpartymirrors here: http://xf.iksaif.net/dev/gentoo/

Note that as part of https://bugs.gentoo.org/show_bug.cgi?id=405533 mgorny already commited a massive amount of patches adding mirror:// when the URI prefix was already present in thirdpartymirrors.

Thanks,

-- 
Corentin Chary
http://xf.iksaif.net/

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]