[gentoo-user] What is the meaning of "mirror" URI scheme in ebuilds?

[gentoo-user] What is the meaning of "mirror" URI scheme in ebuilds?

[Ian Zimmerman]

Example from app-editors/emacs/emacs-24.5-r5 :

SRC_URI="mirror://gnu/emacs/${P}.tar.xz
        https://dev.gentoo.org/~ulm/emacs/${P}-patches-4.tar.xz"


-- 
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.

Re: [gentoo-user] What is the meaning of "mirror" URI scheme in ebuilds?

[David Haller]

Hello,

On Tue, 27 Mar 2018, Ian Zimmerman wrote:
>Example from app-editors/emacs/emacs-24.5-r5 :
>
>SRC_URI="mirror://gnu/emacs/${P}.tar.xz
>        https://dev.gentoo.org/~ulm/emacs/${P}-patches-4.tar.xz"

It means take one of the list specified in /etc/portage/mirrors/ or
${PORTAGE_DIR}/profiles/thirdpartymirrors

==== man 5 portage =====
       /etc/portage/
[..]
              mirrors
                     Whenever portage encounters a mirror:// style URI it will
                     look up the actual hosts here.  If the mirror set is  not
                     found  here,  it  will  check  the global mirrors file at
                     /usr/portage/profiles/thirdpartymirrors.
====

See also https://wiki.gentoo.org/wiki/GENTOO_MIRRORS

So, using your example, portage looks up the mirror for mirror://gnu
by looking in /etc/portage/mirrors (if it exists) and
/usr/portage/profiles/thirdpartymirrors for a line with the word "gnu"
at the beginning, and then tries the mirrors specified there. I.e.:

# grep '^gnu[[:space:]]' /usr/portage/profiles/thirdpartymirrors 
gnu             http://ftpmirror.gnu.org/ http://ftp.gnu.org/gnu/

So, portage will try http://ftpmirror.gnu.org/emacs/${P}.tar.xz and
http://ftp.gnu.org/gnu/emacs/${P}.tar.xz to retrieve the tarball.

HTH,
-dnh

-- 
"one of the ill effects of Microsoft's software is having to pay for it.
Lusers who pirate it are escaping this particular ill effect and aren't
driven quite as hard toward cheaper, better choices."  -- Mark Wells

[gentoo-user] Re: What is the meaning of "mirror" URI scheme in ebuilds?

[Ian Zimmerman]

On 2018-03-27 22:00, David Haller wrote:

> So, using your example, portage looks up the mirror for mirror://gnu
> by looking in /etc/portage/mirrors (if it exists) and
> /usr/portage/profiles/thirdpartymirrors for a line with the word "gnu"
> at the beginning, and then tries the mirrors specified there. I.e.:

Thanks, this helps.  I have a related follow-up question.

Sometimes, the SRC_URI in fact consists of multiple URIs.  What is
confusing about that is that in some cases the individual URI clearly
refer to copies of the same file (I'm guessing this provides redundant
locations where to download it from), while in other cases they're
clearly different files (for example upstream patches to be downloaded
separately).  There are even ebuilds when _both_ cases happen at the
same time, such as media-libs/freetype-2.9 :

SRC_URI="mirror://sourceforge/freetype/${P/_/}.tar.bz2
        mirror://nongnu/freetype/${P/_/}.tar.bz2
        utils?  ( mirror://sourceforge/freetype/ft2demos-${PV}.tar.bz2
                mirror://nongnu/freetype/ft2demos-${PV}.tar.bz2 )
        doc?    ( mirror://sourceforge/freetype/${PN}-doc-${PV}.tar.bz2
                mirror://nongnu/freetype/${PN}-doc-${PV}.tar.bz2 )"

So, WTH is the semantics of this?  How does portage know which URIs are
for the main tarball and which are the extras?  It has to know that to
avoid downloading the main tarball twice.

-- 
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.

Re: [gentoo-user] Re: What is the meaning of "mirror" URI scheme in ebuilds?

[David Haller]

Hello,

On Tue, 27 Mar 2018, Ian Zimmerman wrote:
>On 2018-03-27 22:00, David Haller wrote:
>
>> So, using your example, portage looks up the mirror for mirror://gnu
>> by looking in /etc/portage/mirrors (if it exists) and
>> /usr/portage/profiles/thirdpartymirrors for a line with the word "gnu"
>> at the beginning, and then tries the mirrors specified there. I.e.:
>
>Thanks, this helps.  I have a related follow-up question.
>
>Sometimes, the SRC_URI in fact consists of multiple URIs.  What is
>confusing about that is that in some cases the individual URI clearly
>refer to copies of the same file (I'm guessing this provides redundant
>locations where to download it from),

Yes.

>while in other cases they're clearly different files (for example
>upstream patches to be downloaded separately). There are even ebuilds
>when _both_ cases happen at the same time, such as
>media-libs/freetype-2.9 :
>
>SRC_URI="mirror://sourceforge/freetype/${P/_/}.tar.bz2
>        mirror://nongnu/freetype/${P/_/}.tar.bz2
>        utils?  ( mirror://sourceforge/freetype/ft2demos-${PV}.tar.bz2
>                mirror://nongnu/freetype/ft2demos-${PV}.tar.bz2 )
>        doc?    ( mirror://sourceforge/freetype/${PN}-doc-${PV}.tar.bz2
>                mirror://nongnu/freetype/${PN}-doc-${PV}.tar.bz2 )"
>
>So, WTH is the semantics of this?  How does portage know which URIs are
>for the main tarball and which are the extras?  It has to know that to
>avoid downloading the main tarball twice.

I, too, am not sure how that's handled in portage, much less what the
spec (PMS) say about it, but practical observation leads me to
believe, that for each $(basename $URI) (i.e. each file), portage
tries to get them from the URIs specified, in this case, it tries to
get the freetype ${P/_/}.tar.bz2 tarball from a mirror specified in
the sourceforge or nongnu lists of /etc/portage/mirrors resp. 
${PORTAGE_DIR}/profiles/thirdpartymirrors and then the ft2demos and
-doc- tarballs from (in this case) the same mirror-lists.

Basically, it seems portage build a list of (think '$(basename $URI)')

    file_to_get -> mirrorlist_or_direct_URI

i.e. here:

    freetype-${VER}.tar.bz2     -> URI_LIST
    ft2demos-${VER}.tar.bz2     -> URI_LIST
    freetype-doc-${VER}.tar.bz2 -> URI_LIST

and for each of those it tries the specified URIs (here
mirror://sourceforge and mirror://nongnu mirrorlists). You quite often
also see a "direct" non-mirror:// URI at the end (as a "last resort"
or so).

Anyway, that's my take on it.

And yes (referring to the bug mentioned by Michael in his answer),
quite some mirrors in the (current) thirdpartymirrors list are
outdated, which is why I copied that to /etc/portage/mirrors and
massaged those mirrorlists I noticed "broken" or "slow", either adding
my favored mirrors up front or replacing the complete list with some
chosen "known-good" mirrors.

But for the most-used mirror:// URIs in ebuilds I've come across,
besides the core gentoo list, it's mainly sourceforge or gnu or so. 
For the core "gentoo" mirrors, you can manage that mirrorlist via the
GENTOO_MIRRORS="" variable in make.conf, manually or via mirrorselect.

For me, it's rather easy, there's some "big", reliable "local" mirrors
(mostly at universities and related institutions) that mirror most
opensource software, from core projects (kernel, gnu) and various
projects (X, CTAN, CPAN) to various distros (SuSE, Gentoo, Debian, and
whatnot)... Weirdly enough, neither "thirdpartymirrors" nor
mirrorselect nor https://www.gentoo.org/downloads/mirrors/#DE list the
biggest mirror for gentoo-portage and -distfiles in .de, probably .eu ;)

*adding that one to my mirror-list*

HTH,
-dnh

-- 
Ceci n'est pas une .signature.

Re: [gentoo-user] What is the meaning of "mirror" URI scheme in ebuilds?

[Michael Orlitzky]

On 03/27/2018 02:50 PM, Ian Zimmerman wrote:
> Example from app-editors/emacs/emacs-24.5-r5 :
> 
> SRC_URI="mirror://gnu/emacs/${P}.tar.xz
>         https://dev.gentoo.org/~ulm/emacs/${P}-patches-4.tar.xz"
> 
> 


https://bugs.gentoo.org/635646