[gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Robin H. Johnson]

This is a small feature request, but it will require a modification to
PMS, so I describe it here.

The present thirdpartymirrors file is unwieldy, and difficult to manage
due to it's format with very long lines. It also doesn't permit easy
comments. Presently commits to it look very ugly, because diffs are
line-based, and we pack a lot into each line.

I would like to make it a directory instead of a single file, and extend
the internal syntax.

1. New location: $PROFILEDIR/thirdpartymirrors/$MIRRORNAME
1.1. The name of the mirror is now the name of the file.
1.2. We can have a file extension of .mirrors if somebody would like
     that.

2. New format (for directory-mode):
2.1. Comments permitted, shell-style.
2.2. Blank lines ignored
2.3. One URL per line, optionally prefixed with "-" or "+"
2.4. For stack repos/overlays:
2.4.1. No prefix: replace all prior mirrors from masters with new URLS in this file.
2.4.2. "-" prefix: remove this URL from the list from masters.
2.4.2. "+" prefix: append this URL to the list from masters.

3. New format (for file-mode):
3.1. This is for cases where thirdpartymirrors is still a file.
3.2. The first token on a line remains the name of the mirror.
3.3. Each subsequent token may be prefixed with "+" or "-", and impacts
     prior lines/masters.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Alex Xu]

[-- Attachment #1: Type: text/plain, Size: 1760 bytes --]

On 06/01/14 03:20 PM, Robin H. Johnson wrote:
> This is a small feature request, but it will require a modification to
> PMS, so I describe it here.
> 
> The present thirdpartymirrors file is unwieldy, and difficult to manage
> due to it's format with very long lines. It also doesn't permit easy
> comments. Presently commits to it look very ugly, because diffs are
> line-based, and we pack a lot into each line.
> 
> I would like to make it a directory instead of a single file, and extend
> the internal syntax.

I like the idea, but I'm not too sure about the execution.

> 1. New location: $PROFILEDIR/thirdpartymirrors/$MIRRORNAME
> 1.1. The name of the mirror is now the name of the file.
> 1.2. We can have a file extension of .mirrors if somebody would like
>      that.
> 2. New format (for directory-mode):
> 2.1. Comments permitted, shell-style.
> 2.2. Blank lines ignored
> 2.3. One URL per line, optionally prefixed with "-" or "+"
> 2.4. For stack repos/overlays:
> 2.4.1. No prefix: replace all prior mirrors from masters with new URLS in this file.
> 2.4.2. "-" prefix: remove this URL from the list from masters.
> 2.4.2. "+" prefix: append this URL to the list from masters.

So if *any* line doesn't have a prefix, then *everything* gets
overwritten? What about the prior mirrors listed in the file?

There needs to be some mechanism for specifying this, but I don't think
this is it.

Perhaps a header with a special line?

> 3. New format (for file-mode):
> 3.1. This is for cases where thirdpartymirrors is still a file.
> 3.2. The first token on a line remains the name of the mirror.
> 3.3. Each subsequent token may be prefixed with "+" or "-", and impacts
>      prior lines/masters.
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Robin H. Johnson]

On Mon, Jan 06, 2014 at 04:09:01PM -0500, Alex Xu wrote:
> > 1. New location: $PROFILEDIR/thirdpartymirrors/$MIRRORNAME
> > 1.1. The name of the mirror is now the name of the file.
> > 1.2. We can have a file extension of .mirrors if somebody would like
> >      that.
> > 2. New format (for directory-mode):
> > 2.1. Comments permitted, shell-style.
> > 2.2. Blank lines ignored
> > 2.3. One URL per line, optionally prefixed with "-" or "+"
> > 2.4. For stack repos/overlays:
> > 2.4.1. No prefix: replace all prior mirrors from masters with new URLS in this file.
> > 2.4.2. "-" prefix: remove this URL from the list from masters.
> > 2.4.2. "+" prefix: append this URL to the list from masters.
> 
> So if *any* line doesn't have a prefix, then *everything* gets
> overwritten? What about the prior mirrors listed in the file?
> 
> There needs to be some mechanism for specifying this, but I don't think
> this is it.
> 
> Perhaps a header with a special line?
Suggestions on the format then?

Maybe we have two modes:
1. No prefix, always append
2. Prefix required, one of '+', '-', ' '.

I was also asked by a user to make it possible to adjust the priority of
some mirror URLs, instead of only random choice.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Jeroen Roovers]

On Tue, 7 Jan 2014 21:12:59 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> I was also asked by a user to make it possible to adjust the priority
> of some mirror URLs, instead of only random choice.

While we are at it, we could add keywords for (global) regions, so
that I can set portage to look for a European mirror and portage will
avoid contacting a distant mirror in Asia or America.

It's probably worth it in the long run to do a little more here than
have users simply express priorities for specific mirrors.

We could probably set up the path structure like this:

profiles/thirdpartymirrors/gnu/Europe

and add a blacklist/whitelist/priority structure
in /etc/portage/profiles/thirdpartymirrors, maybe?

Portage might even use the local system's timezone to determine what
mirror set to use.


     jer

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Alex Xu]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

On 08/01/14 10:11 AM, Jeroen Roovers wrote:
> On Tue, 7 Jan 2014 21:12:59 +0000
> "Robin H. Johnson" <robbat2@gentoo.org> wrote:
> 
>> I was also asked by a user to make it possible to adjust the priority
>> of some mirror URLs, instead of only random choice.
> 
> While we are at it, we could add keywords for (global) regions, so
> that I can set portage to look for a European mirror and portage will
> avoid contacting a distant mirror in Asia or America.
> 
> It's probably worth it in the long run to do a little more here than
> have users simply express priorities for specific mirrors.
> 
> We could probably set up the path structure like this:
> 
> profiles/thirdpartymirrors/gnu/Europe
> 
> and add a blacklist/whitelist/priority structure
> in /etc/portage/profiles/thirdpartymirrors, maybe?
> 
> Portage might even use the local system's timezone to determine what
> mirror set to use.
> 

Eww. Geographically-close files should be made available through
GENTOO_MIRRORS and the regular distfiles system.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Jeroen Roovers]

On Wed, 08 Jan 2014 13:37:13 -0500
Alex Xu <alex_y_xu@yahoo.ca> wrote:

> Eww. Geographically-close files should be made available through
> GENTOO_MIRRORS and the regular distfiles system.

How do you define GENTOO_MIRRORS? Where did RESTRICT=mirror go? Oh
right. What if those are not available? What I am proposing is /more/
flexibility, but you think when GENTOO_MIRRORS fails, I should still
collect files from just any site half a world away? How does that help?
Please contribute.


     jer

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On Wed, Jan 8, 2014 at 6:14 PM, Jeroen Roovers <jer@gentoo.org> wrote:

> On Wed, 08 Jan 2014 13:37:13 -0500
> Alex Xu <alex_y_xu@yahoo.ca> wrote:
>
> > Eww. Geographically-close files should be made available through
> > GENTOO_MIRRORS and the regular distfiles system.
>
> How do you define GENTOO_MIRRORS? Where did RESTRICT=mirror go? Oh
> right. What if those are not available? What I am proposing is /more/
> flexibility, but you think when GENTOO_MIRRORS fails, I should still
> collect files from just any site half a world away? How does that help?
> Please contribute.
>

Except the original proposal is fairly simple (+ / - to add remove mirrors)
and doesn't deal with prioritizing nor ordering them....so how would that
work? Labelling each mirror I presume?

-A


>
>
>      jer
>

[-- Attachment #2: Type: text/html, Size: 1389 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Ben Kohler]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

On Wed, Jan 8, 2014 at 12:37 PM, Alex Xu <alex_y_xu@yahoo.ca> wrote:
>
>
> Eww. Geographically-close files should be made available through
> GENTOO_MIRRORS and the regular distfiles system.
>

I think you may be missing the point of this proposal, or are unaware of
how profiles/thirdpartymirrors and SRC_URI="mirror://.." work.

Readability and maintanence issues aside (which themselves are huge), the
current setup with a list of literally hundreds of geographically-random
mirrors for one source, is quite often doing a disservice to our users.
 Most of the very large upstream mirror list sources are already sorted
geographically, it would be great to take advantage of this.

And to me, it seems like the proposed thirdpartymirrors/mirrorname/<Region>
setup seems quite elegant.  I'm sure it would be optional and mirror lists
that can't take advantage of this would just use a flat file at
thirdpartymirrors/mirrorname.

-Ben

[-- Attachment #2: Type: text/html, Size: 1406 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

On 1/6/14, 9:20 PM, Robin H. Johnson wrote:
> This is a small feature request, but it will require a modification to
> PMS, so I describe it here.
> 
> The present thirdpartymirrors file is unwieldy, and difficult to manage
> due to it's format with very long lines. It also doesn't permit easy
> comments. Presently commits to it look very ugly, because diffs are
> line-based, and we pack a lot into each line.
> 
> I would like to make it a directory instead of a single file, and extend
> the internal syntax.

+1 for the general direction.

I think for the final PMS change it would need to be made even more
precise (some thoughts have been expressed in the thread).

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Ben Kohler]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

On Mon, Jan 6, 2014 at 2:20 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:

> This is a small feature request, but it will require a modification to
> PMS, so I describe it here.
>
> The present thirdpartymirrors file is unwieldy, and difficult to manage
> due to it's format with very long lines. It also doesn't permit easy
> comments. Presently commits to it look very ugly, because diffs are
> line-based, and we pack a lot into each line.
>
> I would like to make it a directory instead of a single file, and extend
> the internal syntax.
>
> I am very excited about this whole idea, this thirdpartymirrors setup
badly needs some reworking.   To me it makes the most sense to turn
thirdpartymirrors into a dir, with a file structure like:

thirdpartymirrors/mirror1
thirdpartymirrors/mirror2
thirdpartymirrors/mirror3/
thirdpartymirrors/mirror3/Asia
thirdpartymirrors/mirror3/Europe
thirdpartymirrors/mirror3/Etc
thirdpartymirrors/mirror4

I'm not sure I see much real value in allowing individual profiles to
add/remove mirrors from each group, to be honest.  Maybe I'm just not
thinking of the right scenarios.

But I really do believe just the basic changes like splitting the file so
each group gets its own file, one server per line, with comments, etc...
will be a huge help in using and maintaining these groups.

-Ben

[-- Attachment #2: Type: text/html, Size: 2121 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

Dnia 2014-01-06, o godz. 20:20:03
"Robin H. Johnson" <robbat2@gentoo.org> napisał(a):

> 2.4. For stack repos/overlays:
> 2.4.1. No prefix: replace all prior mirrors from masters with new URLS in this file.
> 2.4.2. "-" prefix: remove this URL from the list from masters.
> 2.4.2. "+" prefix: append this URL to the list from masters.

Just to be clear, what is the exact use case for this? I can't think
of a really good reason to manipulate mirror lists in subsequent repos.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

On Thu, Jan 9, 2014 at 8:58 PM, Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 2014-01-06, o godz. 20:20:03
> "Robin H. Johnson" <robbat2@gentoo.org> napisał(a):
>
> > 2.4. For stack repos/overlays:
> > 2.4.1. No prefix: replace all prior mirrors from masters with new URLS
> in this file.
> > 2.4.2. "-" prefix: remove this URL from the list from masters.
> > 2.4.2. "+" prefix: append this URL to the list from masters.
>
> Just to be clear, what is the exact use case for this? I can't think
> of a really good reason to manipulate mirror lists in subsequent repos.
>
>
I can mostly think of bad reasons, where instead of manipulating the list,
the user should do something else (like use a local mirror.)

-A


> --
> Best regards,
> Michał Górny
>

[-- Attachment #2: Type: text/html, Size: 1495 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 2017 bytes --]

On Thu, Jan 9, 2014 at 9:26 AM, Ben Kohler <bkohler@gmail.com> wrote:

>
> On Mon, Jan 6, 2014 at 2:20 PM, Robin H. Johnson <robbat2@gentoo.org>wrote:
>
>> I would like to make it a directory instead of a single file, and extend
>> the internal syntax.
>>
> I'm not sure I see much real value in allowing individual profiles to
> add/remove mirrors from each group, to be honest.  Maybe I'm just not
> thinking of the right scenarios.
>

Ignorant question: Can overlays modify profile easily (if at all?) If not,
then taking mirror lists out of profile seems rather sane, and would allow
overlays to add their own mirrors.

Spec thought:

1. make.conf should define MIRROR_REGIONS, which is a space separated list
of Region/Subregion/Locality strings (e.g.
"north_america/united_states/texas europe/germany/düsseldorf").
2. Using the file layout as bkohler suggested (e.g.
mirrors/{mirrorname,othermirrorname,thirdmirrorname}/{Asia,Europe}:
mirrorname may be a file or a directory. If it is a directory, it may
contain a file named "mirrorlist" if there are global mirrors, and it may
contain any number of region-named files or directories.
3. If a region-named "file" is a directory, then it must contain either a
file mirrorlist if it has region-level mirrors, or files/directories for
subregions.
4. When emerge fetches a file, if the most specific region (of the first
entry, if there are multiple entries) is not found, move up to the next
greater-sized region (one directory up). If that region exists, then
concatenate all of its subregions into the list to select a URL from. If it
doesn't exist, move up another level, and again, if it is found,
concatenate everything below it into the list of possible URLs. If the
final top-level region isn't found, move to the next MIRROR_REGIONS entry,
if it exists. Otherwise, concatenate every available region for that mirror
into the list.

This could all be done in a flat XML (or YAML, to avoid boilerplate) file,
too.

[-- Attachment #2: Type: text/html, Size: 2896 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

On 10 January 2014 17:58, Michał Górny <mgorny@gentoo.org> wrote:

>
> Just to be clear, what is the exact use case for this? I can't think
> of a really good reason to manipulate mirror lists in subsequent repos.
>

For Perl, in ::gentoo , its considered not too optimal to have backpan
listed as a mirror, and its unnesscary, as ::gentoo gets to mirror
everything on cpan anyway, so that there's no problem when upstream delete
the version gentoo is using from cpan.

But in the overlay, we don't have the luxury of a mirror, so we have
backpan listed as a mirror for when cpan deletes a release.

diff <( grep cpan
/var/paludis/repositories/perl-git/profiles/thirdpartymirrors  ) <( grep
cpan /usr/portage/profiles/thirdpartymirrors  )
1c1
< cpan          http://search.cpan.org/CPAN http://cpan.hexten.net/
http://cpan.cpantesters.org http://www.cpan.org http://backpan.perl.org
---
> cpan          http://search.cpan.org/CPAN http://www.cpan.org

So here, it would be nicer if we could simply specify new mirrors instead
of having to override the whole thing.

-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2014 bytes --]

Re: [gentoo-dev] Portage Feature Request: making thirdpartymirrors easier to manage

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

On 12 January 2014 08:16, Kent Fredric <kentfredric@gmail.com> wrote:

> diff <( grep cpan
> /var/paludis/repositories/perl-git/profiles/thirdpartymirrors  ) <( grep
> cpan /usr/portage/profiles/thirdpartymirrors  )
>


Oh... also, there's a wide variety of location specific CPAN mirrors
available that are location-specific.

Specifying them in GENTOO_MIRRORS will simply mess up portage as they'll
tell portage to try them for gentoo packages too, which will mean a lot of
waiting while gentoo erroneously tries to fetch files from servers that
will never have them.

So having a location-specific subkey for each class of mirrors, and having
a priority within each class of mirrors, so that users can just specify
their location and the fetch system uses an appropriate mirror for each
mirror class, that'd be great.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1345 bytes --]