[gentoo-dev] [PATCH] profiles/thirdpartymirrors: add final fallback for apache

[gentoo-dev] [PATCH] profiles/thirdpartymirrors: add final fallback for apache

[Eli Schwartz]

Apache has a mirror network, which only covers the most recent release
of any given package. They also have an additional site which does not
appear to be a CDN, and is throttled and can maybe ban you if you use it
too much. Unfortunately, it is also the ONLY way to actually get
historic releases of many packages.

Use it, and use it last -- after every other mirror has been tried,
which should handle latest releases. In combination with GENTOO_MIRRORS
this should ensure that users can actually download software when
needed, without running afoul of throttling.

Signed-off-by: Eli Schwartz <eschwartz@gentoo.org>
---
 profiles/thirdpartymirrors | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/profiles/thirdpartymirrors b/profiles/thirdpartymirrors
index bb5398896976..d3a857327495 100644
--- a/profiles/thirdpartymirrors
+++ b/profiles/thirdpartymirrors
@@ -1,4 +1,4 @@
-apache		https://dlcdn.apache.org/ https://apache.mirror.iphh.net/ https://artfiles.org/apache.org/ https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/ https://ftp.fau.de/apache/ https://apache.osuosl.org/
+apache		https://dlcdn.apache.org/ https://apache.mirror.iphh.net/ https://artfiles.org/apache.org/ https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/ https://ftp.fau.de/apache/ https://apache.osuosl.org/ https://archive.apache.org/dist/
 cpan		https://cpan.metacpan.org https://www.cpan.org
 debian		https://deb.debian.org/debian/ http://ftp.au.debian.org/debian/ http://ftp.at.debian.org/debian/ http://ftp.by.debian.org/debian/ http://ftp.be.debian.org/debian/ http://ftp.br.debian.org/debian/ http://ftp.bg.debian.org/debian/ http://ftp.ca.debian.org/debian/ http://ftp2.cn.debian.org/debian/ http://ftp.cn.debian.org/debian/ http://ftp.hr.debian.org/debian/ http://ftp.cz.debian.org/debian/ http://ftp.dk.debian.org/debian/ http://ftp.sv.debian.org/debian/ http://ftp.ee.debian.org/debian/ http://ftp.fi.debian.org/debian/ http://ftp.fr.debian.org/debian/ http://ftp2.de.debian.org/debian/ http://ftp.de.debian.org/debian/ http://ftp.gr.debian.org/debian/ http://ftp.hu.debian.org/debian/ http://ftp.is.debian.org/debian/ http://ftp.ie.debian.org/debian/ http://ftp.it.debian.org/debian/ http://ftp.jp.debian.org/debian/ http://ftp.lt.debian.org/debian/ http://ftp.mx.debian.org/debian/ http://ftp.md.debian.org/debian/ http://ftp.nl.debian.org/debian/ http://ftp.nc.debian.org/debian/ http://ftp.nz.debian.org/debian/ http://ftp.no.debian.org/debian/ http://ftp.pl.debian.org/debian/ http://ftp.pt.debian.org/debian/ http://ftp.ro.debian.org/debian/ http://ftp.ru.debian.org/debian/ http://ftp.sg.debian.org/debian/ http://ftp.sk.debian.org/debian/ http://ftp.si.debian.org/debian/ http://ftp.es.debian.org/debian/ http://ftp.se.debian.org/debian/ http://ftp.ch.debian.org/debian/ http://ftp.tw.debian.org/debian/ http://ftp.tr.debian.org/debian/ http://ftp.ua.debian.org/debian/ http://ftp.uk.debian.org/debian/ http://ftp.us.debian.org/debian/
 gentoo		https://distfiles.gentoo.org/distfiles https://gentoo.osuosl.org/distfiles https://ftp.halifax.rwth-aachen.de/gentoo/distfiles https://ftp.fau.de/gentoo/distfiles
-- 
2.45.2

Re: [gentoo-dev] [PATCH] profiles/thirdpartymirrors: add final fallback for apache

[Sam James]

Eli Schwartz <eschwartz@gentoo.org> writes:

> Apache has a mirror network, which only covers the most recent release
> of any given package. They also have an additional site which does not
> appear to be a CDN, and is throttled and can maybe ban you if you use it
> too much. Unfortunately, it is also the ONLY way to actually get
> historic releases of many packages.
>
> Use it, and use it last -- after every other mirror has been tried,
> which should handle latest releases. In combination with GENTOO_MIRRORS
> this should ensure that users can actually download software when
> needed, without running afoul of throttling.
>
> Signed-off-by: Eli Schwartz <eschwartz@gentoo.org>
> ---

LGTM.

>  profiles/thirdpartymirrors | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/profiles/thirdpartymirrors b/profiles/thirdpartymirrors
> index bb5398896976..d3a857327495 100644
> --- a/profiles/thirdpartymirrors
> +++ b/profiles/thirdpartymirrors
> @@ -1,4 +1,4 @@
> -apache https://dlcdn.apache.org/ https://apache.mirror.iphh.net/
> https://artfiles.org/apache.org/
> https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/
> https://ftp.fau.de/apache/ https://apache.osuosl.org/
> +apache https://dlcdn.apache.org/ https://apache.mirror.iphh.net/
> https://artfiles.org/apache.org/
> https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/
> https://ftp.fau.de/apache/ https://apache.osuosl.org/
> https://archive.apache.org/dist/
>  cpan		https://cpan.metacpan.org https://www.cpan.org
>  debian https://deb.debian.org/debian/
> http://ftp.au.debian.org/debian/ http://ftp.at.debian.org/debian/
> http://ftp.by.debian.org/debian/ http://ftp.be.debian.org/debian/
> http://ftp.br.debian.org/debian/ http://ftp.bg.debian.org/debian/
> http://ftp.ca.debian.org/debian/ http://ftp2.cn.debian.org/debian/
> http://ftp.cn.debian.org/debian/ http://ftp.hr.debian.org/debian/
> http://ftp.cz.debian.org/debian/ http://ftp.dk.debian.org/debian/
> http://ftp.sv.debian.org/debian/ http://ftp.ee.debian.org/debian/
> http://ftp.fi.debian.org/debian/ http://ftp.fr.debian.org/debian/
> http://ftp2.de.debian.org/debian/ http://ftp.de.debian.org/debian/
> http://ftp.gr.debian.org/debian/ http://ftp.hu.debian.org/debian/
> http://ftp.is.debian.org/debian/ http://ftp.ie.debian.org/debian/
> http://ftp.it.debian.org/debian/ http://ftp.jp.debian.org/debian/
> http://ftp.lt.debian.org/debian/ http://ftp.mx.debian.org/debian/
> http://ftp.md.debian.org/debian/ http://ftp.nl.debian.org/debian/
> http://ftp.nc.debian.org/debian/ http://ftp.nz.debian.org/debian/
> http://ftp.no.debian.org/debian/ http://ftp.pl.debian.org/debian/
> http://ftp.pt.debian.org/debian/ http://ftp.ro.debian.org/debian/
> http://ftp.ru.debian.org/debian/ http://ftp.sg.debian.org/debian/
> http://ftp.sk.debian.org/debian/ http://ftp.si.debian.org/debian/
> http://ftp.es.debian.org/debian/ http://ftp.se.debian.org/debian/
> http://ftp.ch.debian.org/debian/ http://ftp.tw.debian.org/debian/
> http://ftp.tr.debian.org/debian/ http://ftp.ua.debian.org/debian/
> http://ftp.uk.debian.org/debian/ http://ftp.us.debian.org/debian/
>  gentoo		https://distfiles.gentoo.org/distfiles https://gentoo.osuosl.org/distfiles https://ftp.halifax.rwth-aachen.de/gentoo/distfiles https://ftp.fau.de/gentoo/distfiles

Re: [gentoo-dev] [PATCH] profiles/thirdpartymirrors: add final fallback for apache

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

On Tue, Oct 22, 2024 at 11:24:29PM -0400, Eli Schwartz wrote:
> Apache has a mirror network, which only covers the most recent release
> of any given package. They also have an additional site which does not
> appear to be a CDN, and is throttled and can maybe ban you if you use it
> too much. Unfortunately, it is also the ONLY way to actually get
> historic releases of many packages.
> 
> Use it, and use it last -- after every other mirror has been tried,
> which should handle latest releases. In combination with GENTOO_MIRRORS
> this should ensure that users can actually download software when
> needed, without running afoul of throttling.
This does not actually do it "last" as you claim.

Portage shuffles the list of thirdpartymirrors:
https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/package/ebuild/fetch.py#n1140

While it increases mirror burden; this should likely be done as a
distinct thirdpartymirror:
apache-historical https://archive.apache.org/dist/

And that gets used in ebuilds when distfiles fall off the main mirrors
[until such time as strictly ordered behavior is available].

In likelihood, unless the ebuild has RESTRICT=mirror, I'd hope that that
the primary request is made to Gentoo's mirrors before the upstreams: so
that we are better open-source participants.

Users who are also trying to use old ebuilds, where the distfile is no
longer on the Gentoo mirrors should also consider thirdpartymirrors in
their profiles, in which case inserting this entry locally would be
appropriate.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] [PATCH] profiles/thirdpartymirrors: add final fallback for apache

[Mike Gilbert]

On Sat, Oct 26, 2024 at 1:03 AM Robin H. Johnson <robbat2@gentoo.org> wrote:
>
> On Tue, Oct 22, 2024 at 11:24:29PM -0400, Eli Schwartz wrote:
> > Apache has a mirror network, which only covers the most recent release
> > of any given package. They also have an additional site which does not
> > appear to be a CDN, and is throttled and can maybe ban you if you use it
> > too much. Unfortunately, it is also the ONLY way to actually get
> > historic releases of many packages.
> >
> > Use it, and use it last -- after every other mirror has been tried,
> > which should handle latest releases. In combination with GENTOO_MIRRORS
> > this should ensure that users can actually download software when
> > needed, without running afoul of throttling.
> This does not actually do it "last" as you claim.
>
> Portage shuffles the list of thirdpartymirrors:
> https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/package/ebuild/fetch.py#n1140
>
> While it increases mirror burden; this should likely be done as a
> distinct thirdpartymirror:
> apache-historical https://archive.apache.org/dist/
>
> And that gets used in ebuilds when distfiles fall off the main mirrors
> [until such time as strictly ordered behavior is available].

I assume Eli was trying to avoid having to update old ebuilds.

I don't really see any point in adding an apache-historical entry in
thirdpartymirrors; just inline https://archive.apache.org/dist/ into
SRC_URI if the ebuild requires it.

Re: [gentoo-dev] [PATCH] profiles/thirdpartymirrors: add final fallback for apache

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 2897 bytes --]

On 26/10/2024 17.25, Mike Gilbert wrote:
> On Sat, Oct 26, 2024 at 1:03 AM Robin H. Johnson <robbat2@gentoo.org> wrote:
>> On Tue, Oct 22, 2024 at 11:24:29PM -0400, Eli Schwartz wrote:
>>> Apache has a mirror network, which only covers the most recent release
>>> of any given package. They also have an additional site which does not
>>> appear to be a CDN, and is throttled and can maybe ban you if you use it
>>> too much. Unfortunately, it is also the ONLY way to actually get
>>> historic releases of many packages.
>>>
>>> Use it, and use it last -- after every other mirror has been tried,
>>> which should handle latest releases. In combination with GENTOO_MIRRORS
>>> this should ensure that users can actually download software when
>>> needed, without running afoul of throttling.
>> This does not actually do it "last" as you claim.
>>
>> Portage shuffles the list of thirdpartymirrors:
>> https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/package/ebuild/fetch.py#n1140
>>
>> While it increases mirror burden; this should likely be done as a
>> distinct thirdpartymirror:
>> apache-historical https://archive.apache.org/dist/
>>
>> And that gets used in ebuilds when distfiles fall off the main mirrors
>> [until such time as strictly ordered behavior is available].
> 
> I assume Eli was trying to avoid having to update old ebuilds.
> 
> I don't really see any point in adding an apache-historical entry in
> thirdpartymirrors;

Such an entry would probably violate the "no single URL entries in 
mirros" rule anyway.

> just inline https://archive.apache.org/dist/ into
> SRC_URI if the ebuild requires it.

I assume the problem is that the ebuild may not require it when being 
written, but at some later point in the ebuild's future.

Tex packaging is in a similar situation.


I wonder if we could introduce priority groups into the mirror entry 
specification. As an example

foo https://mirror.foo.org | https://ftp.baz.de/foo … | 
htttps://archive.mirror.foo.org

The priority groups are separated by '|'.

Here, the first priority is the a DNS bouncer, that delegates to your 
nearest mirror. This is commonly found for some archives.

The second priority groups contains a selected set of real mirrors. 
Having it is helpful in cases where the DNS bouncer is malfunctioning. 
Those events are rare but I believe there have been some reports.

The last priority group consists of the URL to the long-term archive, 
which consists of all artifacts ever released but with rate limiting.

The packages manager would first select one or more entries from the 
first priority group, before proceeding to the next group(s).

I believe Apache and Tex packages would benefit from this.

Could this be something for EAPI 9? (I think it needs a new EAPI, but 
happy stand corrected.)

- Flow






[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 21567 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] profiles/thirdpartymirrors: add final fallback for apache

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 3477 bytes --]

On 10/26/24 1:03 AM, Robin H. Johnson wrote:
> This does not actually do it "last" as you claim.
> 
> Portage shuffles the list of thirdpartymirrors:
> https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/package/ebuild/fetch.py#n1140
> 
> While it increases mirror burden; this should likely be done as a
> distinct thirdpartymirror:
> apache-historical https://archive.apache.org/dist/


Thanks for pointing this out. It's disappointing but my opinion is that
this changes nothing other than the commit message I should use when
committing it.


> And that gets used in ebuilds when distfiles fall off the main mirrors
> [until such time as strictly ordered behavior is available].


This is an unreasonable and irrational burden on ebuild maintainers, and
regardless of your choice of remediation, if it isn't automatically and
transparently handling it via the thirdpartymirrors handling itself,
nobody wins. Not ebuild maintainers (who have to both know the quirks of
apache and watch more closely than usual for updates), not bug triagers
(who run the risk of getting trivially solvable issues to sort through),
and not users (who run a supported configuration and end up with
availability issues).

It's a bug (albeit low priority) if SRC_URI doesn't point to a valid
download location -- why should the fact that thirdpartymirrors has no
priority selection change that fact?

The suggestion downthread about extending this somehow in EAPI 9 is
clever -- we should do it, but we also shouldn't leave the current state
of affairs broken until some unspecified time years in the future.


> In likelihood, unless the ebuild has RESTRICT=mirror, I'd hope that that
> the primary request is made to Gentoo's mirrors before the upstreams: so
> that we are better open-source participants.


Yes. This is why I think any change to thirdpartymirrors is low risk.


> Users who are also trying to use old ebuilds, where the distfile is no
> longer on the Gentoo mirrors should also consider thirdpartymirrors in
> their profiles, in which case inserting this entry locally would be
> appropriate.


Unacceptable.

This isn't about old ebuilds. It's about the very latest ebuilds.
Distfiles go missing from the upstream CDN and are only available via
long-term archival, the same minute (AFAIK, based on their
documentation) that a new patchlevel release is tagged upstream. It is
physically impossible for Gentoo to be that fast in packaging new
versions, and it doesn't matter anyway because it won't help for another
month due to stabilization. And that is assuming it gets packaged as
fast as the maintainer can package it -- forgive me for being skeptical
about that one, we are all human after all. :)

It is reasonable, expected, normal use of computers, and reasonable,
expected, normal use of Gentoo, to download and install "old" versions
of a program minutes or hours after another patch release is out. We
must not penalize those users by requiring them to manually implement
this patch at global scope, running the risk that they will get it wrong
and list only the slow site which they then overload for *all* traffic
rather than at least having shuffle().

For indeed, overriding a mirror in /etc/portage/mirrors is just going to
force that to be used first for ALL packages using mirror://apache/ at
all. At least it respects your listed ordering I guess...


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]