[gentoo-dev] The uncertain future of repository mirrors

[gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1810 bytes --]

Hello, everyone.

TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.


Over 10 years ago, I've started the repository mirror & CI project. 
What started as a bunch of shell scripts on a user-donated server, has
organically grown into a bigger bunch of shell scripts managed by Infra.
Nevertheless, it's still a bunch of hacks glued together.

Things don't work well all the time.  Sometimes stuff randomly crashes,
and I have to SSH and remove local checkouts to make it work.  Sometimes
the git repositories used to transfer logs grow so big they kill infra.
Often some repository starts crashing this or another part and needs to
be disabled.

To be honest, I have no energy to keep maintaining this.  I'm really
tired of having to deal with stuff crashing and spamming my mailbox with
failure mails.  I'm tired of having to go through all the infra hoops
just to disable another repository that can't work for one reason or
another.  In fact, I'm even tired that whenever people add new
repositories to api.gentoo.org, I have to go through that idiotic GitHub
clickety-click UI to stop receiving notifications for everything that
happens in these repositories.

So what I'm thinking about is winding most of the project down.  We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization.  What I'd like to leave is mirroring of gentoo
and guru repositories, since these two we have control of, and are very
important to Gentoo users.

So, well, unless someone convinces me otherwise, I'm going to disable
all other repositories over the next weekend, and remove their mirrors.
Gentoo and GURU will still be mirrored, and CI will keep running
as usual.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[orbea]

On Fri, 21 Mar 2025 14:32:31 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Hello, everyone.
> 
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.
> 
> 
> Over 10 years ago, I've started the repository mirror & CI project. 
> What started as a bunch of shell scripts on a user-donated server, has
> organically grown into a bigger bunch of shell scripts managed by
> Infra. Nevertheless, it's still a bunch of hacks glued together.
> 
> Things don't work well all the time.  Sometimes stuff randomly
> crashes, and I have to SSH and remove local checkouts to make it
> work.  Sometimes the git repositories used to transfer logs grow so
> big they kill infra. Often some repository starts crashing this or
> another part and needs to be disabled.
> 
> To be honest, I have no energy to keep maintaining this.  I'm really
> tired of having to deal with stuff crashing and spamming my mailbox
> with failure mails.  I'm tired of having to go through all the infra
> hoops just to disable another repository that can't work for one
> reason or another.  In fact, I'm even tired that whenever people add
> new repositories to api.gentoo.org, I have to go through that idiotic
> GitHub clickety-click UI to stop receiving notifications for
> everything that happens in these repositories.
> 
> So what I'm thinking about is winding most of the project down.  We'd
> stop mirroring third-party repositories, and remove most of gentoo-
> mirror organization.  What I'd like to leave is mirroring of gentoo
> and guru repositories, since these two we have control of, and are
> very important to Gentoo users.
> 
> So, well, unless someone convinces me otherwise, I'm going to disable
> all other repositories over the next weekend, and remove their
> mirrors. Gentoo and GURU will still be mirrored, and CI will keep
> running as usual.
> 

What does this mean for the libressl overlay? People use that so please
don't remove it.

Re: [gentoo-dev] The uncertain future of repository mirrors

[Alexey Sokolov]

21.03.2025 13:32, Michał Górny пишет:
> Hello, everyone.
>
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.
>
>
> Over 10 years ago, I've started the repository mirror & CI project.
> What started as a bunch of shell scripts on a user-donated server, has
> organically grown into a bigger bunch of shell scripts managed by Infra.
> Nevertheless, it's still a bunch of hacks glued together.
>
> Things don't work well all the time.  Sometimes stuff randomly crashes,
> and I have to SSH and remove local checkouts to make it work.  Sometimes
> the git repositories used to transfer logs grow so big they kill infra.
> Often some repository starts crashing this or another part and needs to
> be disabled.
>
> To be honest, I have no energy to keep maintaining this.  I'm really
> tired of having to deal with stuff crashing and spamming my mailbox with
> failure mails.  I'm tired of having to go through all the infra hoops
> just to disable another repository that can't work for one reason or
> another.  In fact, I'm even tired that whenever people add new
> repositories to api.gentoo.org, I have to go through that idiotic GitHub
> clickety-click UI to stop receiving notifications for everything that
> happens in these repositories.
>
> So what I'm thinking about is winding most of the project down.  We'd
> stop mirroring third-party repositories, and remove most of gentoo-
> mirror organization.  What I'd like to leave is mirroring of gentoo
> and guru repositories, since these two we have control of, and are very
> important to Gentoo users.
>
> So, well, unless someone convinces me otherwise, I'm going to disable
> all other repositories over the next weekend, and remove their mirrors.
> Gentoo and GURU will still be mirrored, and CI will keep running
> as usual.
>
Hi, if you just disable it, it'll break setup of everyone who uses 
/etc/portage/repos.conf/eselect-repo.conf, right?

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

On Fri, 2025-03-21 at 07:12 -0700, orbea wrote:
> What does this mean for the libressl overlay? People use that so please
> don't remove it.

It means people will have to sync straight from the upstream repository.
Also, we won't be reporting bugs when things break hard.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 387 bytes --]

On Fri, 2025-03-21 at 14:12 +0000, Alexey Sokolov wrote:
> Hi, if you just disable it, it'll break setup of everyone who uses 
> /etc/portage/repos.conf/eselect-repo.conf, right?

Sounds about right.  Which is probably preferable over the current state
of mirrors silently stopping to update, and people being stuck forever
on old commits.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Alexey Sokolov]

21.03.2025 14:44, Michał Górny пишет:
> On Fri, 2025-03-21 at 14:12 +0000, Alexey Sokolov wrote:
>> Hi, if you just disable it, it'll break setup of everyone who uses
>> /etc/portage/repos.conf/eselect-repo.conf, right?
> Sounds about right.  Which is probably preferable over the current state
> of mirrors silently stopping to update, and people being stuck forever
> on old commits.
>
In that case I think first need to change `eselect repository enable` to 
add overlays directly instead of its mirror. And somehow update existing 
configs, either semi-automatically, or e.g. via news item

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 756 bytes --]

On Fri, 2025-03-21 at 14:47 +0000, Alexey Sokolov wrote:
> 21.03.2025 14:44, Michał Górny пишет:
> > On Fri, 2025-03-21 at 14:12 +0000, Alexey Sokolov wrote:
> > > Hi, if you just disable it, it'll break setup of everyone who uses
> > > /etc/portage/repos.conf/eselect-repo.conf, right?
> > Sounds about right.  Which is probably preferable over the current state
> > of mirrors silently stopping to update, and people being stuck forever
> > on old commits.
> > 
> In that case I think first need to change `eselect repository enable` to 
> add overlays directly instead of its mirror. And somehow update existing 
> configs, either semi-automatically, or e.g. via news item
> 

Feel free to.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[orbea]

On Fri, 21 Mar 2025 15:44:04 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Fri, 2025-03-21 at 07:12 -0700, orbea wrote:
> > What does this mean for the libressl overlay? People use that so
> > please don't remove it.  
> 
> It means people will have to sync straight from the upstream
> repository. Also, we won't be reporting bugs when things break hard.
> 

Currently when I push changes to the overlay I push to:

git+ssh://git@git.gentoo.org/repo/proj/libressl.git

Is there anything I need to do to make sure I can still push to the
github repo instead? I am not sure I have access to push there directly?

Re: [gentoo-dev] The uncertain future of repository mirrors

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 914 bytes --]

On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
> Hello, everyone.
> 
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.

Unfortunate, but just to say that I have nothing to say against
dropping these if it's a maintenance burden.

> So what I'm thinking about is winding most of the project down.  We'd
> stop mirroring third-party repositories, and remove most of gentoo-
> mirror organization.

Had one concern but complete removal (rather than keeping mirrors
archived) would mostly handle it. Aka so that users can't keep
sync'ing with these dead mirrors by mistake (often caused issues when
we removed overlays but mirror stayed behind, users wouldn't notice a
thing until lacking updates started to break things).

As been mentioned in another post, maybe will need a news item to
explain this still though.
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

On Fri, 2025-03-21 at 16:50 -0700, orbea wrote:
> On Fri, 21 Mar 2025 15:44:04 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > On Fri, 2025-03-21 at 07:12 -0700, orbea wrote:
> > > What does this mean for the libressl overlay? People use that so
> > > please don't remove it.  
> > 
> > It means people will have to sync straight from the upstream
> > repository. Also, we won't be reporting bugs when things break hard.
> > 
> 
> Currently when I push changes to the overlay I push to:
> 
> git+ssh://git@git.gentoo.org/repo/proj/libressl.git
> 
> Is there anything I need to do to make sure I can still push to the
> github repo instead? I am not sure I have access to push there directly?

If you're asking about the mirror @ https://github.com/gentoo/libressl,
then that's not affected.  I'm talking of
https://github.com/gentoo-mirror/libressl.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[orbea]

On Sat, 22 Mar 2025 08:01:58 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Fri, 2025-03-21 at 16:50 -0700, orbea wrote:
> > On Fri, 21 Mar 2025 15:44:04 +0100
> > Michał Górny <mgorny@gentoo.org> wrote:
> >   
> > > On Fri, 2025-03-21 at 07:12 -0700, orbea wrote:  
> > > > What does this mean for the libressl overlay? People use that so
> > > > please don't remove it.    
> > > 
> > > It means people will have to sync straight from the upstream
> > > repository. Also, we won't be reporting bugs when things break
> > > hard. 
> > 
> > Currently when I push changes to the overlay I push to:
> > 
> > git+ssh://git@git.gentoo.org/repo/proj/libressl.git
> > 
> > Is there anything I need to do to make sure I can still push to the
> > github repo instead? I am not sure I have access to push there
> > directly?  
> 
> If you're asking about the mirror @
> https://github.com/gentoo/libressl, then that's not affected.  I'm
> talking of https://github.com/gentoo-mirror/libressl.
> 

Okay, thank you for clarifying.

Re: [gentoo-dev] The uncertain future of repository mirrors

[Jay Faulkner]

On 3/21/2025 6:42 PM, Ionen Wolkens wrote:
> On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
>> Hello, everyone.
>>
>> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
>> except for gentoo and guru.
> Unfortunate, but just to say that I have nothing to say against
> dropping these if it's a maintenance burden.
>
>> So what I'm thinking about is winding most of the project down.  We'd
>> stop mirroring third-party repositories, and remove most of gentoo-
>> mirror organization.
> Had one concern but complete removal (rather than keeping mirrors
> archived) would mostly handle it. Aka so that users can't keep
> sync'ing with these dead mirrors by mistake (often caused issues when
> we removed overlays but mirror stayed behind, users wouldn't notice a
> thing until lacking updates started to break things).


In the OpenStack community, we will put a final commit on HEAD of the 
primary branch removing all the content and putting an EOL notice in the 
readme (e.g. https://opendev.org/openstack/ironic-lib ). This may be a 
good solution in this case as well, potentially?

-Jay

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1497 bytes --]

On Sat, 2025-03-22 at 08:20 -0700, Jay Faulkner wrote:
> On 3/21/2025 6:42 PM, Ionen Wolkens wrote:
> > On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
> > > Hello, everyone.
> > > 
> > > TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> > > except for gentoo and guru.
> > Unfortunate, but just to say that I have nothing to say against
> > dropping these if it's a maintenance burden.
> > 
> > > So what I'm thinking about is winding most of the project down.  We'd
> > > stop mirroring third-party repositories, and remove most of gentoo-
> > > mirror organization.
> > Had one concern but complete removal (rather than keeping mirrors
> > archived) would mostly handle it. Aka so that users can't keep
> > sync'ing with these dead mirrors by mistake (often caused issues when
> > we removed overlays but mirror stayed behind, users wouldn't notice a
> > thing until lacking updates started to break things).
> 
> 
> In the OpenStack community, we will put a final commit on HEAD of the 
> primary branch removing all the content and putting an EOL notice in the 
> readme (e.g. https://opendev.org/openstack/ironic-lib ). This may be a 
> good solution in this case as well, potentially?
> 

Not sure.  It would mean that `emerge --sync` will replace all ebuilds
with a README file that user might not even notice.  I suppose
an explicit "repository not found" error may be better here.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Sam James]

Jay Faulkner <jayf@gentoo.org> writes:

> On 3/21/2025 6:42 PM, Ionen Wolkens wrote:
>> On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
>>> Hello, everyone.
>>>
>>> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
>>> except for gentoo and guru.
>> Unfortunate, but just to say that I have nothing to say against
>> dropping these if it's a maintenance burden.
>>
>>> So what I'm thinking about is winding most of the project down.  We'd
>>> stop mirroring third-party repositories, and remove most of gentoo-
>>> mirror organization.
>> Had one concern but complete removal (rather than keeping mirrors
>> archived) would mostly handle it. Aka so that users can't keep
>> sync'ing with these dead mirrors by mistake (often caused issues when
>> we removed overlays but mirror stayed behind, users wouldn't notice a
>> thing until lacking updates started to break things).
>
>
> In the OpenStack community, we will put a final commit on HEAD of the
> primary branch removing all the content and putting an EOL notice in
> the readme (e.g. https://opendev.org/openstack/ironic-lib ). This may
> be a good solution in this case as well, potentially?

We can even inject a news item for the repository.

>
> -JayF

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]

On Sat, 2025-03-22 at 15:38 +0000, Sam James wrote:
> Jay Faulkner <jayf@gentoo.org> writes:
> 
> > On 3/21/2025 6:42 PM, Ionen Wolkens wrote:
> > > On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
> > > > Hello, everyone.
> > > > 
> > > > TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> > > > except for gentoo and guru.
> > > Unfortunate, but just to say that I have nothing to say against
> > > dropping these if it's a maintenance burden.
> > > 
> > > > So what I'm thinking about is winding most of the project down.  We'd
> > > > stop mirroring third-party repositories, and remove most of gentoo-
> > > > mirror organization.
> > > Had one concern but complete removal (rather than keeping mirrors
> > > archived) would mostly handle it. Aka so that users can't keep
> > > sync'ing with these dead mirrors by mistake (often caused issues when
> > > we removed overlays but mirror stayed behind, users wouldn't notice a
> > > thing until lacking updates started to break things).
> > 
> > 
> > In the OpenStack community, we will put a final commit on HEAD of the
> > primary branch removing all the content and putting an EOL notice in
> > the readme (e.g. https://opendev.org/openstack/ironic-lib ). This may
> > be a good solution in this case as well, potentially?
> 
> We can even inject a news item for the repository.
> 

I'm thinking of doing a news item in ::gentoo instead.  That should have
better UX than the same news item duplicated over a hundred
repositories.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Richard Freeman]

On 3/22/2025 11:33 AM, Michał Górny wrote:
> On Sat, 2025-03-22 at 08:20 -0700, Jay Faulkner wrote:
>> In the OpenStack community, we will put a final commit on HEAD of the
>> primary branch removing all the content and putting an EOL notice in the
>> readme (e.g. https://opendev.org/openstack/ironic-lib ). This may be a
>> good solution in this case as well, potentially?
>>
>> Not sure.  It would mean that `emerge --sync` will replace all ebuilds
>> with a README file that user might not even notice.  I suppose
>> an explicit "repository not found" error may be better here.

You could potentially use a bit of an attention-getter, like a system 
package with a stable --99999 version on all arches that just dies 
loudly.  Maybe sys-apps/portage?  I'm not sure if you can override a 
profile in an overlay but masking everything in default would also do 
the trick if that works.

-- 

Rich

[gentoo-dev] Re: The uncertain future of repository mirrors

[Anna Vyalkova]

On 2025-03-21, Michał Górny wrote:
> Hello, everyone.
> 
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.
> 
> 
> Over 10 years ago, I've started the repository mirror & CI project. 
> What started as a bunch of shell scripts on a user-donated server, has
> organically grown into a bigger bunch of shell scripts managed by Infra.
> Nevertheless, it's still a bunch of hacks glued together.
> 
> Things don't work well all the time.  Sometimes stuff randomly crashes,
> and I have to SSH and remove local checkouts to make it work.  Sometimes
> the git repositories used to transfer logs grow so big they kill infra.
> Often some repository starts crashing this or another part and needs to
> be disabled.
> 
> To be honest, I have no energy to keep maintaining this.  I'm really
> tired of having to deal with stuff crashing and spamming my mailbox with
> failure mails.  I'm tired of having to go through all the infra hoops
> just to disable another repository that can't work for one reason or
> another.  In fact, I'm even tired that whenever people add new
> repositories to api.gentoo.org, I have to go through that idiotic GitHub
> clickety-click UI to stop receiving notifications for everything that
> happens in these repositories.
> 
> So what I'm thinking about is winding most of the project down.  We'd
> stop mirroring third-party repositories, and remove most of gentoo-
> mirror organization.  What I'd like to leave is mirroring of gentoo
> and guru repositories, since these two we have control of, and are very
> important to Gentoo users.
> 
> So, well, unless someone convinces me otherwise, I'm going to disable
> all other repositories over the next weekend, and remove their mirrors.
> Gentoo and GURU will still be mirrored, and CI will keep running
> as usual.

I believe there's a good reason to also keep the "science" overlay 
mirrored, as otherwise it will be probably removed from Repology:

https://github.com/repology/repology-updater/blob/94a135accee475d72ff32a1eb8982b8b307847c1/repos.d/gentoo/overlays.yaml#L90

Re: [gentoo-dev] The uncertain future of repository mirrors

[Gerion Entrup]

[-- Attachment #1: Type: text/plain, Size: 3648 bytes --]

Am Freitag, 21. März 2025, 14:32:31 Mitteleuropäische Normalzeit schrieb Michał Górny:
> Hello, everyone.
> 
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.
> 
> 
> Over 10 years ago, I've started the repository mirror & CI project. 
> What started as a bunch of shell scripts on a user-donated server, has
> organically grown into a bigger bunch of shell scripts managed by Infra.
> Nevertheless, it's still a bunch of hacks glued together.
> 
> Things don't work well all the time.  Sometimes stuff randomly crashes,
> and I have to SSH and remove local checkouts to make it work.  Sometimes
> the git repositories used to transfer logs grow so big they kill infra.
> Often some repository starts crashing this or another part and needs to
> be disabled.
> 
> To be honest, I have no energy to keep maintaining this.  I'm really
> tired of having to deal with stuff crashing and spamming my mailbox with
> failure mails.  I'm tired of having to go through all the infra hoops
> just to disable another repository that can't work for one reason or
> another.  In fact, I'm even tired that whenever people add new
> repositories to api.gentoo.org, I have to go through that idiotic GitHub
> clickety-click UI to stop receiving notifications for everything that
> happens in these repositories.
> 
> So what I'm thinking about is winding most of the project down.  We'd
> stop mirroring third-party repositories, and remove most of gentoo-
> mirror organization.  What I'd like to leave is mirroring of gentoo
> and guru repositories, since these two we have control of, and are very
> important to Gentoo users.
> 
> So, well, unless someone convinces me otherwise, I'm going to disable
> all other repositories over the next weekend, and remove their mirrors.
> Gentoo and GURU will still be mirrored, and CI will keep running
> as usual.

First of all, thank you for running it in the first place!

Maybe you like to also continuing mirroring semi-official dev repos like kde and qt.
They are exclusively maintained by Gentoo devs, very large, and beneficial for users who want bleeding edge software.
I would expect that these repos are not the main factor of the maintenance due to their high quality.
And, if I'm right, this reduces sync times per user since the CI's metadata creation (I remember the days, when eix-sync needed extremely long).

Currently, you completely provide the repo mirror infrastructure and also deal with all the problems.
Is it possible to shift this in a large part to the overlay maintainer (in a opt-in approach)?

E.g. something like this (I don't know if the Gentoo git directly provide a CI for users, but maybe Github can do this):
- Per default a overlay is not mirrored, this is an "award" that must be earned.
- Enable a CI controlled by each overlay dev (overlay devs must take care of the CI scripts, they get all the emails).
- (Just) provide CI scripts that do the necessary overlay checking (if I'm not wrong, you already need these scripts for Gentoo and GURU).
- Overlay devs are responsible to run these CI scripts in their overlay, fix errors etc.
- Provide infrastructure that provides a mirror with metadata only for overlays that have enabled the CI, pass it, and the overlay dev asked for being mirrored.
- Before mirroring a commit, wait for the overlay CI to pass to make sure to get no errors on your side.
- If an overlay dev somehow changes the CI scripts in a way that it make the mirroring infrastructure to fail: Remove the overlay permanently.


Best,
Gerion





[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 4518 bytes --]

On Sun, Mar 23, 2025 at 11:17:20AM +0100, Gerion Entrup wrote:
> Am Freitag, 21. März 2025, 14:32:31 Mitteleuropäische Normalzeit schrieb Michał Górny:
> > Hello, everyone.
> > 
> > TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> > except for gentoo and guru.
> > 
> > 
> > Over 10 years ago, I've started the repository mirror & CI project. 
> > What started as a bunch of shell scripts on a user-donated server, has
> > organically grown into a bigger bunch of shell scripts managed by Infra.
> > Nevertheless, it's still a bunch of hacks glued together.
> > 
> > Things don't work well all the time.  Sometimes stuff randomly crashes,
> > and I have to SSH and remove local checkouts to make it work.  Sometimes
> > the git repositories used to transfer logs grow so big they kill infra.
> > Often some repository starts crashing this or another part and needs to
> > be disabled.
> > 
> > To be honest, I have no energy to keep maintaining this.  I'm really
> > tired of having to deal with stuff crashing and spamming my mailbox with
> > failure mails.  I'm tired of having to go through all the infra hoops
> > just to disable another repository that can't work for one reason or
> > another.  In fact, I'm even tired that whenever people add new
> > repositories to api.gentoo.org, I have to go through that idiotic GitHub
> > clickety-click UI to stop receiving notifications for everything that
> > happens in these repositories.
> > 
> > So what I'm thinking about is winding most of the project down.  We'd
> > stop mirroring third-party repositories, and remove most of gentoo-
> > mirror organization.  What I'd like to leave is mirroring of gentoo
> > and guru repositories, since these two we have control of, and are very
> > important to Gentoo users.
> > 
> > So, well, unless someone convinces me otherwise, I'm going to disable
> > all other repositories over the next weekend, and remove their mirrors.
> > Gentoo and GURU will still be mirrored, and CI will keep running
> > as usual.
> 
> First of all, thank you for running it in the first place!
> 
> Maybe you like to also continuing mirroring semi-official dev repos like kde and qt.

The Qt overlay is pretty much deprecated. Development was moved to the
main tree for Qt6 (including live ebuilds), the old Qt5 live ebuilds
have no reason to exist, and I currently don't plan to use it for Qt7.

At most just being used for a handful of lxqt-related live ebuilds
right now that weren't moved yet. If that changes, will likely
drop the overlay entirely, so I wouldn't bother keeping the mirror.

KDE's is still used, but given it's primarily for development/testing
and (currently) "mostly" only has live ebuilds, could argue the
metadata cache is not *essential* -- not that I'd have anything against
keeping it if it's wanted.

> They are exclusively maintained by Gentoo devs, very large, and beneficial for users who want bleeding edge software.
> I would expect that these repos are not the main factor of the maintenance due to their high quality.
> And, if I'm right, this reduces sync times per user since the CI's metadata creation (I remember the days, when eix-sync needed extremely long).
> 
> Currently, you completely provide the repo mirror infrastructure and also deal with all the problems.
> Is it possible to shift this in a large part to the overlay maintainer (in a opt-in approach)?
> 
> E.g. something like this (I don't know if the Gentoo git directly provide a CI for users, but maybe Github can do this):
> - Per default a overlay is not mirrored, this is an "award" that must be earned.
> - Enable a CI controlled by each overlay dev (overlay devs must take care of the CI scripts, they get all the emails).
> - (Just) provide CI scripts that do the necessary overlay checking (if I'm not wrong, you already need these scripts for Gentoo and GURU).
> - Overlay devs are responsible to run these CI scripts in their overlay, fix errors etc.
> - Provide infrastructure that provides a mirror with metadata only for overlays that have enabled the CI, pass it, and the overlay dev asked for being mirrored.
> - Before mirroring a commit, wait for the overlay CI to pass to make sure to get no errors on your side.
> - If an overlay dev somehow changes the CI scripts in a way that it make the mirroring infrastructure to fail: Remove the overlay permanently.
> 
> 
> Best,
> Gerion
> 
> 
> 
> 



-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Mitchell Dorrell]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

On Fri, Mar 21, 2025 at 9:33 AM Michał Górny <mgorny@gentoo.org> wrote:

> Hello, everyone.
>
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.
>
>
> Over 10 years ago, I've started the repository mirror & CI project.
> What started as a bunch of shell scripts on a user-donated server, has
> organically grown into a bigger bunch of shell scripts managed by Infra.
> Nevertheless, it's still a bunch of hacks glued together.
>
> Things don't work well all the time.  Sometimes stuff randomly crashes,
> and I have to SSH and remove local checkouts to make it work.  Sometimes
> the git repositories used to transfer logs grow so big they kill infra.
> Often some repository starts crashing this or another part and needs to
> be disabled.
>
> To be honest, I have no energy to keep maintaining this.  I'm really
> tired of having to deal with stuff crashing and spamming my mailbox with
> failure mails.  I'm tired of having to go through all the infra hoops
> just to disable another repository that can't work for one reason or
> another.  In fact, I'm even tired that whenever people add new
> repositories to api.gentoo.org, I have to go through that idiotic GitHub
> clickety-click UI to stop receiving notifications for everything that
> happens in these repositories.
>
> So what I'm thinking about is winding most of the project down.  We'd
> stop mirroring third-party repositories, and remove most of gentoo-
> mirror organization.  What I'd like to leave is mirroring of gentoo
> and guru repositories, since these two we have control of, and are very
> important to Gentoo users.
>
> So, well, unless someone convinces me otherwise, I'm going to disable
> all other repositories over the next weekend, and remove their mirrors.
> Gentoo and GURU will still be mirrored, and CI will keep running
> as usual.
>
> --
> Best regards,
> Michał Górny
>
>
I've been following the discussion, but I still don't know enough to have
an opinion. Why was this infrastructure created in the first place?

-MD

[-- Attachment #2: Type: text/html, Size: 2722 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

On Mon, 2025-03-24 at 08:46 -0400, Mitchell Dorrell wrote:
> I've been following the discussion, but I still don't know enough to have
> an opinion. Why was this infrastructure created in the first place?

There's a number of reasons, and they are still valid today:

1. Syncing against a mirror with metadata cache makes package managers
faster, and some simpler tools more functional.  However, this only
works as long as the mirror is updated -- when things break and we
disable one, people end up with outdated repository.

2. We check repositories for major issues, such as ebuilds failing
because of old EAPI or removed eclass.  However, we don't have time to
file bugs and deal with the feedback.

3. Having a single place with all the repositories make it possible to
do some cross-repository searches and analysis easier.  For example, we
can estimate how many ebuilds from third-party repositories are still
using a particular eclass.

4. In the end, mirroring repositories mean there's a copy if
the original repository is removed.  The way we merge things, we also
preserve the history when upstream repository is force-pushed.  The flip
side is that if the repository owner tries to remove some data from
history, we normally preserve it.

5. We provide git mirrors with history for repositories that use non-
history-preserving protocols like rsync.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Jay Faulkner]

[-- Attachment #1: Type: text/plain, Size: 591 bytes --]

On 3/25/2025 1:51 PM, Michał Górny wrote:
> On Mon, 2025-03-24 at 08:46 -0400, Mitchell Dorrell wrote:
>> I've been following the discussion, but I still don't know enough to have
>> an opinion. Why was this infrastructure created in the first place?
> There's a number of reasons, and they are still valid today:

[snip]

I'm mildly surprised that "get someone else to host this for 'free'" 
isn't one of the reasons. I'm assuming there's plenty of headroom on 
Gentoo's git servers to handle the influx of git requests that would 
likely be a side effect of this change?

- Jay Faulkner

[-- Attachment #2: Type: text/html, Size: 1269 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

On Tue, 2025-03-25 at 16:15 -0700, Jay Faulkner wrote:
> On 3/25/2025 1:51 PM, Michał Górny wrote:
> > On Mon, 2025-03-24 at 08:46 -0400, Mitchell Dorrell wrote:
> > > I've been following the discussion, but I still don't know enough to have
> > > an opinion. Why was this infrastructure created in the first place?
> > There's a number of reasons, and they are still valid today:
> 
> [snip]
> 
> I'm mildly surprised that "get someone else to host this for 'free'" 
> isn't one of the reasons. I'm assuming there's plenty of headroom on 
> Gentoo's git servers to handle the influx of git requests that would 
> likely be a side effect of this change?
> 

Most of the mirrored repositories aren't on Gentoo git server.  This is
part of the problem, because we're already seeing git incompatibilities
between Codeberg and GitHub.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] The uncertain future of repository mirrors

[Jay Faulkner]

On 3/25/25 11:01 PM, Michał Górny wrote:
> On Tue, 2025-03-25 at 16:15 -0700, Jay Faulkner wrote:
>> On 3/25/2025 1:51 PM, Michał Górny wrote:
>>> On Mon, 2025-03-24 at 08:46 -0400, Mitchell Dorrell wrote:
>>>> I've been following the discussion, but I still don't know enough to have
>>>> an opinion. Why was this infrastructure created in the first place?
>>> There's a number of reasons, and they are still valid today:
>> [snip]
>>
>> I'm mildly surprised that "get someone else to host this for 'free'"
>> isn't one of the reasons. I'm assuming there's plenty of headroom on
>> Gentoo's git servers to handle the influx of git requests that would
>> likely be a side effect of this change?
>>
> Most of the mirrored repositories aren't on Gentoo git server.  This is
> part of the problem, because we're already seeing git incompatibilities
> between Codeberg and GitHub.
>

Ah, this makes a lot of sense. Codeberg in particular has been hit hard 
with DoS attacks recently; I bet it has been maintenance-hell. Thanks 
for the context.


-Jay

Re: [gentoo-dev] The uncertain future of repository mirrors

[Florian Schmaus]

On 21/03/2025 14.32, Michał Górny wrote:
> Hello, everyone.
> 
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.

Somewhat related: we may want to consider slightly raising the bar for 
adding new (user) overlays. When mangling the overlay addition requests, 
there are requests for overlays containing only a few, sometimes even 
just one, package(s).

I believe in many cases, GURU would be a better place for those ebuilds, 
instead of having an them in an additional overlay. I usually encourage 
users to maintain their ebuilds in GURU instead of an extra overlay.

Of course, this should not be seen as black and white. There may be 
valid reasons for an extra overlay. But we should at least ask the user 
to consider using GURU instead.

- Flow

Re: [gentoo-dev] The uncertain future of repository mirrors

[Michael Mair-Keimberger]

On 2025-03-21 14:32, Michał Górny wrote:
> Hello, everyone.
> 
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.
> 
> 
> Over 10 years ago, I've started the repository mirror & CI project.
> What started as a bunch of shell scripts on a user-donated server, has
> organically grown into a bigger bunch of shell scripts managed by 
> Infra.
> Nevertheless, it's still a bunch of hacks glued together.
> 
> Things don't work well all the time.  Sometimes stuff randomly crashes,
> and I have to SSH and remove local checkouts to make it work.  
> Sometimes
> the git repositories used to transfer logs grow so big they kill infra.
> Often some repository starts crashing this or another part and needs to
> be disabled.
> 
> To be honest, I have no energy to keep maintaining this.  I'm really
> tired of having to deal with stuff crashing and spamming my mailbox 
> with
> failure mails.  I'm tired of having to go through all the infra hoops
> just to disable another repository that can't work for one reason or
> another.  In fact, I'm even tired that whenever people add new
> repositories to api.gentoo.org, I have to go through that idiotic 
> GitHub
> clickety-click UI to stop receiving notifications for everything that
> happens in these repositories.
> 
> So what I'm thinking about is winding most of the project down.  We'd
> stop mirroring third-party repositories, and remove most of gentoo-
> mirror organization.  What I'd like to leave is mirroring of gentoo
> and guru repositories, since these two we have control of, and are very
> important to Gentoo users.
> 
> So, well, unless someone convinces me otherwise, I'm going to disable
> all other repositories over the next weekend, and remove their mirrors.
> Gentoo and GURU will still be mirrored, and CI will keep running
> as usual.

Hi,

I guess i'm a bit late in this discussion but i wanted to let you know 
this would also affect my gentoo qa scripts. 
(https://gentooqa.levelnine.at). Right now i'm checking the gentoo, 
guru, kde, science and pentoo repositories, syncing the repos from 
gentoo-mirror for the pre-created metadata.
While the gentoo repository is probably the most valuable, kde, science 
and pentoo checks would be non functional and i probably would have to 
remove them. (at least until i update my script to create the metadata 
myself..)
I guess it's not a big loss, just wanted to let you know.

Regards
Michael

Re: [gentoo-dev] The uncertain future of repository mirrors

[Tim Harder]

On 2025-03-30 Sun 02:11, Michael Mair-Keimberger wrote:
>I guess i'm a bit late in this discussion but i wanted to let you know 
>this would also affect my gentoo qa scripts. 
>(https://gentooqa.levelnine.at). Right now i'm checking the gentoo, 
>guru, kde, science and pentoo repositories, syncing the repos from 
>gentoo-mirror for the pre-created metadata.
>While the gentoo repository is probably the most valuable, kde, 
>science and pentoo checks would be non functional and i probably would 
>have to remove them. (at least until i update my script to create the 
>metadata myself..)

Just to point out, generating metadata can be relatively quick these
days for large repos using non-portage tools. Anyone using egencache or
`emerge --regen` from portage (especially for new repos without any
metadata) is wasting a lot of CPU cycles and memory mainly due to
portage's inefficient design executing a new instance of bash per
ebuild.

Alternatives would be `pmaint regen` from pkgcore (roughly 4x faster
than portage) and `pk repo metadata regen` from pkgcraft-tools (roughly
10x faster than portage while using 10x less memory than pkgcore). On
semi-decent desktop hardware a full gentoo repo metadata run using
pkgcraft takes ~10-20 seconds and less than a second for full
verification only. For those interested in more design discussion and
benchmarks see [1].

Tim

[1]: https://pkgcraft.github.io/posts/metadata-cache-generation/

Re: [gentoo-dev] The uncertain future of repository mirrors

[Florian Schmaus]

[-- Attachment #1.1.1: Type: text/plain, Size: 1071 bytes --]

On 30/03/2025 10.11, Michael Mair-Keimberger wrote:
> On 2025-03-21 14:32, Michał Górny wrote:
>> Hello, everyone.
>>
>> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
>> except for gentoo and guru.
> 
> Hi,
> 
> I guess i'm a bit late in this discussion but i wanted to let you know 
> this would also affect my gentoo qa scripts. (https:// 
> gentooqa.levelnine.at). Right now i'm checking the gentoo, guru, kde, 
> science and pentoo repositories, syncing the repos from gentoo-mirror 
> for the pre-created metadata.
> While the gentoo repository is probably the most valuable, kde, science 
> and pentoo checks would be non functional and i probably would have to 
> remove them. (at least until i update my script to create the metadata 
> myself..)

Updating the script would ideally just consist of adding the line

     pkg repo metadata regen --use-local $REPO

to the script.

As radhermit wrote, pkgcraft is pretty fast these days. I do this for 
multiple repos on every sync and barely notice it.

- Flow

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 21567 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]