Re: [gentoo-dev] The uncertain future of repository mirrors

[Gerion Entrup <gerion.entrup@flump.de>]

[-- Attachment #1: Type: text/plain, Size: 3648 bytes --]

Am Freitag, 21. März 2025, 14:32:31 Mitteleuropäische Normalzeit schrieb Michał Górny:
> Hello, everyone.
> 
> TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
> except for gentoo and guru.
> 
> 
> Over 10 years ago, I've started the repository mirror & CI project. 
> What started as a bunch of shell scripts on a user-donated server, has
> organically grown into a bigger bunch of shell scripts managed by Infra.
> Nevertheless, it's still a bunch of hacks glued together.
> 
> Things don't work well all the time.  Sometimes stuff randomly crashes,
> and I have to SSH and remove local checkouts to make it work.  Sometimes
> the git repositories used to transfer logs grow so big they kill infra.
> Often some repository starts crashing this or another part and needs to
> be disabled.
> 
> To be honest, I have no energy to keep maintaining this.  I'm really
> tired of having to deal with stuff crashing and spamming my mailbox with
> failure mails.  I'm tired of having to go through all the infra hoops
> just to disable another repository that can't work for one reason or
> another.  In fact, I'm even tired that whenever people add new
> repositories to api.gentoo.org, I have to go through that idiotic GitHub
> clickety-click UI to stop receiving notifications for everything that
> happens in these repositories.
> 
> So what I'm thinking about is winding most of the project down.  We'd
> stop mirroring third-party repositories, and remove most of gentoo-
> mirror organization.  What I'd like to leave is mirroring of gentoo
> and guru repositories, since these two we have control of, and are very
> important to Gentoo users.
> 
> So, well, unless someone convinces me otherwise, I'm going to disable
> all other repositories over the next weekend, and remove their mirrors.
> Gentoo and GURU will still be mirrored, and CI will keep running
> as usual.

First of all, thank you for running it in the first place!

Maybe you like to also continuing mirroring semi-official dev repos like kde and qt.
They are exclusively maintained by Gentoo devs, very large, and beneficial for users who want bleeding edge software.
I would expect that these repos are not the main factor of the maintenance due to their high quality.
And, if I'm right, this reduces sync times per user since the CI's metadata creation (I remember the days, when eix-sync needed extremely long).

Currently, you completely provide the repo mirror infrastructure and also deal with all the problems.
Is it possible to shift this in a large part to the overlay maintainer (in a opt-in approach)?

E.g. something like this (I don't know if the Gentoo git directly provide a CI for users, but maybe Github can do this):
- Per default a overlay is not mirrored, this is an "award" that must be earned.
- Enable a CI controlled by each overlay dev (overlay devs must take care of the CI scripts, they get all the emails).
- (Just) provide CI scripts that do the necessary overlay checking (if I'm not wrong, you already need these scripts for Gentoo and GURU).
- Overlay devs are responsible to run these CI scripts in their overlay, fix errors etc.
- Provide infrastructure that provides a mirror with metadata only for overlays that have enabled the CI, pass it, and the overlay dev asked for being mirrored.
- Before mirroring a commit, wait for the overlay CI to pass to make sure to get no errors on your side.
- If an overlay dev somehow changes the CI scripts in a way that it make the mirroring infrastructure to fail: Remove the overlay permanently.


Best,
Gerion





[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 659 bytes --]