From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 5BDDC1581EE for ; Sun, 23 Mar 2025 17:51:18 +0000 (UTC) Received: from lists.gentoo.org (bobolink.gentoo.org [140.211.166.189]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) (Authenticated sender: relay-lists.gentoo.org@gentoo.org) by smtp.gentoo.org (Postfix) with ESMTPSA id 3458C34316E for ; Sun, 23 Mar 2025 17:51:18 +0000 (UTC) Received: from bobolink.gentoo.org (localhost [127.0.0.1]) by bobolink.gentoo.org (Postfix) with ESMTP id F39F61104B5; Sun, 23 Mar 2025 17:50:32 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bobolink.gentoo.org (Postfix) with ESMTPS id 374B41103B8 for ; Sun, 23 Mar 2025 17:50:32 +0000 (UTC) Received: from eversor (unknown [24.157.215.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: ionen) by smtp.gentoo.org (Postfix) with ESMTPSA id C4811343318 for ; Sun, 23 Mar 2025 17:50:31 +0000 (UTC) Date: Sun, 23 Mar 2025 13:50:29 -0400 From: Ionen Wolkens To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] The uncertain future of repository mirrors Message-ID: Mail-Followup-To: gentoo-dev@lists.gentoo.org References: <6b358608f6e244cb96ce527ad47b3e0483eaf0c6.camel@gentoo.org> <23876217.6Emhk5qWAg@falbala> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="yewn/LrQInGdREV6" Content-Disposition: inline In-Reply-To: <23876217.6Emhk5qWAg@falbala> X-Archives-Salt: 32caabd5-4c67-4e61-b39a-3313a5825167 X-Archives-Hash: 7d91f080093ff7fa06f055603ee0f1a6 --yewn/LrQInGdREV6 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Mar 23, 2025 at 11:17:20AM +0100, Gerion Entrup wrote: > Am Freitag, 21. M=C3=A4rz 2025, 14:32:31 Mitteleurop=C3=A4ische Normalzei= t schrieb Micha=C5=82 G=C3=B3rny: > > Hello, everyone. > >=20 > > TL;DR: I'm thinking of shutting down all gentoo-mirror repositories, > > except for gentoo and guru. > >=20 > >=20 > > Over 10 years ago, I've started the repository mirror & CI project.=20 > > What started as a bunch of shell scripts on a user-donated server, has > > organically grown into a bigger bunch of shell scripts managed by Infra. > > Nevertheless, it's still a bunch of hacks glued together. > >=20 > > Things don't work well all the time. Sometimes stuff randomly crashes, > > and I have to SSH and remove local checkouts to make it work. Sometimes > > the git repositories used to transfer logs grow so big they kill infra. > > Often some repository starts crashing this or another part and needs to > > be disabled. > >=20 > > To be honest, I have no energy to keep maintaining this. I'm really > > tired of having to deal with stuff crashing and spamming my mailbox with > > failure mails. I'm tired of having to go through all the infra hoops > > just to disable another repository that can't work for one reason or > > another. In fact, I'm even tired that whenever people add new > > repositories to api.gentoo.org, I have to go through that idiotic GitHub > > clickety-click UI to stop receiving notifications for everything that > > happens in these repositories. > >=20 > > So what I'm thinking about is winding most of the project down. We'd > > stop mirroring third-party repositories, and remove most of gentoo- > > mirror organization. What I'd like to leave is mirroring of gentoo > > and guru repositories, since these two we have control of, and are very > > important to Gentoo users. > >=20 > > So, well, unless someone convinces me otherwise, I'm going to disable > > all other repositories over the next weekend, and remove their mirrors. > > Gentoo and GURU will still be mirrored, and CI will keep running > > as usual. >=20 > First of all, thank you for running it in the first place! >=20 > Maybe you like to also continuing mirroring semi-official dev repos like = kde and qt. The Qt overlay is pretty much deprecated. Development was moved to the main tree for Qt6 (including live ebuilds), the old Qt5 live ebuilds have no reason to exist, and I currently don't plan to use it for Qt7. At most just being used for a handful of lxqt-related live ebuilds right now that weren't moved yet. If that changes, will likely drop the overlay entirely, so I wouldn't bother keeping the mirror. KDE's is still used, but given it's primarily for development/testing and (currently) "mostly" only has live ebuilds, could argue the metadata cache is not *essential* -- not that I'd have anything against keeping it if it's wanted. > They are exclusively maintained by Gentoo devs, very large, and beneficia= l for users who want bleeding edge software. > I would expect that these repos are not the main factor of the maintenanc= e due to their high quality. > And, if I'm right, this reduces sync times per user since the CI's metada= ta creation (I remember the days, when eix-sync needed extremely long). >=20 > Currently, you completely provide the repo mirror infrastructure and also= deal with all the problems. > Is it possible to shift this in a large part to the overlay maintainer (i= n a opt-in approach)? >=20 > E.g. something like this (I don't know if the Gentoo git directly provide= a CI for users, but maybe Github can do this): > - Per default a overlay is not mirrored, this is an "award" that must be = earned. > - Enable a CI controlled by each overlay dev (overlay devs must take care= of the CI scripts, they get all the emails). > - (Just) provide CI scripts that do the necessary overlay checking (if I'= m not wrong, you already need these scripts for Gentoo and GURU). > - Overlay devs are responsible to run these CI scripts in their overlay, = fix errors etc. > - Provide infrastructure that provides a mirror with metadata only for ov= erlays that have enabled the CI, pass it, and the overlay dev asked for bei= ng mirrored. > - Before mirroring a commit, wait for the overlay CI to pass to make sure= to get no errors on your side. > - If an overlay dev somehow changes the CI scripts in a way that it make = the mirroring infrastructure to fail: Remove the overlay permanently. >=20 >=20 > Best, > Gerion >=20 >=20 >=20 >=20 --=20 ionen --yewn/LrQInGdREV6 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEx3SLh1HBoPy/yLVYskQGsLCsQzQFAmfgSeQACgkQskQGsLCs QzT4jgf+LTQJrZF6Xx6oQiu56NVNf5KAC/DFGlAIP2ORov2XhvK/TOr7DG1I3gJ3 ow1VVlChukGILl+tBbHd47+DlTsGVkxux3G1l+PeQ8LjyDqZ0AMYam0GzkdXwyoC 2e7rnJ47Y1YoR/eF5i91qKHmHcctcc2IZL4vtx2tcwzM1bFKT0eFV1c7bbj69uM5 ihJkS9yFEuFPoaCsoLpBdG9CikO/yJOf+wP/mWDtleezi1Wr24DIp605Cp7eRCcs 9ULhJ/okfunttrypEM3wycHzAi+ZmN9pu1i0xomE1z2TO5xAIkjAoYIgUUXhl9So WPARFLJFefUz1XF6C0goFEiNIin6ZQ== =+1KD -----END PGP SIGNATURE----- --yewn/LrQInGdREV6--