Re: [gentoo-project] Call for agenda items - Council meeting 2021-04-11

[Alec Warner <antarus@gentoo.org>]

On Thu, Apr 8, 2021 at 10:22 AM Joonas Niilola <juippis@gentoo.org> wrote:
>
>
>
> On 4/8/21 7:37 PM, Alec Warner wrote:
> >
> > It's admittedly a grey area here. We use CDNs for various web
> > components (packages.gentoo.org for example) and we use a CDN for
> > distfiles.gentoo.org. Is Github simply a CDN for gentoo.git? Its an
> > open question we have discussed on the gentoo-nfp list.
> >
> > In general I'm not really sold on the benefits of git as a replication
> > protocol; while I dislike running a global rsync network the
> > maintenance of the network is basically nil from infra's end and so I
> > don't feel significant pressure to move to git. Could you perhaps
> > articulate why you think it's important for clients to move to git?
> >
> > -A
> >
>

So I want to discuss two thrusts here then. One is that your argument
below is not very convincing (because it has no data that I can use to
verify anything you said) and the second is just the social contract;
like we are basically saying "hey we can't make git work, so we are
going to use a non-free solution" and is that OK; I happen to think it
is not, but see more below.

> It provides much better user experience with continuously faster
> sync-times. Also there's error posts frequently in the forums when using
> rsync, even recently.

I don't want to dig too deep here, but I'm looking again for more
engineering focussed stuff. If we are going to make a technical
argument, make a technical argument.
"it provides a much better user experience" and "it's faster" are not
arguments; or, they are insufficient to convince me of much. I did do
some experimentation.

For example on my box:
 rsync takes about 5s for an up-to-date-check.
 rsync takes about 1m for a daily-like sync (includes up-to-date
check, incremental delivery, and GPG verification with WKD.)
 rsync takes longer for a full sync, but I tend to use web-rsync for that task.
 Github seems to have taken about 4s for an empty sync (e.g. up to date)
 Github seems to have taken about 6s for a small sync (e.g. daily)
 Github seems to have taken 20s for a full sync.

My git-sync verification doesn't seem to be working (isn't it supposed
to check tip-of-free for a sig? It errors out on me.)

I'm skeptical people care deeply about how long syncing takes (60s and
5s are both fast enough for me; but I sync once a day or less.) I
agree prima facie that Github is likely more reliable than rsync for
most users as it's a better maintained product with a scalable
interface.

I also make a repo for anongit.g.o and it's very slow; probably 10-20x
slower than github. We know we have a bunch of tuning to do on the
git-serving side but no staff to do it.


>
> The way I see it, utilizing Github, it can already be implemented
> world-wide (wrt rsync mirrors). And those disliking Github can still
> keep using rsync, or pick the Gentoo-infra hosted sync-repo (until it
> breaks). And regarding that, didn't you say you have a lot of money
> needing to be used? ;)

So two things here. One is that we should care about the social
contract and I think bi-furcating the core parts of gentoo into free
and non-free is a non-trivial change. If we move people to git syncing
and github makes some changes, or goes away, or whatever...it's not
like we have equivalent free setup; our git hosting is literally not
up to the task of serving those users. I think this is the difference
between the core offering (e.g. "gentoo") and the non-free software in
the tree (the add-ons or additional functionality, or whatever.)
Keeping that logic, we could keep rsync as the default sync method and
tell people to use github if they want (syncing from github works
today just fine.) I think that is different from making github the
default git sync provider. I think this is similar in concept to say,
having only free software by default; which is a change we made
somewhat recently, iirc.

This is less true for our CDN stuff, where we could just turn off the
CDN and be fine[0].

>
> Now I don't know if it's actually *doable* already, or if this is one of
> those things nobody brought up yet. That's why I made the post. (infra,
> releng, handbook). If it *is* doable, I don't see why the defaults can't
> be updated to use git. At some point at least, once we figure out the
> global requirements.
>
> -- juippis
>

[0] This is an engineering argument, is the CDN a latency cache (make
things fast) or a capacity cache (make it so our origin servers do not
melt.) I assert the former. If it's the latter we are in the same
situation as with github; if the CDN is gone and our origin servers
melt, it means we failed.