From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 28F291382C5 for ; Fri, 9 Apr 2021 04:20:09 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id D7DBFE092D; Fri, 9 Apr 2021 04:20:07 +0000 (UTC) Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id B830FE0922 for ; Fri, 9 Apr 2021 04:20:07 +0000 (UTC) Received: by mail-ej1-x62a.google.com with SMTP id e14so6483258ejz.11 for ; Thu, 08 Apr 2021 21:20:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gentoo-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=gklN63KtXCrIW4YWBPeMuNiZlv/+yo8p82oFNvXcbdU=; b=djswLlBdLmneXNR0+F3wzH2dCA7eygMuty1An4SRMEZUMijnynAvS7AjXsD/iRKA/X qYi91uvzKYIqBakrn1KPf9jtgT3vbMcGsvE4lHvF0mcUqqbFZ7iROJm/SJjr/HAnZDTp dG0Tz73f1MF66pVDBhT86OH7Fljiszai50dgH4futNnFebQPr8Tdv32TOS0Kxx25oVb/ fyheN/focl1VO5aS/ObvAowyi/Eu7Ky7BRMyw0u5tz2THs45bY90TU/Vb36+So0/Ndh0 pkWXLmZoPPia9aOEVyPu3FEZjHqFlMkl1lWtQk0wVZMy9nN+wcjeyGZOAryaN0NqNNwg 5dtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=gklN63KtXCrIW4YWBPeMuNiZlv/+yo8p82oFNvXcbdU=; b=Kw5xaYTve4bAk0l0LarL6nrtW2j+H2bocJpqU+XtRSRhG44EpkJL+pd5t3MvIMSVTX Y7QFTvfr/j83fqED1dD6P1GmdZnyTT4Vz2/6FVdwav/9Y6wGTYTak1nRYenksPMmv/bD KC7nByYF2g2unGWQunTikSmfq3rMQPD106nrgleA80WFyrqtEAKsBzW6LuJoYUCbFn1V NjnNhIMxQFsmCJsRSgVDobzsVJ8rcZZwKrUJ5U8vxtgG9i0kNGvnKqBWDGi9ozH60+aO KMGW8oIVHM20hk8FoLGwzd8ob2k2ARHRJg4oXjCJWL8uhTbcaVpKP1Tg5mmyQy+87z4Y 7UTA== X-Gm-Message-State: AOAM533XEjob7lYuSRaBK0MCvxWlYbvyzaL/td0pWhqFGbUqsG6mlXtL K7pq4ac68H3R/mQxecc+iPdAKSATw3Mi9pjVirIs4B15CYOSOQ== X-Google-Smtp-Source: ABdhPJwDcpGGyS86S+0beXrz8gXvtVtVUReZRjEAM9u6QKfoHKDoj0X6gXrzeRfA7i76fTqnJWu1Cbxro3qCCm9Icxo= X-Received: by 2002:a17:906:94d2:: with SMTP id d18mr6617042ejy.531.1617942005825; Thu, 08 Apr 2021 21:20:05 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Project discussion list X-BeenThere: gentoo-project@lists.gentoo.org Reply-To: gentoo-project@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <2061435.irdbgypaU6@pinacolada> <9661885f-88ac-c176-cc1f-258c4a7654fc@gentoo.org> <9ed4e995-77d4-f2df-14ce-ec1bdfd3959f@gentoo.org> In-Reply-To: From: Alec Warner Date: Thu, 8 Apr 2021 21:19:55 -0700 Message-ID: Subject: Re: [gentoo-project] Call for agenda items - Council meeting 2021-04-11 To: gentoo-project Content-Type: text/plain; charset="UTF-8" X-Archives-Salt: 7705437f-e121-4dc0-8803-f01446b17c68 X-Archives-Hash: 7292f16a084800998d212421281dea9b On Thu, Apr 8, 2021 at 9:03 PM Alec Warner wrote: > > On Thu, Apr 8, 2021 at 10:22 AM Joonas Niilola wrote: > > > > > > > > On 4/8/21 7:37 PM, Alec Warner wrote: > > > > > > It's admittedly a grey area here. We use CDNs for various web > > > components (packages.gentoo.org for example) and we use a CDN for > > > distfiles.gentoo.org. Is Github simply a CDN for gentoo.git? Its an > > > open question we have discussed on the gentoo-nfp list. > > > > > > In general I'm not really sold on the benefits of git as a replication > > > protocol; while I dislike running a global rsync network the > > > maintenance of the network is basically nil from infra's end and so I > > > don't feel significant pressure to move to git. Could you perhaps > > > articulate why you think it's important for clients to move to git? > > > > > > -A > > > > > > > So I want to discuss two thrusts here then. One is that your argument > below is not very convincing (because it has no data that I can use to > verify anything you said) and the second is just the social contract; > like we are basically saying "hey we can't make git work, so we are > going to use a non-free solution" and is that OK; I happen to think it > is not, but see more below. > > > It provides much better user experience with continuously faster > > sync-times. Also there's error posts frequently in the forums when using > > rsync, even recently. > > I don't want to dig too deep here, but I'm looking again for more > engineering focussed stuff. If we are going to make a technical > argument, make a technical argument. > "it provides a much better user experience" and "it's faster" are not > arguments; or, they are insufficient to convince me of much. I did do > some experimentation. > > For example on my box: > rsync takes about 5s for an up-to-date-check. > rsync takes about 1m for a daily-like sync (includes up-to-date > check, incremental delivery, and GPG verification with WKD.) > rsync takes longer for a full sync, but I tend to use web-rsync for that task. > Github seems to have taken about 4s for an empty sync (e.g. up to date) > Github seems to have taken about 6s for a small sync (e.g. daily) > Github seems to have taken 20s for a full sync. > > My git-sync verification doesn't seem to be working (isn't it supposed > to check tip-of-free for a sig? It errors out on me.) > > I'm skeptical people care deeply about how long syncing takes (60s and > 5s are both fast enough for me; but I sync once a day or less.) I > agree prima facie that Github is likely more reliable than rsync for > most users as it's a better maintained product with a scalable > interface. > > I also make a repo for anongit.g.o and it's very slow; probably 10-20x > slower than github. We know we have a bunch of tuning to do on the > git-serving side but no staff to do it. Ah I had a bug here; anongit is: 4s for empty update. 5m for empty sync. I have not done an incremental (because I just fixed my bug.) So I think if we can make it scale it might reach an rsync level of performance, but we need to do the necessary tuning and buildout. -A > > > > > > The way I see it, utilizing Github, it can already be implemented > > world-wide (wrt rsync mirrors). And those disliking Github can still > > keep using rsync, or pick the Gentoo-infra hosted sync-repo (until it > > breaks). And regarding that, didn't you say you have a lot of money > > needing to be used? ;) > > So two things here. One is that we should care about the social > contract and I think bi-furcating the core parts of gentoo into free > and non-free is a non-trivial change. If we move people to git syncing > and github makes some changes, or goes away, or whatever...it's not > like we have equivalent free setup; our git hosting is literally not > up to the task of serving those users. I think this is the difference > between the core offering (e.g. "gentoo") and the non-free software in > the tree (the add-ons or additional functionality, or whatever.) > Keeping that logic, we could keep rsync as the default sync method and > tell people to use github if they want (syncing from github works > today just fine.) I think that is different from making github the > default git sync provider. I think this is similar in concept to say, > having only free software by default; which is a change we made > somewhat recently, iirc. > > This is less true for our CDN stuff, where we could just turn off the > CDN and be fine[0]. > > > > > Now I don't know if it's actually *doable* already, or if this is one of > > those things nobody brought up yet. That's why I made the post. (infra, > > releng, handbook). If it *is* doable, I don't see why the defaults can't > > be updated to use git. At some point at least, once we figure out the > > global requirements. > > > > -- juippis > > > > [0] This is an engineering argument, is the CDN a latency cache (make > things fast) or a capacity cache (make it so our origin servers do not > melt.) I assert the former. If it's the latter we are in the same > situation as with github; if the CDN is gone and our origin servers > melt, it means we failed.