From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 49FED138334 for ; Tue, 18 Dec 2018 11:36:54 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id D4CABE0ABE; Tue, 18 Dec 2018 11:36:52 +0000 (UTC) Received: from mail-oi1-x243.google.com (mail-oi1-x243.google.com [IPv6:2607:f8b0:4864:20::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 9513FE0ABB for ; Tue, 18 Dec 2018 11:36:52 +0000 (UTC) Received: by mail-oi1-x243.google.com with SMTP id a77so1678741oii.5 for ; Tue, 18 Dec 2018 03:36:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=sr/glzVLTEW7BRbZqXxe0yfiwXDyK7VzjJg5BEpy5qI=; b=WmzYdYOmD61Sqs3pUSaiNq/BoJcEq7KpzRI2ZARpXuh+uIZwecuGSAkqD1T8TG+VMu tbShdb9qvlz0Xc/6EJu9Rc9LGzNOAEgT/9tXOlHMsSbR/HOXSk/TlkQHlhMrWZDXKFmR RCqK9MC24DTpvMB928b4lQb0EcawOI6AWo0/bs+Deo9t83jnfdTzDV/amjAx7JkWBjCZ 0RbCu27v67OKk8+e51gUEMk0rg14VPOSpAOIItC87Ggk45Iau9zkN+/8RJr2WvwNda5d /A5xPCQ+kQRaqvYDdkaOm603FhtJobvGlZEptXwirHdys1+hT75pgGH/cm4/6ANj7wS7 K/og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=sr/glzVLTEW7BRbZqXxe0yfiwXDyK7VzjJg5BEpy5qI=; b=JeTScMe4Di2uaeIyuBA9EVA929Cdd1Y7gWJ89SA9vhgeMW6xyLPqCU957Bal08bOls qlJ3xiCDHYR+wli0tjMVD03Oug24ky13j6yLBzl+5UxUme4uhKwuIT9FkSrijW6LB0k6 di2vkZVpPEDsQTIVKmFbJBN2ZHS+TaQMnqWuHx17gdXlqLZNIMBtakcxT4CZaPn7/grC Gn392e/VqWoesT3DqpEFB7NZYkUx5iOjJntsf+m9/e7pZOfa9Iw4gNYjlx+YrbauzAj+ pOwhLTVL0VpoxfJMsswUd5tjvcenzCvY9ysuenhQj0M8GbzctfpPW+TErKPuYzasPDmT Egaw== X-Gm-Message-State: AA+aEWY4jl5L3qCJmbFEkFusLl2QYjiuL3b7Q7zzIhRx/8zlSc+WeWnd 73aeZ1vYR0NLiEIjlwGFO90kGTGnRNd/X/l9x41/Ag== X-Google-Smtp-Source: AFSGD/W1AJ2HGG2jTrx06hS0yOUzob5F+RuqNPF+GP1payIFIKooRSeS9ol07wPjdmBx8/3Qkvt6yhGwXTiBZm9nuuY= X-Received: by 2002:aca:da02:: with SMTP id r2mr7250259oig.66.1545133010962; Tue, 18 Dec 2018 03:36:50 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Project discussion list X-BeenThere: gentoo-project@lists.gentoo.org Reply-To: gentoo-project@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <20181218125555.1927321328046d0a2ecd3e16@gentoo.org> In-Reply-To: <20181218125555.1927321328046d0a2ecd3e16@gentoo.org> From: Raymond Jennings Date: Tue, 18 Dec 2018 03:36:14 -0800 Message-ID: Subject: Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method To: gentoo-project@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" X-Archives-Salt: 29a05935-cd1e-4712-93f0-fcf7caf4a387 X-Archives-Hash: 18a3c897f6142dccb70ceb407663bc97 On Tue, Dec 18, 2018 at 1:56 AM Andrew Savchenko wrote: > On Sat, 15 Dec 2018 23:15:47 -0500 Alec Warner wrote: > > Hi, > > > > I am currently embarking on a plan to redo our existing rsync[0] mirror > > network. The current network has aged a bit. Its likely too large and is > > under-maintained. I think in the ideal case we would instead pivot this > > project to scaling out our git mirror capabilities and slowly migrate all > > consumers to pulling the git tree directly. To that end, I'm looking for > > blockers as to why various customers cannot switch to pulling the gentoo > > ebuild repository from git[1] instead of rsync. > > > > So for example: > > > > - bandwidth concerns (preferably with documentation / data.) > > - Firewall concerns > > - CPU concerns (e.g. rsync is great for tiny systems?) > > - Disk usage for git vs rsync > > - Other things i have not thought of. > > My main concern with git is downlink fault tolerance. If rsync > connection is broken, it can be easily restored without much data > retransmission. If git download connection is broken, it has to > start all over again. So there are cases where rsync will be always > much more preferable than git. Are you talking about in comparison to the initial clone? If so, would having the clone default to shallow mitigate this? For the curious, I ran a benchmark. With a completely purged /usr/portage: emerge-webrsync took 30.302s emerge-sync (with git clone --depth 1) took 33.902s emerge-sync (with regular rsync) took a whoping 1m25.863s After a fresh sync: emerge-sync (with regular rsync) took 7.564s emerge-sync (with git fetch --depth 1, and after priming the repo with a full clone) took 2.086s Up front, webrsync seems to be a small winner for initial setups, with git clone a close second, and regular rsync is 3 fold worse Routine syncs would seem to prefer git, especially if they are done with presistent regularity which IMO would amortize things. My opinion is that over time git would also place less stress on the servers since it only has to look at the commit chain instead of checksumming every single file. That said, would I be correct to surmise that you're advancing a robustness issue and not simply a performance issue? > Best regards, > Andrew Savchenko