Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method

[Raymond Jennings <shentino@gmail.com>]

On Tue, Dec 18, 2018 at 1:56 AM Andrew Savchenko <bircoph@gentoo.org> wrote:
> On Sat, 15 Dec 2018 23:15:47 -0500 Alec Warner wrote:
> > Hi,
> >
> > I am currently embarking on a plan to redo our existing rsync[0] mirror
> > network. The current network has aged a bit. Its likely too large and is
> > under-maintained. I think in the ideal case we would instead pivot this
> > project to scaling out our git mirror capabilities and slowly migrate all
> > consumers to pulling the git tree directly. To that end, I'm looking for
> > blockers as to why various customers cannot switch to pulling the gentoo
> > ebuild repository from git[1] instead of rsync.
> >
> > So for example:
> >
> > - bandwidth concerns (preferably with documentation / data.)
> > - Firewall concerns
> > - CPU concerns (e.g. rsync is great for tiny systems?)
> > - Disk usage for git vs rsync
> > - Other things i have not thought of.
>
> My main concern with git is downlink fault tolerance. If rsync
> connection is broken, it can be easily restored without much data
> retransmission. If git download connection is broken, it has to
> start all over again. So there are cases where rsync will be always
> much more preferable than git.

Are you talking about in comparison to the initial clone?
If so, would having the clone default to shallow mitigate this?

For the curious, I ran a benchmark.

With a completely purged /usr/portage:

emerge-webrsync took 30.302s
emerge-sync (with git clone --depth 1) took 33.902s
emerge-sync (with regular rsync) took a whoping 1m25.863s

After a fresh sync:

emerge-sync (with regular rsync) took 7.564s
emerge-sync (with git fetch --depth 1, and after priming the repo with
a full clone) took 2.086s



Up front, webrsync seems to be a small winner for initial setups, with
git clone a close second, and regular rsync is 3 fold worse

Routine syncs would seem to prefer git, especially if they are done
with presistent regularity which IMO would amortize things.  My
opinion is that over time git would also place less stress on the
servers since it only has to look at the commit chain instead of
checksumming every single file.



That said, would I be correct to surmise that you're advancing a
robustness issue and not simply a performance issue?


> Best regards,
> Andrew Savchenko