From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 91FF7138334 for ; Tue, 18 Dec 2018 20:30:08 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 1A252E07FE; Tue, 18 Dec 2018 20:30:07 +0000 (UTC) Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id C54A5E0788 for ; Tue, 18 Dec 2018 20:30:06 +0000 (UTC) Received: by mail-ed1-f44.google.com with SMTP id b14so14998466edt.6 for ; Tue, 18 Dec 2018 12:30:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=R1yiMEvU9d8Z43Ic5bwQVHHT4uD+fSCO4XN9uPlk3FQ=; b=UWamiN0Wm6NsEt1CUfl6D6zm2bYm+sC75KX/i1V/wPao2YPJE45HDLPyrH68uYbYXy V9QsNrA7juH+KSrpJaqmW1odXm1bSfTvaqIuDpaZVqNq5LGznmtUfxndh5yPSLTj6g9T xlzXHiaM55Mgpo1yIzCU2+pENxsjtxu/U/2216QKOBgEVquKApQfPhugKA3wXP6PDuGw NDnOr/jMzs9CfeCVTYFW63ATZpQNy5biYmUlbiEYVGZPoJwW6rb2RT27vhILpFckV5qZ zAFXifiO7/WFpjkK3XddufBQhr10jwD9SwVeYLRUDmsVVXAT4E7GAWV+QJ1Ylogb5aGq Yebg== X-Gm-Message-State: AA+aEWZlwfb4qzLwyu7sGB2n82op5/5ir6GiH+c9N4bsFMsIwdLR77mX W3pX2Ja+9ZV+nSMawentvX9iR4saOt02mO4qecOxhTPc1uI= X-Google-Smtp-Source: AFSGD/WCqLKJXeTAumLzQebez4D2BtCrLrh1roXPa0c5Q2wmByTPvxX9J/WkWvuLG0Ml4mdgp2niIBpR0okvlSqdCqQ= X-Received: by 2002:a50:a726:: with SMTP id h35mr17665513edc.192.1545165004826; Tue, 18 Dec 2018 12:30:04 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Project discussion list X-BeenThere: gentoo-project@lists.gentoo.org Reply-To: gentoo-project@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <7da3ce86-d7c5-0336-886d-c43a6144a5fb@gentoo.org> In-Reply-To: From: Alec Warner Date: Tue, 18 Dec 2018 15:29:52 -0500 Message-ID: Subject: Re: [gentoo-project] RFC: Dropping rsync as a tree distribution method To: gentoo-project Content-Type: multipart/alternative; boundary="0000000000003b820f057d51c330" X-Archives-Salt: 880767ef-43eb-4d91-a6a6-40653c9e7537 X-Archives-Hash: 468989977f7f469c40a7966955e618e2 --0000000000003b820f057d51c330 Content-Type: text/plain; charset="UTF-8" On Tue, Dec 18, 2018 at 1:39 PM Raymond Jennings wrote: > What if as a first step, rsync was only dropped as the default? > > If you change the default from rsync to git, you'd be closer to > removing rsync, but it's not as drastic as a sudden removal. Would > give time to make sure it works properly without the risk of breaking > everything. > To clarify, my proposal is not a sudden removal of the rsync network. Cost-wise it is cheap to operate. Operationally, I'd prefer to operate fewer systems out of human concerns (fewer moving parts are better.) I'm trying to ascertain what use cases need to be taken into account before rsync is discontinued, hence this thread. -A > > On Tue, Dec 18, 2018 at 10:37 AM Alec Warner wrote: > > > > > > > > On Tue, Dec 18, 2018 at 1:15 PM Brian Evans wrote: > >> > >> On 12/15/2018 11:15 PM, Alec Warner wrote: > >> > Hi, > >> > > >> > I am currently embarking on a plan to redo our existing rsync[0] > mirror > >> > network. The current network has aged a bit. Its likely too large and > is > >> > under-maintained. I think in the ideal case we would instead pivot > this > >> > project to scaling out our git mirror capabilities and slowly migrate > >> > all consumers to pulling the git tree directly. To that end, I'm > looking > >> > for blockers as to why various customers cannot switch to pulling the > >> > gentoo ebuild repository from git[1] instead of rsync. > >> > > >> > So for example: > >> > > >> > - bandwidth concerns (preferably with documentation / data.) > >> > - Firewall concerns > >> > - CPU concerns (e.g. rsync is great for tiny systems?) > >> > - Disk usage for git vs rsync > >> > - Other things i have not thought of. > >> > > >> > -A > >> > > >> > [0] This excludes emerge-webrsync; which I don't plan on touching. > >> > [1] Rich talked about some downsides earlier > >> > at https://lwn.net/Articles/759539/; but while these are challenges > >> > (some fixable) they are not necessarily blockers. > >> > >> I personally would be sad to see rsync go as I use the git developer > >> tree as my main repository on 2 machines. This is so I can develop and > >> update from the single source. These have no news or md5-cache and it > >> can be painful to generate metadata on one of them. > > > > > > So my strawperson response is that you should have 2 repos. > > > > PORTDIR=https://gitweb.gentoo.org/repo/sync/gentoo.git/log/?h=master # > a local copy of this thing. > > PORTDIR_OVERLAY=/path/to/your/checkout/of/gentoo.git > > > > I suspect however that this likely performs ...poorly, particularly in > worst case situations as the 'overlay' would of course be massive in this > configuration. > > > >> > >> > >> I rely on scripts to pull down the rsync metadata to expedite this > >> process. eg. rsync /gentoo-portage/metadata/md5-cache/. Git has > >> no easy sub-tree download equivalent that I know of. > > > > > > So I think overlaying the news and GSLA bits are easy (you have a > post-sync script that cd's into various directories and clones the news and > GSLA repos.) The costly bit is likely the metadata regeneration for your > development branch of the tree. I'd be curious to see how much this costs > (both cold and hot) for you to generate locally. > > > > -A > > > >> > >> > >> Brian > >> > > --0000000000003b820f057d51c330 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, Dec 18, 2018 at 1:39 PM Raymond J= ennings <shentino@gmail.com>= ; wrote:
What if as a first step, rsync was only dropped as the = default?

If you change the default from rsync to git, you'd be closer to
removing rsync, but it's not as drastic as a sudden removal.=C2=A0 Woul= d
give time to make sure it works properly without the risk of breaking
everything.

To clarify, my proposal is = not a sudden removal of the rsync network. Cost-wise it is cheap to operate= .
Operationally, I'd prefer to operate fewer systems out of h= uman concerns (fewer moving parts are better.)

I&#= 39;m trying to ascertain what use cases need to be taken into account befor= e rsync is discontinued, hence this thread.

-A
=C2=A0

On Tue, Dec 18, 2018 at 10:37 AM Alec Warner <antarus@gentoo.org> wrote:
>
>
>
> On Tue, Dec 18, 2018 at 1:15 PM Brian Evans <grknight@gentoo.org> wrote:
>>
>> On 12/15/2018 11:15 PM, Alec Warner wrote:
>> > Hi,
>> >
>> > I am currently embarking on a plan to redo our existing rsync= [0] mirror
>> > network. The current network has aged a bit. Its likely too l= arge and is
>> > under-maintained. I think in the ideal case we would instead = pivot this
>> > project to scaling out our git mirror capabilities and slowly= migrate
>> > all consumers to pulling the git tree directly. To that end, = I'm looking
>> > for blockers as to why various customers cannot switch to pul= ling the
>> > gentoo ebuild repository from git[1] instead of rsync.
>> >
>> > So for example:
>> >
>> > - bandwidth concerns (preferably with documentation / data.)<= br> >> > - Firewall concerns
>> > - CPU concerns (e.g. rsync is great for tiny systems?)
>> > - Disk usage for git vs rsync
>> > - Other things i have not thought of.
>> >
>> > -A
>> >
>> > [0] This excludes emerge-webrsync; which I don't plan on = touching.
>> > [1] Rich talked about some downsides earlier
>> > at https://lwn.net/Articles/759539/; but while the= se are challenges
>> > (some fixable) they are not necessarily blockers.
>>
>> I personally would be sad to see rsync go as I use the git develop= er
>> tree as my main repository on 2 machines. This is so I can develop= and
>> update from the single source.=C2=A0 These have no news or md5-cac= he and it
>> can be painful to generate metadata on one of them.
>
>
> So my strawperson response is that you should have 2 repos.
>
> PORTDIR=3Dhttps://gitweb.gentoo.o= rg/repo/sync/gentoo.git/log/?h=3Dmaster # a local copy of this thing. > PORTDIR_OVERLAY=3D/path/to/your/checkout/of/gentoo.git
>
> I suspect however that this likely performs ...poorly, particularly in= worst case situations as the 'overlay' would of course be massive = in this configuration.
>
>>
>>
>> I rely on scripts to pull down the rsync metadata to expedite this=
>> process. eg. rsync <host>/gentoo-portage/metadata/md5-cache/= .=C2=A0 Git has
>> no easy sub-tree download equivalent that I know of.
>
>
> So I think overlaying the news and GSLA bits are easy (you have a post= -sync script that cd's into various directories and clones the news and= GSLA repos.) The costly bit is likely the metadata regeneration for your d= evelopment branch of the tree. I'd be curious to see how much this cost= s (both cold and hot) for you to generate locally.
>
> -A
>
>>
>>
>> Brian
>>

--0000000000003b820f057d51c330--