On 18/12/18 18:00, Alec Warner wrote:

On Tue, Dec 18, 2018 at 12:14 PM Andrew Savchenko <bircoph@gentoo.org> wrote:

>
> That said, would I be correct to surmise that you're advancing a
> robustness issue and not simply a performance issue?

Yes, my interest here is in robustness, not performance. Sometimes I
have to use unreliable uplink and other users may face the same
problem.

I agree that in most cases git should be a preferred way to go, but
there are exceptions. So it would be nice to have rsync backup just
in case. 

Daily or weekly portage snapshots available via rsync should be a
solution as well.

Two things here. One is that in an ideal world we would run no rsync service and any design should keep that outcome in mind. Operationally we should continue to offer rsync until these types of problems are addressed by the new system.

The second is that in this case I think the plan is to, as Robin mentioned, offer "git bundles" that are over raw http and support resume-able downloads. So instead of downloading an "rsync snapshot" you download a git bundle over http. Infra would offer these git bundles in a similar way to existing rsync snapshot offerings[0]. These bundles would be applied to a machine local clone of a git repo. Does this conceptually address your problem? I agree it will be difficult to know outside of actual practical testing.

-A

[0] http://gentoo.ussg.indiana.edu/snapshots/ is one example of the current system. Instead of tarballs of an 'rsync tree' these would be git bundles[1] that you fetch and apply locally. We would support a worldwide mirror network for these bundles.
 

Best regards,
Andrew Savchenko
I'm inclined to suggest that perhaps you set up the necessary infra to do the git bundles, etc, and we give it a trial - we can postulate and pontificate as long as we like (otherwise known simply as 'bikeshedding') .. but we'll have no "real world data" until we actually implement it (and discover all the pitfalls en route).

We then have to option of pushing through the migration process if it works, or we revert back if it doesn't.

How does that grab you Alec?! :)

MJE/veremitz.