[gentoo-scm] GIT->rsync

[gentoo-scm] GIT->rsync

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2281 bytes --]

Hello,

Another important topic related to the git migration is how rsync tree
is supposed to be created. Here is how I see it.

I. GIT repository side.

The only related mechanism at the GIT repository would be a post-commit
hook, sending a signal to the rsync server. The signal would only
notify the rsync side that the repository needs to be pulled, without
further information.

II. rsync server side.

The main part of the server layout is a hybrid git-rsync tree. It
starts as a GIT repository checkout, which is being filled with missing
data (i.e. ebuild checksums) by rsync scripting.

After receiving the signal, the rsync scripting pulls from
the repository and updates the working copy. Afterwards, it determines
which files have changed (using diffstat?) and calls repoman (or other
script) to regenerate the full Manifests for the related directories.

That's the simplest scenario. In fact, as our working tree would
contain Manifests changed for rsync users, most of the merges would
result in conflicts -- which should be resolved through always using
the remote version (as our own is going to be recreated anyway).

The problem is that git doesn't seem to like having conflicts with
local uncommited changes. Is there a way to force it to replace them
with the remote versions?

There would be probably some more problems related to file removal too.

Another useful thing would be atomization of the rsync updates to avoid
the possibility of downloading Thin Manifests by end user. The simplest
solution for that would be to keep two distinct trees, with working
tree being rsynced onto public tree after the updates are done.

But that requires keeping two copies of the tree, and doesn't provide
full atomization anyway (it's still possible that end-user downloads
partially copied directory, i.e. with new ebuilds and Manifest not yet
updated).

Thus, it'd be best to have the atomization (i.e. commit-update)
behaviour on filesystem level. If that's not possible, we could still
use poor man's solution similar to VGA page switching - two switching
rsync trees, one being used by end-users and the other being updated.

-- 
Best regards,
Michał Górny

<http://mgorny.alt.pl>
<xmpp:mgorny@jabber.ru>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-scm] GIT->rsync

[Jeremy Olexa]

On Mon, 19 Apr 2010 17:31:28 +0200, Michał Górny <gentoo@mgorny.alt.pl>
wrote:
> Hello,

Hello, overall I am confused about what you are asking or proposing. Some
comments inline.

> 
> Another important topic related to the git migration is how rsync tree
> is supposed to be created. Here is how I see it.
> 
> I. GIT repository side.
> 
> The only related mechanism at the GIT repository would be a post-commit
> hook, sending a signal to the rsync server. The signal would only
> notify the rsync side that the repository needs to be pulled, without
> further information.

On the rsync staging server, it should/will be time interval based.

> 
> II. rsync server side.
> 
> The main part of the server layout is a hybrid git-rsync tree. It
> starts as a GIT repository checkout, which is being filled with missing
> data (i.e. ebuild checksums) by rsync scripting.

I feel like this should be solved by a new manifest proposal not in the
git->rsync generation. Eg. MetaManifests or ThinManifests.

> After receiving the signal, the rsync scripting pulls from
> the repository and updates the working copy. Afterwards, it determines
> which files have changed (using diffstat?) and calls repoman (or other
> script) to regenerate the full Manifests for the related directories.

repoman seems like it would be too expensive for this.

> That's the simplest scenario. In fact, as our working tree would
> contain Manifests changed for rsync users, most of the merges would
> result in conflicts -- which should be resolved through always using
> the remote version (as our own is going to be recreated anyway).
> 
> The problem is that git doesn't seem to like having conflicts with
> local uncommited changes. Is there a way to force it to replace them
> with the remote versions?
> 
> There would be probably some more problems related to file removal too.

Where would the file removal problems be? rsync (emerge --sync) will
handle this like normal.

> 
> Another useful thing would be atomization of the rsync updates to avoid
> the possibility of downloading Thin Manifests by end user. The simplest
> solution for that would be to keep two distinct trees, with working
> tree being rsynced onto public tree after the updates are done.

This is handled on the rsync staging host as well. All downstream mirrors
will get an "atomic tree" from the staged copy.

> 
> But that requires keeping two copies of the tree, and doesn't provide
> full atomization anyway (it's still possible that end-user downloads
> partially copied directory, i.e. with new ebuilds and Manifest not yet
> updated).
> 
> Thus, it'd be best to have the atomization (i.e. commit-update)
> behaviour on filesystem level. If that's not possible, we could still
> use poor man's solution similar to VGA page switching - two switching
> rsync trees, one being used by end-users and the other being updated.

In short, let's work out a defined Manifest proposal before worrying about
the rsync generation for mirrors. I think that this proposal is just
re-highlighting a known issue. Issue being, current Manifests are not
ideal.

-Jeremy