From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from <gentoo-scm+bounces-218-garchives=archives.gentoo.org@lists.gentoo.org>) id 1O3tmA-0002OJ-RV for garchives@archives.gentoo.org; Mon, 19 Apr 2010 16:24:47 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 5E695E07CF; Mon, 19 Apr 2010 16:24:45 +0000 (UTC) Received: from mail.jolexa.net (mail.jolexa.net [69.164.203.224]) by pigeon.gentoo.org (Postfix) with ESMTP id 2134FE07CF for <gentoo-scm@lists.gentoo.org>; Mon, 19 Apr 2010 16:24:45 +0000 (UTC) Received: by mail.jolexa.net (Postfix, from userid 5001) id 9F0615D6A2; Mon, 19 Apr 2010 16:24:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.2.5-gr2 (2008-06-10) on helios.jolexa.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=disabled version=3.2.5-gr2 Received: from webmail.jolexa.net (mail.jolexa.net [69.164.203.224]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.jolexa.net (Postfix) with ESMTPSA id 2E24A5D62F for <gentoo-scm@lists.gentoo.org>; Mon, 19 Apr 2010 16:24:44 +0000 (UTC) Precedence: bulk List-Post: <mailto:gentoo-scm@lists.gentoo.org> List-Help: <mailto:gentoo-scm+help@lists.gentoo.org> List-Unsubscribe: <mailto:gentoo-scm+unsubscribe@lists.gentoo.org> List-Subscribe: <mailto:gentoo-scm+subscribe@lists.gentoo.org> List-Id: Gentoo SCM discussions <gentoo-scm.gentoo.org> X-BeenThere: gentoo-scm@gentoo.org X-BeenThere: gentoo-scm@lists.gentoo.org MIME-Version: 1.0 Date: Mon, 19 Apr 2010 16:24:43 +0000 From: Jeremy Olexa <darkside@gentoo.org> To: <gentoo-scm@lists.gentoo.org> Subject: Re: [gentoo-scm] GIT->rsync In-Reply-To: <20100419173128.33ae2090@pomiot.lan> References: <20100419173128.33ae2090@pomiot.lan> Message-ID: <4c46cffc84e68feae364c62cc7a11e2a@localhost> X-Sender: darkside@gentoo.org User-Agent: RoundCube Webmail/0.3.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 92278a58-16a6-47cd-8c1d-cfc8d8bbd8ac X-Archives-Hash: 241babf41ff57b1560324af036b889c3 On Mon, 19 Apr 2010 17:31:28 +0200, Micha=C5=82 G=C3=B3rny <gentoo@mgorny= .alt.pl> wrote: > Hello, Hello, overall I am confused about what you are asking or proposing. Some comments inline. >=20 > Another important topic related to the git migration is how rsync tree > is supposed to be created. Here is how I see it. >=20 > I. GIT repository side. >=20 > The only related mechanism at the GIT repository would be a post-commit > hook, sending a signal to the rsync server. The signal would only > notify the rsync side that the repository needs to be pulled, without > further information. On the rsync staging server, it should/will be time interval based. >=20 > II. rsync server side. >=20 > The main part of the server layout is a hybrid git-rsync tree. It > starts as a GIT repository checkout, which is being filled with missing > data (i.e. ebuild checksums) by rsync scripting. I feel like this should be solved by a new manifest proposal not in the git->rsync generation. Eg. MetaManifests or ThinManifests. > After receiving the signal, the rsync scripting pulls from > the repository and updates the working copy. Afterwards, it determines > which files have changed (using diffstat?) and calls repoman (or other > script) to regenerate the full Manifests for the related directories. repoman seems like it would be too expensive for this. > That's the simplest scenario. In fact, as our working tree would > contain Manifests changed for rsync users, most of the merges would > result in conflicts -- which should be resolved through always using > the remote version (as our own is going to be recreated anyway). >=20 > The problem is that git doesn't seem to like having conflicts with > local uncommited changes. Is there a way to force it to replace them > with the remote versions? >=20 > There would be probably some more problems related to file removal too. Where would the file removal problems be? rsync (emerge --sync) will handle this like normal. >=20 > Another useful thing would be atomization of the rsync updates to avoid > the possibility of downloading Thin Manifests by end user. The simplest > solution for that would be to keep two distinct trees, with working > tree being rsynced onto public tree after the updates are done. This is handled on the rsync staging host as well. All downstream mirrors will get an "atomic tree" from the staged copy. >=20 > But that requires keeping two copies of the tree, and doesn't provide > full atomization anyway (it's still possible that end-user downloads > partially copied directory, i.e. with new ebuilds and Manifest not yet > updated). >=20 > Thus, it'd be best to have the atomization (i.e. commit-update) > behaviour on filesystem level. If that's not possible, we could still > use poor man's solution similar to VGA page switching - two switching > rsync trees, one being used by end-users and the other being updated. In short, let's work out a defined Manifest proposal before worrying abou= t the rsync generation for mirrors. I think that this proposal is just re-highlighting a known issue. Issue being, current Manifests are not ideal. -Jeremy