Re: [gentoo-scm] Git Migration: launch plan & schedule (2015/Aug/08-09)

[Rich Freeman <rich0@gentoo.org>]

On Thu, Jul 2, 2015 at 8:51 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Thu, Jul 02, 2015 at 06:05:10PM -0400, Rich Freeman wrote:
>> On Thu, Jul 2, 2015 at 5:39 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
>> > 2015/08/08 15:00 UTC - Freeze
>> > 2015/08/11           - History repo available to graft
>> I'm not sure if you already had plans for creating the history, but
>> I've generally been able to turn them around in 12 hours without
>> really trying too hard.
> I expect verification of the conversion to take longer than the runs,
> and that we're going to find we miss something at least once, so it'll
> be 2 or 3 runs before we get output we're happy with.

I'm sure we'll be doing 10-20 runs over the subsequent 3 years before
we really get it "perfect."  :)  I've never had an issue with doing a
migration, but if you look closely enough I'm sure you'll find
something that could be done better, especially with the commit
merging.  In particular you'll probably almost never get a tree in the
past where all the Manfests are valid.  I don't really see that as
being important, however.

>
>> I have it all running in a container/chroot
>> where I mount the cvsroot so I could probably get it running on EC2 or
>> elsewhere easily enough if we're in a real hurry.
> Can you publish the container? Letting other people be able to replicate
> the result here would be a benefit, and we can easily shuffle it as
> well.

I'll take a look at it and see if I can clean out anything sensitive.
I have published all my conversion sources on github:
https://github.com/gentoo/git-migration-scripts-rich0

I'll publish a tarball of the container when I can make sure it
doesn't contain any keys/etc.  I do believe that you need a patch for
cvs2svn to get it to use the old date format.

>
>> If I had a ton of RAM and store everything in tmpfs then I suspect
>> that it could be done faster -
> How much RAM & CPU have you been running it with? I was planning on
> 64GB+ tmpfs and 12+ dedicated cores.

I've been running it on 16GB+4 amd64 cores (ie nothing all that
fancy).  I do the output to tmpfs and use a squashfs cvsroot as the
input typically, but that isn't ideal in a hurry since creating a
squashfs takes time (the goal was to cut down on disk IO).  The actual
migration uses very little RAM - it is just whatever you stick in
tmpfs.

>> With git replace we don't need the history to go live, and this also
>> means that if we find a mistake in the history we can fix it and issue
>> a new history.  So, there isn't really a rush.  But, if you don't
>> already have the history conversion taken care of or need it faster, I
>> can take care of it.
> The history is needed within a few days if we go with the changelog
> generation step; but is NOT in the critical path for developers being
> able to carry on committing.
>

I'll go ahead and try to give you what I have.  It sounds like you
have it under control, but if you need me to do anything just let me
know.  I'll see about getting that container packaged up in case it is
useful.  That was why I containerized it in the first place.  I
typically run it in a network namespace using nspawn for convenience,
but I'm sure you could easily just chroot into the thing after
bind-mounting in whatever you need.

--
Rich