Re: [gentoo-scm] Git Migration: launch plan & schedule (2015/Aug/08-09)

["Robin H. Johnson" <robbat2@gentoo.org>]

On Thu, Jul 02, 2015 at 06:05:10PM -0400, Rich Freeman wrote:
> On Thu, Jul 2, 2015 at 5:39 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> > 2015/08/08 15:00 UTC - Freeze
> > 2015/08/11           - History repo available to graft
> I'm not sure if you already had plans for creating the history, but
> I've generally been able to turn them around in 12 hours without
> really trying too hard. 
I expect verification of the conversion to take longer than the runs,
and that we're going to find we miss something at least once, so it'll
be 2 or 3 runs before we get output we're happy with.

> I have it all running in a container/chroot
> where I mount the cvsroot so I could probably get it running on EC2 or
> elsewhere easily enough if we're in a real hurry.  
Can you publish the container? Letting other people be able to replicate
the result here would be a benefit, and we can easily shuffle it as
well.

> If I had a ton of RAM and store everything in tmpfs then I suspect
> that it could be done faster - 
How much RAM & CPU have you been running it with? I was planning on
64GB+ tmpfs and 12+ dedicated cores.

> ferringb's design does each package category in a thread so the
> largest categories become the limiter if you have enough cores (it
> does them in reverse order by size so the longest ones do start
> first). 
Yes, that was the design I settled on with him.

> At some point the fetching of cvsroots and pushing of git
> repos becomes limiting - for github the push takes tens of minutes,
> and the repository takes much longer to show up, though I'm sure that
> could all be done faster on our infra, either via git push or letting
> infra clone from a bundle.
I did a test push of a full conversion previously, to measure the
performance, and it was pretty much bandwidth-limited for us (at
more than 100Mbit inbound) not by anything else.

> With git replace we don't need the history to go live, and this also
> means that if we find a mistake in the history we can fix it and issue
> a new history.  So, there isn't really a rush.  But, if you don't
> already have the history conversion taken care of or need it faster, I
> can take care of it.
The history is needed within a few days if we go with the changelog
generation step; but is NOT in the critical path for developers being
able to carry on committing.

> The most recent conversion is at:
> https://github.com/gentoo/gentoo-gitmig-20150614
Can you publish the container you mention, rather than just the results?

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85