From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 184FD138CD0 for ; Fri, 3 Jul 2015 00:51:26 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 6C390E087F; Fri, 3 Jul 2015 00:51:20 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 09889E087F for ; Fri, 3 Jul 2015 00:51:19 +0000 (UTC) Received: from grubbs.orbis-terrarum.net (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 2782F34086C for ; Fri, 3 Jul 2015 00:51:19 +0000 (UTC) Received: (qmail 15473 invoked by uid 10000); 3 Jul 2015 00:51:18 -0000 Date: Fri, 3 Jul 2015 00:51:18 +0000 From: "Robin H. Johnson" To: gentoo-scm@lists.gentoo.org Subject: Re: [gentoo-scm] Git Migration: launch plan & schedule (2015/Aug/08-09) Message-ID: References: Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo SCM discussions X-BeenThere: gentoo-scm@gentoo.org X-BeenThere: gentoo-scm@lists.gentoo.org Reply-To: gentoo-scm@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-Archives-Salt: 95468dba-73e6-4b91-b5e4-c9359a439925 X-Archives-Hash: b30438dcb592fd444c146a08d9fdef8c On Thu, Jul 02, 2015 at 06:05:10PM -0400, Rich Freeman wrote: > On Thu, Jul 2, 2015 at 5:39 PM, Robin H. Johnson wrote: > > 2015/08/08 15:00 UTC - Freeze > > 2015/08/11 - History repo available to graft > I'm not sure if you already had plans for creating the history, but > I've generally been able to turn them around in 12 hours without > really trying too hard. I expect verification of the conversion to take longer than the runs, and that we're going to find we miss something at least once, so it'll be 2 or 3 runs before we get output we're happy with. > I have it all running in a container/chroot > where I mount the cvsroot so I could probably get it running on EC2 or > elsewhere easily enough if we're in a real hurry. Can you publish the container? Letting other people be able to replicate the result here would be a benefit, and we can easily shuffle it as well. > If I had a ton of RAM and store everything in tmpfs then I suspect > that it could be done faster - How much RAM & CPU have you been running it with? I was planning on 64GB+ tmpfs and 12+ dedicated cores. > ferringb's design does each package category in a thread so the > largest categories become the limiter if you have enough cores (it > does them in reverse order by size so the longest ones do start > first). Yes, that was the design I settled on with him. > At some point the fetching of cvsroots and pushing of git > repos becomes limiting - for github the push takes tens of minutes, > and the repository takes much longer to show up, though I'm sure that > could all be done faster on our infra, either via git push or letting > infra clone from a bundle. I did a test push of a full conversion previously, to measure the performance, and it was pretty much bandwidth-limited for us (at more than 100Mbit inbound) not by anything else. > With git replace we don't need the history to go live, and this also > means that if we find a mistake in the history we can fix it and issue > a new history. So, there isn't really a rush. But, if you don't > already have the history conversion taken care of or need it faster, I > can take care of it. The history is needed within a few days if we go with the changelog generation step; but is NOT in the critical path for developers being able to carry on committing. > The most recent conversion is at: > https://github.com/gentoo/gentoo-gitmig-20150614 Can you publish the container you mention, rather than just the results? -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robbat2@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85