From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id E89AB1387FD for ; Tue, 10 Jun 2014 23:16:04 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 0656EE0931; Tue, 10 Jun 2014 23:16:00 +0000 (UTC) Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id EF1F2E090B for ; Tue, 10 Jun 2014 23:15:58 +0000 (UTC) Received: by mail-wi0-f175.google.com with SMTP id f8so7008225wiw.2 for ; Tue, 10 Jun 2014 16:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=WVge3CRSjn6Icq/nnnE+7CqzGu0qUpX/VozRRHVs2Ss=; b=BLAQwc8qQKbjDvsFAk6v33g995J8/H6mGQysFs7XXwVuaWJN1VGMDKQtQc04yiKl7l a0a1tJSPG2vY7sE6xZmv0O2XYGQx3/Kghs/uRGitlKJr7IQSIyjhl3HxplqVKl/nT0QJ M3dnP4CrO0rmp5ExgWq+PajTAjwiaHjq9ex+dF/cJGHL67TpgGnVBzm4Tyry+JO5tgln gI31FNCNqdrBeR5bUxhodSaYGTtetUJA3OxCTXa4HIc/LDvTbLtxBEfX5gZWnq9C/xV6 FEUaYfuzFegw/uE/orjwLE36vj8ikUTtRy8nhvP2NduhWb+AHMpIx1bgYzPzZDSHrxEy Zp/Q== Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 X-Received: by 10.180.77.70 with SMTP id q6mr41965498wiw.28.1402442157589; Tue, 10 Jun 2014 16:15:57 -0700 (PDT) Sender: freemanrich@gmail.com Received: by 10.216.32.132 with HTTP; Tue, 10 Jun 2014 16:15:57 -0700 (PDT) In-Reply-To: <53978DD1.8020603@gentoo.org> References: <20140607201920.0e0ccd5c@gentoo.org> <53937778.7020604@sporkbox.us> <20140607230815.07bc18e7@marga.jer-c2.orkz.net> <5393B6D8.4080201@gentoo.org> <20140608135616.581807d8@marga.jer-c2.orkz.net> <53947600.50506@gentoo.org> <20140608171543.45bf6ce7@marga.jer-c2.orkz.net> <53948A21.6020809@gentoo.org> <5396106A.4010209@gentoo.org> <53962AF6.1050500@gentoo.org> <20140610194500.2185800828ffd57eb1947614@gmail.com> <53978DD1.8020603@gentoo.org> Date: Tue, 10 Jun 2014 19:15:57 -0400 X-Google-Sender-Auth: CH-yLJvKsy0JjjGphCiXQBe5AW0 Message-ID: Subject: Re: [gentoo-dev] The infinite git migration From: Rich Freeman To: gentoo-dev Content-Type: text/plain; charset=UTF-8 X-Archives-Salt: 04748640-dc29-44a6-95e9-0be55f254043 X-Archives-Hash: 056f3d3b9491ca8264eaba53e0861f49 On Tue, Jun 10, 2014 at 6:59 PM, Patrick Lauer wrote: > The first migration attempt failed after consuming nearly 100GB of RAM! > When it did work it took obscene amounts of time, and the result was > unusably large (e.g. initial checkout would take 16GB RAM on the server, > thus not allowing a few hundred devs to do checkouts the same day). Well, effort required to do the migration isn't too big of a deal since it is a one-time event. As long as it can be done correctly, it is good enough. Ferringb was routinely crunching it in something like an hour at the end of last year, granted on a system with an obscene number of cores. > The current state is almost usable, but it is still obscenely slow (e.g. > initial clone taking ~10 CPU-minutes just to figure out what to do), but > we can just throw more hardware at it. > (10 minutes @ 3.6Ghz, so on my notebook it'll take about 4h to just > clone the friggin repository. Too awesome!) Hmm, can't say I've tried it on a variety of systems but I never had trouble cloning it. You are cloning from a bundle right, and not just hitting the server for the whole tree, right? The thought is to have a hook that will block anybody from actually doing a full clone from the server. Instead users would clone from a bundle (either full or with shallow history) and then do pulls from the server vs that. The bundle would just be distributed on the mirrors. > > Another part: Few people thought about the difficult problems in the > migration. For example the rsync mirrors, which will still be used - the > scripts that generate those will need to be changed, tested, and then > replaced if a migration ever happens. Well, I wouldn't say that few have thought about it. The bigger issue is that I don't think any of the existing code is published anywhere, so it is a bit hard to contribute to it. If that isn't the case I'm happy to be corrected. But, yes, that is the bigger block right now. Rich