[gentoo-scm] Git Migration: launch plan & schedule (2015/Aug/08-09)

[gentoo-scm] Git Migration: launch plan & schedule (2015/Aug/08-09)

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

Hi all,

The Git migration is moving forward, and I'd like to announce a
tentative schedule for that end.
https://wiki.gentoo.org/wiki/Project:Infrastructure/Git_migration#Status

2015/08/08 15:00 UTC - Freeze
2015/08/08 19:00 UTC - Git commits open for developers
2015/08/09 01:00 UTC - Rsync live again (with lagged changelog)
2015/08/11           - History repo available to graft
2015/08/12           - rsync mirrors carry up-to-date changelogs again

I've allocated time for an 8 hour freeze, but hope to be completed much
sooner than that.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 445 bytes --]

Re: [gentoo-scm] Git Migration: launch plan & schedule (2015/Aug/08-09)

[Rich Freeman]

On Thu, Jul 2, 2015 at 5:39 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> 2015/08/08 15:00 UTC - Freeze
> 2015/08/11           - History repo available to graft

I'm not sure if you already had plans for creating the history, but
I've generally been able to turn them around in 12 hours without
really trying too hard.  I have it all running in a container/chroot
where I mount the cvsroot so I could probably get it running on EC2 or
elsewhere easily enough if we're in a real hurry.  If I had a ton of
RAM and store everything in tmpfs then I suspect that it could be done
faster - ferringb's design does each package category in a thread so
the largest categories become the limiter if you have enough cores (it
does them in reverse order by size so the longest ones do start
first). At some point the fetching of cvsroots and pushing of git
repos becomes limiting - for github the push takes tens of minutes,
and the repository takes much longer to show up, though I'm sure that
could all be done faster on our infra, either via git push or letting
infra clone from a bundle.

With git replace we don't need the history to go live, and this also
means that if we find a mistake in the history we can fix it and issue
a new history.  So, there isn't really a rush.  But, if you don't
already have the history conversion taken care of or need it faster, I
can take care of it.

The most recent conversion is at:
https://github.com/gentoo/gentoo-gitmig-20150614

-- 
Rich

Re: [gentoo-scm] Git Migration: launch plan & schedule (2015/Aug/08-09)

[Robin H. Johnson]

On Thu, Jul 02, 2015 at 06:05:10PM -0400, Rich Freeman wrote:
> On Thu, Jul 2, 2015 at 5:39 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> > 2015/08/08 15:00 UTC - Freeze
> > 2015/08/11           - History repo available to graft
> I'm not sure if you already had plans for creating the history, but
> I've generally been able to turn them around in 12 hours without
> really trying too hard. 
I expect verification of the conversion to take longer than the runs,
and that we're going to find we miss something at least once, so it'll
be 2 or 3 runs before we get output we're happy with.

> I have it all running in a container/chroot
> where I mount the cvsroot so I could probably get it running on EC2 or
> elsewhere easily enough if we're in a real hurry.  
Can you publish the container? Letting other people be able to replicate
the result here would be a benefit, and we can easily shuffle it as
well.

> If I had a ton of RAM and store everything in tmpfs then I suspect
> that it could be done faster - 
How much RAM & CPU have you been running it with? I was planning on
64GB+ tmpfs and 12+ dedicated cores.

> ferringb's design does each package category in a thread so the
> largest categories become the limiter if you have enough cores (it
> does them in reverse order by size so the longest ones do start
> first). 
Yes, that was the design I settled on with him.

> At some point the fetching of cvsroots and pushing of git
> repos becomes limiting - for github the push takes tens of minutes,
> and the repository takes much longer to show up, though I'm sure that
> could all be done faster on our infra, either via git push or letting
> infra clone from a bundle.
I did a test push of a full conversion previously, to measure the
performance, and it was pretty much bandwidth-limited for us (at
more than 100Mbit inbound) not by anything else.

> With git replace we don't need the history to go live, and this also
> means that if we find a mistake in the history we can fix it and issue
> a new history.  So, there isn't really a rush.  But, if you don't
> already have the history conversion taken care of or need it faster, I
> can take care of it.
The history is needed within a few days if we go with the changelog
generation step; but is NOT in the critical path for developers being
able to carry on committing.

> The most recent conversion is at:
> https://github.com/gentoo/gentoo-gitmig-20150614
Can you publish the container you mention, rather than just the results?

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-scm] Git Migration: launch plan & schedule (2015/Aug/08-09)

[Rich Freeman]

On Thu, Jul 2, 2015 at 8:51 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Thu, Jul 02, 2015 at 06:05:10PM -0400, Rich Freeman wrote:
>> On Thu, Jul 2, 2015 at 5:39 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
>> > 2015/08/08 15:00 UTC - Freeze
>> > 2015/08/11           - History repo available to graft
>> I'm not sure if you already had plans for creating the history, but
>> I've generally been able to turn them around in 12 hours without
>> really trying too hard.
> I expect verification of the conversion to take longer than the runs,
> and that we're going to find we miss something at least once, so it'll
> be 2 or 3 runs before we get output we're happy with.

I'm sure we'll be doing 10-20 runs over the subsequent 3 years before
we really get it "perfect."  :)  I've never had an issue with doing a
migration, but if you look closely enough I'm sure you'll find
something that could be done better, especially with the commit
merging.  In particular you'll probably almost never get a tree in the
past where all the Manfests are valid.  I don't really see that as
being important, however.

>
>> I have it all running in a container/chroot
>> where I mount the cvsroot so I could probably get it running on EC2 or
>> elsewhere easily enough if we're in a real hurry.
> Can you publish the container? Letting other people be able to replicate
> the result here would be a benefit, and we can easily shuffle it as
> well.

I'll take a look at it and see if I can clean out anything sensitive.
I have published all my conversion sources on github:
https://github.com/gentoo/git-migration-scripts-rich0

I'll publish a tarball of the container when I can make sure it
doesn't contain any keys/etc.  I do believe that you need a patch for
cvs2svn to get it to use the old date format.

>
>> If I had a ton of RAM and store everything in tmpfs then I suspect
>> that it could be done faster -
> How much RAM & CPU have you been running it with? I was planning on
> 64GB+ tmpfs and 12+ dedicated cores.

I've been running it on 16GB+4 amd64 cores (ie nothing all that
fancy).  I do the output to tmpfs and use a squashfs cvsroot as the
input typically, but that isn't ideal in a hurry since creating a
squashfs takes time (the goal was to cut down on disk IO).  The actual
migration uses very little RAM - it is just whatever you stick in
tmpfs.

>> With git replace we don't need the history to go live, and this also
>> means that if we find a mistake in the history we can fix it and issue
>> a new history.  So, there isn't really a rush.  But, if you don't
>> already have the history conversion taken care of or need it faster, I
>> can take care of it.
> The history is needed within a few days if we go with the changelog
> generation step; but is NOT in the critical path for developers being
> able to carry on committing.
>

I'll go ahead and try to give you what I have.  It sounds like you
have it under control, but if you need me to do anything just let me
know.  I'll see about getting that container packaged up in case it is
useful.  That was why I containerized it in the first place.  I
typically run it in a network namespace using nspawn for convenience,
but I'm sure you could easily just chroot into the thing after
bind-mounting in whatever you need.

--
Rich

[gentoo-scm] Re: [gentoo-dev] Git Migration: launch plan & schedule (2015/Aug/08-09)

[Robin H. Johnson]

On Thu, Jul 02, 2015 at 09:39:52PM +0000, Robin H. Johnson wrote:
> 2015/08/08 15:00 UTC - Freeze
> 2015/08/08 19:00 UTC - Git commits open for developers
> 2015/08/09 01:00 UTC - Rsync live again (with lagged changelog)
> 2015/08/11           - History repo available to graft
> 2015/08/12           - rsync mirrors carry up-to-date changelogs again
> 
> I've allocated time for an 8 hour freeze, but hope to be completed much
> sooner than that.
Starting late due to $reasons, freeze is now at 18:00 UTC (14 minutes
from now).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[gentoo-scm] Git Migration: go-live!

[Robin H. Johnson]

On Sat, Aug 08, 2015 at 05:47:14PM +0000, Robin H. Johnson wrote:
> On Thu, Jul 02, 2015 at 09:39:52PM +0000, Robin H. Johnson wrote:
> > 2015/08/08 15:00 UTC - Freeze
> > 2015/08/08 19:00 UTC - Git commits open for developers
This is going live in a few minutes. There was a lot of delays and snags
that were hit. QA has a lot of reviewing to do of in-tree patches with
long-standing CVS keyword damage. gkeys is also not sufficiently baked,
so we're using some scripting for now instead [1].

The new setup DOES enforce that commits AND pushes are signed.

I'm only 90% sure that everything works, but I've spent almost the
entire day on it, and there's more to go tomorrow.

Other old CVS repos are still closed for the moment, they will re-open
tomorrow.

> > 2015/08/09 01:00 UTC - Rsync live again (with lagged changelog)
> > 2015/08/11           - History repo available to graft
> > 2015/08/12           - rsync mirrors carry up-to-date changelogs again
These parts are still pending.

Quick instructions:
Set PORTAGE_GPG_KEY="0xLONG-GPG-KEY" in your make.conf
$ git config user.signingkey 0xLONG-GPG-KEY
$ git clone git+ssh://git@git.gentoo.org/repo/gentoo.git
$ vim ...
$ repoman commit -m '...' [2] 
$ git push --signed

(some time later, when you have local unpushed commits you want to
rebase instead of merging)
$ git pull --rebase -S
$ vim ...
$ repoman commit -m '...'
$ git push --signed

(some time later, when you have a local branch you want to merge)
$ git merge -S some-branch
$ git push --signed

[1]
The keys as they are in LDAP right now have been used. If you need to
change your key, please ping infra as well, so I can update the
temporary setup.
$ ldapsearch 'gentooStatus=active' gpgfingerprint -Z -LLL \
	|grep gpgfingerprint |cut -d: -f2- |tr -d ' '  \
	|grep -v 'undefined'  | xargs gpg --recv 

[2]
If you commit directly with "git commit" you MUST pass -S (and ideally
-s).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[gentoo-scm] Re: [gentoo-core] [gentoo-dev] Git Migration: go-live!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1052 bytes --]

On 09 Aug 2015 11:31, Marc Schiffbauer wrote:
> * Michael Weber schrieb am 09.08.15 um 11:00 Uhr:
> > On 08/09/2015 07:36 AM, Robin H. Johnson wrote:
> > > I'm only 90% sure that everything works, but I've spent almost the 
> > > entire day on it, and there's more to go tomorrow.
> > Thanks a lot!
> > 
> > use case: my cvs tree had uncommitted ebuild work (yes, you caught me
> > actually doing something).
> > now `cvs diff` no longer works, how can i track down my local changes?
> > besides diffing against git tree, brain memory aka shell history and
> > find -newer?
> 
> I'd say: 
> 
> - tar your *.ebuild and files/* stuff away
> - then "git clone" the new git repo.
> - untar your files in the new git repo
> - use "git diff"

there will be a ton of cvs keyword noise in there though.  need to run
a sed on the files to clear it out.

it also will include noise where your local checkout was behind the latest
tree, so it'll only really work if you ran `cvs up` in the whole tree just
before it was shutdown.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-scm] Re: [gentoo-dev] Git Migration: go-live!

[Robin H. Johnson]

On Sun, Aug 09, 2015 at 01:16:19PM +0300, Andrew Savchenko wrote:
> > Out of curiosity, is it impossible to have a read only CVS server with
> > the state at the time of the freeze?
> Seconded here. Read-only CVS should not consume much resources, but
> will facilitate migration.
Read-only access to gentoo-x86 is restored (and write to the other CVS
repos).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[gentoo-scm] Re: [gentoo-core] Git Migration: go-live!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 423 bytes --]

On 09 Aug 2015 12:18, Agostino Sarubbo wrote:
> 1) The git workflow [1] does not mention to add PORTAGE_GPG_KEY to make.conf 
> and neither to use the long gpg key. Does the short create a problem?

it does not

> 2) The git workflow [1] says to do git config --local commit.gpgsign 1 - Do we 
> need to use git push --signed if was already in the config ?

yes.  commit & push are independent operations.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-scm] Re: [gentoo-dev] Git Migration: go-live!

[Alexey Shvetsov]

Hi all!

Current repoman complains about headers in ebuilds

>>> Creating Manifest for /home/alexxy/Gentoo/gentoo/sys-cluster/open-mx
   ebuild.badheader              1
    sys-cluster/open-mx/open-mx-1.5.4.ebuild: Malformed CVS Header on 
line: 3

So may be its better to drop $Id: $ completely?

Robin H. Johnson писал 09-08-2015 08:36:
> On Sat, Aug 08, 2015 at 05:47:14PM +0000, Robin H. Johnson wrote:
>> On Thu, Jul 02, 2015 at 09:39:52PM +0000, Robin H. Johnson wrote:
>> > 2015/08/08 15:00 UTC - Freeze
>> > 2015/08/08 19:00 UTC - Git commits open for developers
> This is going live in a few minutes. There was a lot of delays and 
> snags
> that were hit. QA has a lot of reviewing to do of in-tree patches with
> long-standing CVS keyword damage. gkeys is also not sufficiently baked,
> so we're using some scripting for now instead [1].
> 
> The new setup DOES enforce that commits AND pushes are signed.
> 
> I'm only 90% sure that everything works, but I've spent almost the
> entire day on it, and there's more to go tomorrow.
> 
> Other old CVS repos are still closed for the moment, they will re-open
> tomorrow.
> 
>> > 2015/08/09 01:00 UTC - Rsync live again (with lagged changelog)
>> > 2015/08/11           - History repo available to graft
>> > 2015/08/12           - rsync mirrors carry up-to-date changelogs again
> These parts are still pending.
> 
> Quick instructions:
> Set PORTAGE_GPG_KEY="0xLONG-GPG-KEY" in your make.conf
> $ git config user.signingkey 0xLONG-GPG-KEY
> $ git clone git+ssh://git@git.gentoo.org/repo/gentoo.git
> $ vim ...
> $ repoman commit -m '...' [2]
> $ git push --signed
> 
> (some time later, when you have local unpushed commits you want to
> rebase instead of merging)
> $ git pull --rebase -S
> $ vim ...
> $ repoman commit -m '...'
> $ git push --signed
> 
> (some time later, when you have a local branch you want to merge)
> $ git merge -S some-branch
> $ git push --signed
> 
> [1]
> The keys as they are in LDAP right now have been used. If you need to
> change your key, please ping infra as well, so I can update the
> temporary setup.
> $ ldapsearch 'gentooStatus=active' gpgfingerprint -Z -LLL \
> 	|grep gpgfingerprint |cut -d: -f2- |tr -d ' '  \
> 	|grep -v 'undefined'  | xargs gpg --recv
> 
> [2]
> If you commit directly with "git commit" you MUST pass -S (and ideally
> -s).

-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Best Regards,
Alexey 'Alexxy' Shvetsov, PhD
Department of Molecular and Radiation Biophysics
FSBI Petersburg Nuclear Physics Institute, NRC Kurchatov Institute,
Leningrad region, Gatchina, Russia
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

Re: [gentoo-scm] Re: [gentoo-dev] Git Migration: go-live!

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

On 09 Aug 2015 14:54, Alexey Shvetsov wrote:
> Hi all!

please don't top post

> Current repoman complains about headers in ebuilds
> 
> >>> Creating Manifest for /home/alexxy/Gentoo/gentoo/sys-cluster/open-mx
>    ebuild.badheader              1
>     sys-cluster/open-mx/open-mx-1.5.4.ebuild: Malformed CVS Header on 
> line: 3
> 
> So may be its better to drop $Id: $ completely?

it should look like:
# $Id$

but even then, yes, we should just trim the line
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Git Migration: go-live!

[Rich Freeman]

On Sun, Aug 9, 2015 at 8:43 AM, Mike Frysinger <vapier@gentoo.org> wrote:
> On 09 Aug 2015 14:54, Alexey Shvetsov wrote:
>> Hi all!
>
> please don't top post
>
>> Current repoman complains about headers in ebuilds
>>
>> >>> Creating Manifest for /home/alexxy/Gentoo/gentoo/sys-cluster/open-mx
>>    ebuild.badheader              1
>>     sys-cluster/open-mx/open-mx-1.5.4.ebuild: Malformed CVS Header on
>> line: 3
>>
>> So may be its better to drop $Id: $ completely?
>
> it should look like:
> # $Id$
>

We shouldn't do anything (besides adapting repoman to the $Id$ format)
without some discussion first.  However...

Do we really need to have ANY keywords in our files?  Keyword
expansion was just a PITA all-around with cvs.  Do we really need to
embed them in our git files?

At least git doesn't expand them in the actual repository.

I'm not really a big fan of any kind of in-band signaling.  I guess
the one advantage of this is that if you find a file OUTSIDE of a
repository you know where it came from, but how important is that?

-- 
Rich

Re: [gentoo-scm] Re: [gentoo-dev] Git Migration: go-live!

[Robin H. Johnson]

On Sun, Aug 09, 2015 at 09:10:30AM -0400, Rich Freeman wrote:
> We shouldn't do anything (besides adapting repoman to the $Id$ format)
> without some discussion first.  However...
> 
> Do we really need to have ANY keywords in our files?  Keyword
> expansion was just a PITA all-around with cvs.  Do we really need to
> embed them in our git files?
> 
> At least git doesn't expand them in the actual repository.
> 
> I'm not really a big fan of any kind of in-band signaling.  I guess
> the one advantage of this is that if you find a file OUTSIDE of a
> repository you know where it came from, but how important is that?
The expansion is ONLY for the rsync export, so that we can identify the
version of the file on a user's system.

When previously discussed, that was a strongly requested item, as
developers DO sometimes ask users to please test rev 1.xyz of an ebuild.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85