[gentoo-scm] Progress summary, 2009/06/01

[gentoo-scm] Progress summary, 2009/06/01

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 3874 bytes --]

Update on this TODO list, and extension for new items.
I've deliberately broken the threading so that more people read it. This
was a response to the mail with the subject of "Converting a recent CVS
copy - Item 2: statistics"

Executive summary:
- We've gone from 18.5 hours to 9 hours, all in a single portion of the
  conversion, thanks to help from upstream. There's lots more room for
  improvement.
- C (for git) and Python (for cvs2svn) coders very welcome to challenge
  the problems.
- Testers wanted 
- Actually forming this up to a project with a team is probably due
  soon. I've had interest/direct offers of help from: WilliamH, Calchan,
  Betelgeuse over the last week.
  If you want something, pick it from this mail, and try to flesh it out
  on the list with me while you work on it.

New TODO items:
- Finish new hooks for git: 
  upload-pack.c - get_common_commits:
  right before the 'return 0', check for and launch a new hook, passing
  the have/want headers via stdin, and use the return code (AND the
  stderr) to see if we should halt.
- Review commit signing
  - pclouds (a former Gentoo dev) contributed this prototype:
	http://thread.gmane.org/gmane.comp.version-control.git/115562/focus=118788
  - I'm not entirely convinced the above is right, as the commit message
	seems to end up unsigned.
  - Wait for the commit-notes patches onto upstream Git?
- Test git-cvsserver usage.
  - Both remote and local modes.
  - mips and narrow checkouts may want this.

On Tue, Apr 14, 2009 at 01:33:24AM -0700, Robin H. Johnson wrote:
> TODO: 
> - Could somebody with Python-foo please look hard at cvs2svn with an eye to
>   making it multi-threaded?
>   - Focus on pass1 and pass9.
>   - pass8 maybe as well, but I think it will be harder by design.
- pass1 optimization was completed 20 minutes ago by mhagger (one of the
  cvs2svn upstream lead developers). From an old time of 36204 seconds,
  it's now only 1598 seconds (and we only used 2 CPU cores so far, we
  have 6 more for later). 22x speedup :-).
- mhagger from upstream needed hardware to test on, so I hooked him up
  with access to the experimental conversion box.
- TODO: pass9/pass8 remain.
- Need to validate output of new mode against the previous mode.

> - We need incremental conversion stuff badly.
Incremental may become practically unneeded if we can get the conversion
time under 2 hours.

> - I had to use the RCSRevisionReader, as InternalRevisionReader seemed to be
>   broken. Would make pass1 faster as well.
This is now completely unneeded. The pass1 solution integrated the
rcsparse code into the new generate_blobs.py.

> - Should probably ignore the '.frozen' files.
No support to exclude files presently.
TODO: Implement inside def _generate_cvs_files

> - Review RCS state of ALL ,v files. There are a few non-dead files in Attic.
Thanks to William Hubbs (williamh) for his review. I've got a few more
items to process from this review.

> - Maybe trim out the Manifest/digest contents during the conversion, leaving
>   only DIST lines? 
>   Con: _WOULD_ break old GPG signatures.
>   Pro: probably help size a lot.
Additional con:
- I think it will massively slows down the conversion right now.
- Nothing more than a bad prototype I did. I'd like somebody else to
  attack the problem without having seen my prototype, rather than be
  infected with my bad ideas in it.

Upstream misc:
- Subtree checkouts
  - Is progressing upstream. Now known as checkout modes: narrow,
	sparse, shallow
	= narrow: some directory that is not the root.
	= sparse: a subset of files in a directory.
	= shallow: subset of recent history.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

[gentoo-scm] gpg signing of commits, was: Progress summary, 2009/06/01

[Robert Buchholz]

[-- Attachment #1: Type: text/plain, Size: 1839 bytes --]

Hey Robin,

thanks for the summary.


On Tuesday 02 June 2009, Robin H. Johnson wrote:
> - Review commit signing
>   - pclouds (a former Gentoo dev) contributed this prototype:
> 	http://thread.gmane.org/gmane.comp.version-control.git/115562/focus=
>118788 - I'm not entirely convinced the above is right, as the commit
> message seems to end up unsigned.

I was wondering why we need GPG signing of commits at all. I was 
thinking about the following two arguments:

0. Intro
git stores the SHA1 hashes of objects and one can check for errors in 
the transmission or on the disk. This makes the (unsigned) Manifest 
parts unnecessary. Commit signing is the equivalent of Manifest file 
signing we have right now.

1. It's not needed for tree signing
The tree signing GLEP does not require signing of either commits or 
Manifests. It relies on the main infra repository is not being 
compromised.

2. It is not well designed (cryptographically)
OpenGPG allows the usage of a set of cryptographic hash function to sign 
a document. This allows people to switch to a different function once 
attacks against one algorithm become known. This has been recently seen 
with SHA-1: http://www.debian-administration.org/users/dkg/weblog/48

The git signing, however, relies on the collision resistance of SHA-1 as 
that algorithm is used to identify objects in the repository. We cannot 
migrate away from it easily. This has been discussed upstream at length 
and Linus pointed out that 'the "signed tags" security does depend on 
the hashes being cryptographically strong.':
http://thread.gmane.org/gmane.comp.version-control.git/26106/focus=26125


What if we just drop the commit signing and coresponding hooks, and 
focus on the tree signing and push logging even though we use gitosis ?


Robert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-scm] gpg signing of commits, was: Progress summary, 2009/06/01

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 3136 bytes --]

On Fri, Jun 05, 2009 at 02:59:18PM +0200, Robert Buchholz wrote:
> On Tuesday 02 June 2009, Robin H. Johnson wrote:
> > - Review commit signing
> >   - pclouds (a former Gentoo dev) contributed this prototype:
> > 	http://thread.gmane.org/gmane.comp.version-control.git/115562/focus=
> >118788 - I'm not entirely convinced the above is right, as the commit
> > message seems to end up unsigned.
> I was wondering why we need GPG signing of commits at all. I was 
> thinking about the following two arguments:
The commit signing I'm after is so that we can be absolutely certain who
introduced a given commit to the tree (who committed, AND who pushed the
merge/fast-forward), and have that information distributed inside the
tree.

This is related to the push logging issue, if you've seen the
discussions on tracking who committed vs. who pushed.

> 0. Intro
> git stores the SHA1 hashes of objects and one can check for errors in 
> the transmission or on the disk. This makes the (unsigned) Manifest 
> parts unnecessary. Commit signing is the equivalent of Manifest file 
> signing we have right now.
Yes, it's the replacement for the existing Manifest signing. The point
of that is proof of origin from developer BACK to infra.


> 1. It's not needed for tree signing
> The tree signing GLEP does not require signing of either commits or 
> Manifests. It relies on the main infra repository is not being 
> compromised.
That's the external distribution portion of tree signing: infra -> world
It's unrelated to the problem at hand within Git.

> 2. It is not well designed (cryptographically)
> OpenGPG allows the usage of a set of cryptographic hash function to sign 
> a document. This allows people to switch to a different function once 
> attacks against one algorithm become known. This has been recently seen 
> with SHA-1: http://www.debian-administration.org/users/dkg/weblog/48
I only stated that we need to offer GPG signing of commits. I did NOT
specify the content of commits, other than noting that the commit
message and the content needs to be signed together.

> The git signing, however, relies on the collision resistance of SHA-1 as 
> that algorithm is used to identify objects in the repository. We cannot 
> migrate away from it easily. This has been discussed upstream at length 
> and Linus pointed out that 'the "signed tags" security does depend on 
> the hashes being cryptographically strong.':
> http://thread.gmane.org/gmane.comp.version-control.git/26106/focus=26125
The collision is going to come along anyway. 

Resigning would have to be done regardless of what we sign in Git.
Not sure if you followed more recent discussions than one in 2006.
The entire Git foodchain will suffer when it comes time to migrate away
from SHA-2. Presently discussions of it imply that it's to be done
probably as a versioned change, after the NIST hash competition comes up
with a viable answer.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-scm] Progress summary, 2009/06/01

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On 16:45 Mon 01 Jun     , Robin H. Johnson wrote:
> - Actually forming this up to a project with a team is probably due
>   soon. I've had interest/direct offers of help from: WilliamH, Calchan,
>   Betelgeuse over the last week.

I was thinking of starting a Gentoo R&D project for things like this 
that aren't going to live forever but will be "temporary" things. There 
are lots of semi-major efforts like this and it would be nice to give 
them (and the devs working on them) a home.

Also I think it sounds cool to be part of Gentoo R&D. =)

Any thoughts?

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-scm] Progress summary, 2009/06/01

[Nirbheek Chauhan]

On Mon, Jun 8, 2009 at 3:48 AM, Donnie Berkholz<dberkholz@gentoo.org> wrote:
> I was thinking of starting a Gentoo R&D project for things like this
> that aren't going to live forever but will be "temporary" things. There
> are lots of semi-major efforts like this and it would be nice to give
> them (and the devs working on them) a home.
>
> Also I think it sounds cool to be part of Gentoo R&D. =)
>
> Any thoughts?

++

Having a project advertises that help is appreciated, and helps in
organisation as well. Teams and Projects are always good!

-- 
~Nirbheek Chauhan

Re: [gentoo-scm] gpg signing of commits, was: Progress summary, 2009/06/01

[Robert Buchholz]

[-- Attachment #1: Type: text/plain, Size: 2239 bytes --]

On Friday 05 June 2009, Robin H. Johnson wrote:
> On Fri, Jun 05, 2009 at 02:59:18PM +0200, Robert Buchholz wrote:
...
> > 2. It is not well designed (cryptographically)
> > OpenGPG allows the usage of a set of cryptographic hash function to
> > sign a document. This allows people to switch to a different
> > function once attacks against one algorithm become known. This has
> > been recently seen with SHA-1:
> > http://www.debian-administration.org/users/dkg/weblog/48
>
> I only stated that we need to offer GPG signing of commits. I did NOT
> specify the content of commits, other than noting that the commit
> message and the content needs to be signed together.

I don't think I understood what you meant to say, sorry. As I understand 
the current proposal, it would be over the SHA-1 of the objects, the 
parent and the commit message.


> > The git signing, however, relies on the collision resistance of
> > SHA-1 as that algorithm is used to identify objects in the
> > repository. We cannot migrate away from it easily. This has been
> > discussed upstream at length and Linus pointed out that 'the
> > "signed tags" security does depend on the hashes being
> > cryptographically strong.':
> > http://thread.gmane.org/gmane.comp.version-control.git/26106/focus=
> >26125
>
> The collision is going to come along anyway.
>
> Resigning would have to be done regardless of what we sign in Git.
> Not sure if you followed more recent discussions than one in 2006.
> The entire Git foodchain will suffer when it comes time to migrate
> away from SHA-2. Presently discussions of it imply that it's to be
> done probably as a versioned change, after the NIST hash competition
> comes up with a viable answer.

I have not seen any statements that would indicate they intended to 
switch ever, do you have a reference? I only found discussions as 
recent as April 2008. If it will be possible to use one (at that time) 
stronger hash function, my argument is defeated. I wanted to point out 
that right now they only support one function that is increasingly 
weakened, and I have the feeling upstream will only act once collisions 
become practical, which is -IMHO- too late.


Robert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-scm] gpg signing of commits, was: Progress summary, 2009/06/01

[Robin H. Johnson]

On Tue, Jun 09, 2009 at 03:50:35AM +0200, Robert Buchholz wrote:
> > I only stated that we need to offer GPG signing of commits. I did NOT
> > specify the content of commits, other than noting that the commit
> > message and the content needs to be signed together.
> I don't think I understood what you meant to say, sorry. As I understand 
> the current proposal, it would be over the SHA-1 of the objects, the 
> parent and the commit message.
That's what I'd like it to be over yes.

> I have not seen any statements that would indicate they intended to 
> switch ever, do you have a reference?
I'll dig around for it, it was just in reading the list directly.
There is minimal value in switching to even SHA-512 right now for Git.
The SHA-1 attacks have been extended to the entire SHA family.

> I only found discussions as recent as April 2008. If it will be
> possible to use one (at that time) stronger hash function, my argument
> is defeated. I wanted to point out that right now they only support
> one function that is increasingly weakened, and I have the feeling
> upstream will only act once collisions become practical, which is
> -IMHO- too late.
We're at their mercy already. If you can attack SHA1 and choose the hash
of your malicious content given the only restriction as the file size,
you can insert a file anywhere in the repository already.

All of the attacks thusfar have been chosen plaintext and preimage
attacks. Current state of the art for SHA-1 is 2^52, as announced here:
http://eurocrypt2009rump.cr.yp.to/837a0a8086fa6ca714249409ddfae43d.pdf
(and I think SHA-512 is around 2^140, weaker than even bruteforce
against SHA-1).

I'd be far more concerned about a user introducing a chosen plaintext
that he already has the attack against.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[gentoo-scm] New hooks for git, was: Progress summary, 2009/06/01

[Arun Raghavan]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On Mon, 2009-06-01 at 16:45 -0700, Robin H. Johnson wrote:
> - Finish new hooks for git: 
>   upload-pack.c - get_common_commits:
>   right before the 'return 0', check for and launch a new hook,
> passing
>   the have/want headers via stdin, and use the return code (AND the
>   stderr) to see if we should halt.

Could you expand on this? Doesn't sound too complex - if I understand
this,  'have' is the set of heads of branches that the server has and
'want' is the set of branches that the client figures it needs to update
(simplifying a little from [1]).

That said, when do we want to terminate the upload?

Best,
Arun

[1] http://git-scm.com/gitserver.txt

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-scm] New hooks for git, was: Progress summary, 2009/06/01

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1646 bytes --]

On Wed, Jun 10, 2009 at 01:12:47AM +0530, Arun Raghavan wrote:
> On Mon, 2009-06-01 at 16:45 -0700, Robin H. Johnson wrote:
> > - Finish new hooks for git: 
> >   upload-pack.c - get_common_commits:
> >   right before the 'return 0', check for and launch a new hook,
> > passing
> >   the have/want headers via stdin, and use the return code (AND the
> >   stderr) to see if we should halt.
> Could you expand on this? Doesn't sound too complex - if I understand
> this,  'have' is the set of heads of branches that the server has and
> 'want' is the set of branches that the client figures it needs to update
> (simplifying a little from [1]).
- You'll have to replicate something like the run_receive_hook()
  function (replace the body of the second for loop), maybe call it
  run_upload_hook().
- Look at where upload-pack.c returns ACK/NAK presently.
- Add a means to run the hook function there. Be sure to pass stderr
  back!
- See that get_common_commits sends stderr to the client (I think it
  should handle this perfectly already).

> That said, when do we want to terminate the upload?
Basically, I see something like this (I've skipped the length parts of
the mesage, but see section 5.3 in the RFC):
C: want $ID\0...\n
C: ....
C: done\n
S: NAK\n
S: \002Please visit http://.... and use the bundle for the initial
download\n
S: (disconnect)

The client sent no 'have' blocks (either at all, or sufficiently new),
so we tell them to GTFO.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

[gentoo-scm] Progress summary, 2009/08/27

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

Progress since the last report has been practically non-existent.
I haven't had any time of my own to touch it, but I haven't seen anybody
else step up either.

pre-upload-hook:
----------------
ford_prefect volunteered, zero progress seen.
Upstream has a post-upload-pack hook on the mailing list right now, that
would be very trivial to extract this from.

cvs2svn performance:
--------------------
No changes known. No work done it at all that I'm aware of.

Commit signing:
---------------
No direct progress seen upstream. commit-notes upstream patch still not
merged either, but maturing still. rbu, would you like to pursue this?

sparse trees:
-------------
Upstream patch still championed by pclouds, has major progress. Possible
to merge in 1.6.5 series I think.

Testing git-cvsserver:
----------------------
Bueller? Bueller?

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]