[gentoo-scm] Progress on cvs->git migration

[gentoo-scm] Progress on cvs->git migration

[Alexey Shvetsov]

Hi all!

Is there any progress since last update on cvs->git migration?


-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Petersburg Nuclear Physics Institute, Russia
Department of Molecular and Radiation Biophysics
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

[gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Mon, Aug 22, 2011 at 08:28:29PM +0400, Alexey Shvetsov wrote:
> Is there any progress since last update on cvs->git migration?
The list for it is gentoo-scm, not gentoo-dev.

There's an experimental tree that mirrors CVS:
http://git-exp.overlays.gentoo.org/gitweb/?p=exp/gentoo-x86.git;a=summary

Unresolved items:
- commit signing
- thin Manifests
- merge policies

In terms of the tree size question, we were going to produce a
zero-history point at the switchover time, and offer a graftable pack
with all prior history. This got the initial packfile down under 80MiB
(with the historical packfile being ~900MiB).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

On 19:28 Mon 22 Aug     , Robin H. Johnson wrote:
> In terms of the tree size question, we were going to produce a
> zero-history point at the switchover time, and offer a graftable pack
> with all prior history. This got the initial packfile down under 80MiB
> (with the historical packfile being ~900MiB).

On this note, how in the world do you do that? There appears to be 
plenty of documentation on splicing things back together but none on how 
to split them in the first place.

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer
Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Mon, Aug 22, 2011 at 09:38:20PM -0500, Donnie Berkholz wrote:
> On 19:28 Mon 22 Aug     , Robin H. Johnson wrote:
> > In terms of the tree size question, we were going to produce a
> > zero-history point at the switchover time, and offer a graftable pack
> > with all prior history. This got the initial packfile down under 80MiB
> > (with the historical packfile being ~900MiB).
> On this note, how in the world do you do that? There appears to be 
> plenty of documentation on splicing things back together but none on how 
> to split them in the first place.
It's in the first post on thread 'repo layout & graft / split-history',
November 1st, 2010 (git fast-export).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 746 bytes --]

On Mon, 22 Aug 2011 21:38:20 -0500
Donnie Berkholz <dberkholz@gentoo.org> wrote:

> On 19:28 Mon 22 Aug     , Robin H. Johnson wrote:
> > In terms of the tree size question, we were going to produce a
> > zero-history point at the switchover time, and offer a graftable
> > pack with all prior history. This got the initial packfile down
> > under 80MiB (with the historical packfile being ~900MiB).
> 
> On this note, how in the world do you do that? There appears to be 
> plenty of documentation on splicing things back together but none on
> how to split them in the first place.

git clone --bare foo historical.git
cd foo
rm -rf .git
git init
git add -A
git commit -m 'Split point.'

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Tue, Aug 23, 2011 at 08:46:28AM +0200, Michał Górny wrote:
> > On this note, how in the world do you do that? There appears to be 
> > plenty of documentation on splicing things back together but none on
> > how to split them in the first place.
> git clone --bare foo historical.git
> cd foo
> rm -rf .git
> git init
> git add -A
> git commit -m 'Split point.'
That introduces an artificial commit at the split point, instead of
using an existing commit like my fast-export solution.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Alexey Shvetsov]

On Mon, 22 Aug 2011 19:28:57 +0000, Robin H. Johnson wrote:
> On Mon, Aug 22, 2011 at 08:28:29PM +0400, Alexey Shvetsov wrote:
>> Is there any progress since last update on cvs->git migration?
> The list for it is gentoo-scm, not gentoo-dev.
>
> There's an experimental tree that mirrors CVS:
> 
> http://git-exp.overlays.gentoo.org/gitweb/?p=exp/gentoo-x86.git;a=summary
>
> Unresolved items:
> - commit signing
> - thin Manifests
> - merge policies
>
> In terms of the tree size question, we were going to produce a
> zero-history point at the switchover time, and offer a graftable pack
> with all prior history. This got the initial packfile down under 
> 80MiB
> (with the historical packfile being ~900MiB).

Ok. What is problems with thin Manifests (some kind of this already 
implented in funtoo) and commit signing (this means gpg signing or 
something else?).
-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Petersburg Nuclear Physics Institute, Russia
Department of Molecular and Radiation Biophysics
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Zac Medico]

On 08/23/2011 07:02 AM, Alexey Shvetsov wrote:
> Ok. What is problems with thin Manifests (some kind of this already
> implented in funtoo)

This is really easy to do. Like the manifest1 -> manifest2 migration,
we'll need some kind of repository marker which indicates the manifest
format. For example, we could use an entry in metadata/layout.conf for
this purpose (as I've already suggested in bug #333691).

> and commit signing (this means gpg signing or something else?).

I guess the existing manifest signing technique is likely to trigger
merge conflicts in the manifests. I suppose we could use another marker,
similar to the thin manifest marker, to indicate that the existing
manifest signing technique should not be used in the git tree.
-- 
Thanks,
Zac

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Alexey Shvetsov]

On Tue, 23 Aug 2011 07:57:24 -0700, Zac Medico wrote:
> On 08/23/2011 07:02 AM, Alexey Shvetsov wrote:
>> Ok. What is problems with thin Manifests (some kind of this already
>> implented in funtoo)
>
> This is really easy to do. Like the manifest1 -> manifest2 migration,
> we'll need some kind of repository marker which indicates the 
> manifest
> format. For example, we could use an entry in metadata/layout.conf 
> for
> this purpose (as I've already suggested in bug #333691).
>
>> and commit signing (this means gpg signing or something else?).
>
> I guess the existing manifest signing technique is likely to trigger
> merge conflicts in the manifests. I suppose we could use another 
> marker,
> similar to the thin manifest marker, to indicate that the existing
> manifest signing technique should not be used in the git tree.

Yep signing git commits with gpg should avoid conflicts. May we can use 
something like this [1]
[1] 
http://weierophinney.net/matthew/archives/236-GPG-signing-Git-Commits.html
-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Petersburg Nuclear Physics Institute, Russia
Department of Molecular and Radiation Biophysics
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

On Tue, 23 Aug 2011 19:22:06 +0400
Alexey Shvetsov <alexxy@gentoo.org> wrote:

> On Tue, 23 Aug 2011 07:57:24 -0700, Zac Medico wrote:
> > On 08/23/2011 07:02 AM, Alexey Shvetsov wrote:
> >> Ok. What is problems with thin Manifests (some kind of this already
> >> implented in funtoo)
> >
> > This is really easy to do. Like the manifest1 -> manifest2
> > migration, we'll need some kind of repository marker which
> > indicates the manifest
> > format. For example, we could use an entry in metadata/layout.conf 
> > for
> > this purpose (as I've already suggested in bug #333691).
> >
> >> and commit signing (this means gpg signing or something else?).
> >
> > I guess the existing manifest signing technique is likely to trigger
> > merge conflicts in the manifests. I suppose we could use another 
> > marker,
> > similar to the thin manifest marker, to indicate that the existing
> > manifest signing technique should not be used in the git tree.
> 
> Yep signing git commits with gpg should avoid conflicts. May we can
> use something like this [1]
> [1] 
> http://weierophinney.net/matthew/archives/236-GPG-signing-Git-Commits.html

Er, no. Signing commits != signing commit message text.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Piotr Jaroszyński]

On 23 August 2011 17:22, Alexey Shvetsov <alexxy@gentoo.org> wrote:
> On Tue, 23 Aug 2011 07:57:24 -0700, Zac Medico wrote:
>>
>> On 08/23/2011 07:02 AM, Alexey Shvetsov wrote:
>>>
>>> Ok. What is problems with thin Manifests (some kind of this already
>>> implented in funtoo)
>>
>> This is really easy to do. Like the manifest1 -> manifest2 migration,
>> we'll need some kind of repository marker which indicates the manifest
>> format. For example, we could use an entry in metadata/layout.conf for
>> this purpose (as I've already suggested in bug #333691).
>>
>>> and commit signing (this means gpg signing or something else?).
>>
>> I guess the existing manifest signing technique is likely to trigger
>> merge conflicts in the manifests. I suppose we could use another marker,
>> similar to the thin manifest marker, to indicate that the existing
>> manifest signing technique should not be used in the git tree.
>
> Yep signing git commits with gpg should avoid conflicts. May we can use
> something like this [1]
> [1]
> http://weierophinney.net/matthew/archives/236-GPG-signing-Git-Commits.html

After a quick look, it doesn't seem to add any security whatsoever -
it signs only the commit message.

-- 
Pozdrowienia
Piotr Jaroszyński

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Alexey Shvetsov]

Or may be something like this according to old disscussion

http://git.661346.n2.nabble.com/GPG-signing-for-git-commit-td2582986.html

On Tue, 23 Aug 2011 18:19:31 +0200, Michał Górny wrote:
> On Tue, 23 Aug 2011 19:22:06 +0400
> Alexey Shvetsov <alexxy@gentoo.org> wrote:
>
>> On Tue, 23 Aug 2011 07:57:24 -0700, Zac Medico wrote:
>> > On 08/23/2011 07:02 AM, Alexey Shvetsov wrote:
>> >> Ok. What is problems with thin Manifests (some kind of this 
>> already
>> >> implented in funtoo)
>> >
>> > This is really easy to do. Like the manifest1 -> manifest2
>> > migration, we'll need some kind of repository marker which
>> > indicates the manifest
>> > format. For example, we could use an entry in metadata/layout.conf
>> > for
>> > this purpose (as I've already suggested in bug #333691).
>> >
>> >> and commit signing (this means gpg signing or something else?).
>> >
>> > I guess the existing manifest signing technique is likely to 
>> trigger
>> > merge conflicts in the manifests. I suppose we could use another
>> > marker,
>> > similar to the thin manifest marker, to indicate that the existing
>> > manifest signing technique should not be used in the git tree.
>>
>> Yep signing git commits with gpg should avoid conflicts. May we can
>> use something like this [1]
>> [1]
>> 
>> http://weierophinney.net/matthew/archives/236-GPG-signing-Git-Commits.html
>
> Er, no. Signing commits != signing commit message text.

-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Petersburg Nuclear Physics Institute, Russia
Department of Molecular and Radiation Biophysics
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1404 bytes --]

On Tuesday, August 23, 2011 12:19:31 Michał Górny wrote:
> On Tue, 23 Aug 2011 19:22:06 +0400 Alexey Shvetsov wrote:
> > On Tue, 23 Aug 2011 07:57:24 -0700, Zac Medico wrote:
> > > On 08/23/2011 07:02 AM, Alexey Shvetsov wrote:
> > >> Ok. What is problems with thin Manifests (some kind of this already
> > >> implented in funtoo)
> > > 
> > > This is really easy to do. Like the manifest1 -> manifest2
> > > migration, we'll need some kind of repository marker which
> > > indicates the manifest
> > > format. For example, we could use an entry in metadata/layout.conf
> > > for
> > > this purpose (as I've already suggested in bug #333691).
> > > 
> > >> and commit signing (this means gpg signing or something else?).
> > > 
> > > I guess the existing manifest signing technique is likely to trigger
> > > merge conflicts in the manifests. I suppose we could use another
> > > marker,
> > > similar to the thin manifest marker, to indicate that the existing
> > > manifest signing technique should not be used in the git tree.
> > 
> > Yep signing git commits with gpg should avoid conflicts. May we can
> > use something like this [1]
> > [1]
> > http://weierophinney.net/matthew/archives/236-GPG-signing-Git-Commits.htm
> > l
> 
> Er, no. Signing commits != signing commit message text.

and people shouldnt confuse this guy's post with signed annotated tags
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 142 bytes --]

On Monday, August 22, 2011 15:28:57 Robin H. Johnson wrote:
> Unresolved items:

did we settle on the changelog autogeneration method ?
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Tue, Aug 23, 2011 at 03:33:29PM -0400, Mike Frysinger wrote:
> On Monday, August 22, 2011 15:28:57 Robin H. Johnson wrote:
> > Unresolved items:
> 
> did we settle on the changelog autogeneration method ?
No.

My personal opinion on it is that the ChangeLog isn't auto-generated
because it's NOT the same as the commit messages. The commit messages
are detailed technical info about a commit, while the ChangeLog is a
short version. 

If we want to enforce the changelog being the shortlog output, that's
doable as long as all devs actually use it.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 802 bytes --]

On Tuesday, August 23, 2011 17:26:30 Robin H. Johnson wrote:
> On Tue, Aug 23, 2011 at 03:33:29PM -0400, Mike Frysinger wrote:
> > On Monday, August 22, 2011 15:28:57 Robin H. Johnson wrote:
> > > Unresolved items:
> > did we settle on the changelog autogeneration method ?
> 
> No.

when i said "settle", i meant "logistics", not "are we doing it"

> My personal opinion on it is that the ChangeLog isn't auto-generated
> because it's NOT the same as the commit messages. The commit messages
> are detailed technical info about a commit, while the ChangeLog is a
> short version.

funny, as that's the exact reason i gave for omitting changelog entries in the 
first place.  but that ship has sailed, and last i looked, council approved 
changelog autogeneration in the tree.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Lance Albertson]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On 08/23/2011 02:26 PM, Robin H. Johnson wrote:
> On Tue, Aug 23, 2011 at 03:33:29PM -0400, Mike Frysinger wrote:
>> On Monday, August 22, 2011 15:28:57 Robin H. Johnson wrote:
>>> Unresolved items:
>>
>> did we settle on the changelog autogeneration method ?
> No.
> 
> My personal opinion on it is that the ChangeLog isn't auto-generated
> because it's NOT the same as the commit messages. The commit messages
> are detailed technical info about a commit, while the ChangeLog is a
> short version. 
> 
> If we want to enforce the changelog being the shortlog output, that's
> doable as long as all devs actually use it.

1+

I think using the shortlog output is the sane solution otherwise you're
just replicating what you do in the commit.

-- 
Lance Albertson
Systems Administrator / Architect                        Open Source Lab
Information Services                             Oregon State University


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Tue, Aug 23, 2011 at 06:20:35PM -0400, Mike Frysinger wrote:
> when i said "settle", i meant "logistics", not "are we doing it"
That's why the next paragraph after the one you quoted proposed the use
of shortlog.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 376 bytes --]

On 15:49 Tue 23 Aug     , Lance Albertson wrote:
> I think using the shortlog output is the sane solution otherwise you're
> just replicating what you do in the commit.

It's not replication if users continue to use rsync; they won't have 
commit info.

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer
Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 347 bytes --]

On Tuesday, August 23, 2011 19:12:48 Robin H. Johnson wrote:
> On Tue, Aug 23, 2011 at 06:20:35PM -0400, Mike Frysinger wrote:
> > when i said "settle", i meant "logistics", not "are we doing it"
> 
> That's why the next paragraph after the one you quoted proposed the use
> of shortlog.

so how is this *not* an unresolved issue ?
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Matt Turner]

On Tue, Aug 23, 2011 at 10:30 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> On 15:49 Tue 23 Aug     , Lance Albertson wrote:
>> I think using the shortlog output is the sane solution otherwise you're
>> just replicating what you do in the commit.
>
> It's not replication if users continue to use rsync; they won't have
> commit info.

Do we really want users to continue using rsync? Isn't git pull so
much faster? What's the downside of users using git directly?

Matt

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Fabian Groffen]

On 24-08-2011 00:44:57 -0400, Matt Turner wrote:
> On Tue, Aug 23, 2011 at 10:30 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> > On 15:49 Tue 23 Aug     , Lance Albertson wrote:
> >> I think using the shortlog output is the sane solution otherwise you're
> >> just replicating what you do in the commit.
> >
> > It's not replication if users continue to use rsync; they won't have
> > commit info.
> 
> Do we really want users to continue using rsync? Isn't git pull so
> much faster? What's the downside of users using git directly?

ehm, that you need git?  that you need to use git to get information
about changes?  that you need a whole new infrastructure of mirrors to
get it running (vs the rsync infrastructure)?  that you need at minimum
800MiB to be able to look at some history, iso. 286MiB as the rsync tree
is now?

Besides from that git doesn't even work on all platforms, but I can
imagine you don't care about that.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

On Wed, 24 Aug 2011 00:44:57 -0400
Matt Turner <mattst88@gentoo.org> wrote:

> On Tue, Aug 23, 2011 at 10:30 PM, Donnie Berkholz
> <dberkholz@gentoo.org> wrote:
> > On 15:49 Tue 23 Aug     , Lance Albertson wrote:
> >> I think using the shortlog output is the sane solution otherwise
> >> you're just replicating what you do in the commit.
> >
> > It's not replication if users continue to use rsync; they won't have
> > commit info.
> 
> Do we really want users to continue using rsync? Isn't git pull so
> much faster? What's the downside of users using git directly?

We still need to supply them with metadata cache, and we don't really
want to put that in the git tree. And a few other files are merged into
the tree as well (AFAIR herds.xml is an example); we could try some
kind of hybrid rsync+git but I guess not yet.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Wed, Aug 24, 2011 at 12:44:57AM -0400, Matt Turner wrote:
> On Tue, Aug 23, 2011 at 10:30 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> > On 15:49 Tue 23 Aug     , Lance Albertson wrote:
> >> I think using the shortlog output is the sane solution otherwise you're
> >> just replicating what you do in the commit.
> >
> > It's not replication if users continue to use rsync; they won't have
> > commit info.
> 
> Do we really want users to continue using rsync? Isn't git pull so
> much faster? What's the downside of users using git directly?
rsync is still being supported. There is NO size or bandwidth advantage
to the users having Git, quite the opposite in fact, the users would be
disadvantaged if we required them to use git.

Size: The size of an rsync tree will ALWAYS be smaller than _ANY_ VCS
checkout, because none of the VCS overhead will be present (no CVS/,
.git, etc).

Bandwidth: Along the same lines, rsync will always be able to use less
bandwidth than Git, because none of the intermediate commits need to be
transfered. This will be esp. evident as a user tree gets older (the
amount of mtime/checksum metadata scales linearly with the size of the
tree, not the age of the tree. The actual file content transfered scales
linearly with the age of the tree).

The only advantage to users with Git is for those where metadata
operations on their local disk is slower than the equivalent index
operations in git (mtime checks on cold cache causing lots of seek vs
being able to read a single file).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Wed, Aug 24, 2011 at 08:58:49AM +0200, Fabian Groffen wrote:
> > Do we really want users to continue using rsync? Isn't git pull so
> > much faster? What's the downside of users using git directly?
> ehm, that you need git?  that you need to use git to get information
> about changes?  that you need a whole new infrastructure of mirrors to
> get it running (vs the rsync infrastructure)?  that you need at minimum
> 800MiB to be able to look at some history, iso. 286MiB as the rsync tree
> is now?
That's not a valid size comparison, see my previous email.

> Besides from that git doesn't even work on all platforms, but I can
> imagine you don't care about that.
Very valid as well.


-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Tue, Aug 23, 2011 at 11:03:24PM -0400, Mike Frysinger wrote:
> On Tuesday, August 23, 2011 19:12:48 Robin H. Johnson wrote:
> > On Tue, Aug 23, 2011 at 06:20:35PM -0400, Mike Frysinger wrote:
> > > when i said "settle", i meant "logistics", not "are we doing it"
> > That's why the next paragraph after the one you quoted proposed the use
> > of shortlog.
> so how is this *not* an unresolved issue ?
The means of changelog generation needs to be driven by Council, since
they are the ones that are demanding changelogs for every tiny change.
I proposed shortlog as something that is simple and works for now.

Regardless of generated changelogs or them being manually written, the
only way that we can ensure that changelogs don't contain any reordering
is by either hand-merging them (then we might as well hand-write them),
or taking snapshots of shortlog changelogs (which can lead to dupes
after some merges).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Nirbheek Chauhan]

On Wed, Aug 24, 2011 at 12:28 PM, Fabian Groffen <grobian@gentoo.org> wrote:
> On 24-08-2011 00:44:57 -0400, Matt Turner wrote:
>> On Tue, Aug 23, 2011 at 10:30 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
>> > On 15:49 Tue 23 Aug     , Lance Albertson wrote:
>> >> I think using the shortlog output is the sane solution otherwise you're
>> >> just replicating what you do in the commit.
>> >
>> > It's not replication if users continue to use rsync; they won't have
>> > commit info.
>>
>> Do we really want users to continue using rsync? Isn't git pull so
>> much faster? What's the downside of users using git directly?
>
> ehm, that you need git?  that you need to use git to get information
> about changes?  that you need a whole new infrastructure of mirrors to
> get it running (vs the rsync infrastructure)?  that you need at minimum
> 800MiB to be able to look at some history, iso. 286MiB as the rsync tree
> is now?
>
> Besides from that git doesn't even work on all platforms, but I can
> imagine you don't care about that.
>

Actually, the major blocker as I understand it, is portage metadata
cache regeneration.


-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1361 bytes --]

On Wed, 24 Aug 2011 07:30:11 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> On Wed, Aug 24, 2011 at 12:44:57AM -0400, Matt Turner wrote:
> > On Tue, Aug 23, 2011 at 10:30 PM, Donnie Berkholz
> > <dberkholz@gentoo.org> wrote:
> > > On 15:49 Tue 23 Aug     , Lance Albertson wrote:
> > >> I think using the shortlog output is the sane solution otherwise
> > >> you're just replicating what you do in the commit.
> > >
> > > It's not replication if users continue to use rsync; they won't
> > > have commit info.
> > 
> > Do we really want users to continue using rsync? Isn't git pull so
> > much faster? What's the downside of users using git directly?
> Bandwidth: Along the same lines, rsync will always be able to use less
> bandwidth than Git, because none of the intermediate commits need to
> be transfered. This will be esp. evident as a user tree gets older
> (the amount of mtime/checksum metadata scales linearly with the size
> of the tree, not the age of the tree. The actual file content
> transfered scales linearly with the age of the tree).

I tend not to agree here. With very rare updates, this is true indeed.
But I guess that with daily or maybe even weekly updates, git should be
able to consume less bandwidth because it doesn't need to check all
unmodified files.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Fabian Groffen]

On 24-08-2011 07:35:05 +0000, Robin H. Johnson wrote:
> The means of changelog generation needs to be driven by Council, since
> they are the ones that are demanding changelogs for every tiny change.
> I proposed shortlog as something that is simple and works for now.
> 
> Regardless of generated changelogs or them being manually written, the
> only way that we can ensure that changelogs don't contain any reordering
> is by either hand-merging them (then we might as well hand-write them),
> or taking snapshots of shortlog changelogs (which can lead to dupes
> after some merges).

What do you mean by reordering?  From what I see from generating
ChangeLog files from both CVS as well as SVN (prefix-tree overlay),
there is no difference in time, and hardly in message.  Just things like
"[this is a placeholder, please ignore]" show up.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Eray Aslan]

[-- Attachment #1: Type: text/plain, Size: 1053 bytes --]

On Wed, Aug 24, 2011 at 07:30:11AM +0000, Robin H. Johnson wrote:
> Bandwidth: Along the same lines, rsync will always be able to use less
> bandwidth than Git, because none of the intermediate commits need to be
> transfered. This will be esp. evident as a user tree gets older (the
> amount of mtime/checksum metadata scales linearly with the size of the
> tree, not the age of the tree.

Yes, rsync protocol scales with the project size.

> The actual file content transfered scales linearly with the age of the tree).

(assuming you are talking about git here)
Uhm no? git protocol scales with chageset size.  You don't retransfer
already transferred content.  So you will be tranferring only content
from the last git pull.

rsync checks the whole tree everytime.  The extra intermediate commits
that git sends out might be a lot smaller than cheking the whole tree
(and will be if you git pull frequently).  I find the "always" in the
first line highly surprising.  Did you make any tests?

-- 
Eray Aslan <eras@gentoo.org>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Wed, Aug 24, 2011 at 10:35:29AM +0200, Fabian Groffen wrote:
> What do you mean by reordering?  From what I see from generating
> ChangeLog files from both CVS as well as SVN (prefix-tree overlay),
> there is no difference in time, and hardly in message.  Just things like
> "[this is a placeholder, please ignore]" show up.
CVS and SVN have global order, hence their ability to create changelogs
that are always growing in a single direction. Git has no global order
when merges are involved. If I merge in a commit made months ago, where
is the correct place for it in the generated changelog? When the
generated changelog is generated from scratch, where does it go? Does it
stay in the same place after more merges?

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Wed, Aug 24, 2011 at 11:48:05AM +0300, Eray Aslan wrote:
> On Wed, Aug 24, 2011 at 07:30:11AM +0000, Robin H. Johnson wrote:
> > Bandwidth: Along the same lines, rsync will always be able to use less
> > bandwidth than Git, because none of the intermediate commits need to be
> > transfered. This will be esp. evident as a user tree gets older (the
> > amount of mtime/checksum metadata scales linearly with the size of the
> > tree, not the age of the tree.
> Yes, rsync protocol scales with the project size.
> > The actual file content transfered scales linearly with the age of the tree).
> (assuming you are talking about git here)
> Uhm no? git protocol scales with chageset size.  You don't retransfer
> already transferred content.  So you will be tranferring only content
> from the last git pull.
No, I'm talking about rsync, not Git.

Rsync bandwidth usage comprises two parts:
1. Bandwidth used by the up-to-date-checks.
   Scales with the size of the tree.
2. Bandwidth used by sending newest versions of files.
   Scales with the number of changed files.

Let's call this B(checked) + B(changed)

However, most importantly, it's that rsync only sends the NEWEST version
of the files, not the complete history.

Git bandwidth usage comprises one thing, the set of commits since the
last time you did a fetch.

Let's call this B(commits)

If you run rsync very frequently, B(changed) tends towards zero,
while B(checked) becomes constant.

If you run git very frequently, B(commits) tends towards zero.

Now, the important thing happens when you run them seldom.
The sum of B(changed)+B(checked) will become smaller than B(commits)
when there are many commits that have been superseded by newer commits.

Yes, there are cases where EITHER side can win, but for the rate of
change of the tree and how often rsync should be used (not more than
every 12 hours), I believe that the rsync bandwidth usage will be less
than that of Git.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Wed, 24 Aug 2011 07:35:05 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> On Tue, Aug 23, 2011 at 11:03:24PM -0400, Mike Frysinger wrote:
> > On Tuesday, August 23, 2011 19:12:48 Robin H. Johnson wrote:
> > > On Tue, Aug 23, 2011 at 06:20:35PM -0400, Mike Frysinger wrote:
> > > > when i said "settle", i meant "logistics", not "are we doing it"
> > > That's why the next paragraph after the one you quoted proposed
> > > the use of shortlog.
> > so how is this *not* an unresolved issue ?
> 
> Regardless of generated changelogs or them being manually written, the
> only way that we can ensure that changelogs don't contain any
> reordering is by either hand-merging them (then we might as well
> hand-write them), or taking snapshots of shortlog changelogs (which
> can lead to dupes after some merges).

Didn't we want to disallow merges in gx86?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Nirbheek Chauhan]

On Wed, Aug 24, 2011 at 2:52 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Wed, Aug 24, 2011 at 10:35:29AM +0200, Fabian Groffen wrote:
>> What do you mean by reordering?  From what I see from generating
>> ChangeLog files from both CVS as well as SVN (prefix-tree overlay),
>> there is no difference in time, and hardly in message.  Just things like
>> "[this is a placeholder, please ignore]" show up.

> CVS and SVN have global order, hence their ability to create changelogs
> that are always growing in a single direction. Git has no global order
> when merges are involved. If I merge in a commit made months ago, where
> is the correct place for it in the generated changelog? When the
> generated changelog is generated from scratch, where does it go? Does it
> stay in the same place after more merges?
>

I could've sworn we had decided to disallow merge commits because of
this problem, the noise it creates, and the unnecessarily complicated
history it would create. There's no value in preserving exact history
for the portage tree.

-- 
~Nirbheek Chauhan

Gentoo GNOME+Mozilla Team

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Fabian Groffen]

On 24-08-2011 09:22:00 +0000, Robin H. Johnson wrote:
> On Wed, Aug 24, 2011 at 10:35:29AM +0200, Fabian Groffen wrote:
> > What do you mean by reordering?  From what I see from generating
> > ChangeLog files from both CVS as well as SVN (prefix-tree overlay),
> > there is no difference in time, and hardly in message.  Just things like
> > "[this is a placeholder, please ignore]" show up.
> CVS and SVN have global order, hence their ability to create changelogs
> that are always growing in a single direction. Git has no global order
> when merges are involved. If I merge in a commit made months ago, where
> is the correct place for it in the generated changelog? When the
> generated changelog is generated from scratch, where does it go? Does it
> stay in the same place after more merges?

I see, thank you.  Does it actually matter if the order changes?  Given
our current ChangeLog format we can't show the graph that's behind it.
All that counts for the user, is that the changes are documented.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Zac Medico]

On 08/24/2011 01:02 AM, Nirbheek Chauhan wrote:
> On Wed, Aug 24, 2011 at 12:28 PM, Fabian Groffen <grobian@gentoo.org> wrote:
>> On 24-08-2011 00:44:57 -0400, Matt Turner wrote:
>>> On Tue, Aug 23, 2011 at 10:30 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
>>>> On 15:49 Tue 23 Aug     , Lance Albertson wrote:
>>>>> I think using the shortlog output is the sane solution otherwise you're
>>>>> just replicating what you do in the commit.
>>>>
>>>> It's not replication if users continue to use rsync; they won't have
>>>> commit info.
>>>
>>> Do we really want users to continue using rsync? Isn't git pull so
>>> much faster? What's the downside of users using git directly?
>>
>> ehm, that you need git?  that you need to use git to get information
>> about changes?  that you need a whole new infrastructure of mirrors to
>> get it running (vs the rsync infrastructure)?  that you need at minimum
>> 800MiB to be able to look at some history, iso. 286MiB as the rsync tree
>> is now?
>>
>> Besides from that git doesn't even work on all platforms, but I can
>> imagine you don't care about that.
>>
> 
> Actually, the major blocker as I understand it, is portage metadata
> cache regeneration.

If anybody needs some background information on this, the "[RFC] DIGESTS
metadata variable for cache validation" thread can serve as a useful
reference:


http://archives.gentoo.org/gentoo-dev/msg_cfa80e33ee5fa6f854120ddfb9b468b3.xml

Also, note that it's possible to use post-sync scripts to tweak mtimes
of files pulled with git:

  https://bugs.gentoo.org/show_bug.cgi?id=355313

We've had support for git post-sync timestamp handling in emerge for
some time now:


http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=8e72dfe64208d0329a66d9d329d58ec458e79890
-- 
Thanks,
Zac

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 985 bytes --]

On Wednesday, August 24, 2011 00:44:57 Matt Turner wrote:
> On Tue, Aug 23, 2011 at 10:30 PM, Donnie Berkholz wrote:
> > On 15:49 Tue 23 Aug     , Lance Albertson wrote:
> >> I think using the shortlog output is the sane solution otherwise you're
> >> just replicating what you do in the commit.
> > 
> > It's not replication if users continue to use rsync; they won't have
> > commit info.
> 
> Do we really want users to continue using rsync? Isn't git pull so
> much faster? What's the downside of users using git directly?

other aspects that havent been covered so far:
 - we need mirrors to distribute the load, and people willing to create rsync 
mirrors are much easier than git
 - along those lines, simplicity ... very easy to bring up a rsync mirror, and 
the people maintaining our mirrors are familiar with it
 - load on the system is higher with git than rsync (at least cpu wise), thus 
we need beefier machines to handle the same # of clients
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-scm] thin manifests

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1876 bytes --]

On Monday, August 22, 2011 15:28:57 Robin H. Johnson wrote:
> Unresolved items:
> - commit signing
> - thin Manifests

how exactly are these two supposed to interact ?  the previous discussion 
seemed to miss signing.  if devs sign the thin manifests, when we go to 
produce the full manifest for rsync, we invalidate the signature.

also, a previous assertion was made which i think is incorrect:
	Due to the distributed nature of git, to do mischief, you need to
	change every clone in the world to be successful
each new sha1 comes from the previous state + new data.  so injecting code 
into the tip and finding a collision is not impossible and does not require 
modification of anything before it.  it would only be detected automatically 
by people who have the original commit, make new commits on top of that, and 
then attempt to push back again to the modified tree.  i.e. the attack is made 
against the source Gentoo repo sitting on our machines.

the other attack we want to prevent is MITM when people sync.  in this case, 
someone who syncs over git:// is perpetually vulnerable with thin manifests as 
the attacker can keep recomputing the collisions so that the modified tree 
keeps ending up with the same digests as the public one.  and the end user 
never notices without manually reviewing everything themselves.

further, it was stated:
	This has nothing to do with strength of the hash used by git
well, it sort of does.  sha1 has been shown to be weaker than brute forcing, 
and while right now it might not be computationally feasible to inject useful 
code in realtime, that is not something we should be betting on.  attacks only 
get better over time ... even in 2004 security conscious people started 
talking about migrating away from it.  and now in 2012, we want to talk about 
migrating purely to it ?
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

On Wed, Aug 24, 2011 at 03:55:59PM +0530, Nirbheek Chauhan wrote:
> > CVS and SVN have global order, hence their ability to create changelogs
> > that are always growing in a single direction. Git has no global order
> > when merges are involved. If I merge in a commit made months ago, where
> > is the correct place for it in the generated changelog? When the
> > generated changelog is generated from scratch, where does it go? Does it
> > stay in the same place after more merges?
> >
> 
> I could've sworn we had decided to disallow merge commits because of
> this problem, the noise it creates, and the unnecessarily complicated
> history it would create. There's no value in preserving exact history
> for the portage tree.

I realize I'm pretty late on this discussion, but I also would vote for
disallowing merge commits in the portage tree.

William


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-scm] Re: [gentoo-dev] Progress on cvs->git migration

[Robin H. Johnson]

On Wed, Aug 24, 2011 at 12:39:33PM +0200, Fabian Groffen wrote:
> On 24-08-2011 09:22:00 +0000, Robin H. Johnson wrote:
> > On Wed, Aug 24, 2011 at 10:35:29AM +0200, Fabian Groffen wrote:
> > > What do you mean by reordering?  From what I see from generating
> > > ChangeLog files from both CVS as well as SVN (prefix-tree overlay),
> > > there is no difference in time, and hardly in message.  Just things like
> > > "[this is a placeholder, please ignore]" show up.
> > CVS and SVN have global order, hence their ability to create changelogs
> > that are always growing in a single direction. Git has no global order
> > when merges are involved. If I merge in a commit made months ago, where
> > is the correct place for it in the generated changelog? When the
> > generated changelog is generated from scratch, where does it go? Does it
> > stay in the same place after more merges?
> I see, thank you.  Does it actually matter if the order changes?  Given
> our current ChangeLog format we can't show the graph that's behind it.
> All that counts for the user, is that the changes are documented.
Specifically, the order only matters for generating the ChangeLog from
scratch, because users will probably expect to be reading it linearly,
not out-of-order.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] thin manifests

[Robin H. Johnson]

On Thu, Aug 25, 2011 at 12:23:40AM -0400, Mike Frysinger wrote:
> On Monday, August 22, 2011 15:28:57 Robin H. Johnson wrote:
> > Unresolved items:
> > - commit signing
> > - thin Manifests
> how exactly are these two supposed to interact ?  the previous discussion 
> seemed to miss signing.  if devs sign the thin manifests, when we go to 
> produce the full manifest for rsync, we invalidate the signature.
Thin Manifests are not going to be explicitly signed like the current
signatures.

To summarize this better:
1. Thin Manifests contain DIST lines, and _nothing_ else.
1.1. Specifically: no signatures, and esp. not any other files that
     appear in Git.
2. Commits (or pushes [1]) are signed going into Git.
2.1. Non-signed commits/pushes are REJECTED by git-receive-pack on the
     server-side.
3. Git->rsync build phase:
3.1. Verify all commit signatures.
3.2. Add metadata and other files.
3.3. Build thick Manifests.
3.4. Produce new signatures for Manifests.
3.5. MetaManifest?

> the other attack we want to prevent is MITM when people sync.  in this case, 
> someone who syncs over git:// is perpetually vulnerable with thin manifests as 
> the attacker can keep recomputing the collisions so that the modified tree 
> keeps ending up with the same digests as the public one.  and the end user 
> never notices without manually reviewing everything themselves.
I don't follow this attack. The commits are signed, and the git:// user
can verify them.

> well, it sort of does.  sha1 has been shown to be weaker than brute forcing, 
...
> talking about migrating away from it.  and now in 2012, we want to talk about 
> migrating purely to it ?
RESO UPSTREAM(git). It looks like Git will probably migrate to whatever
hash wins the SHA-3 contest.

Footnotes:
[1] Current state of commit signing, 2011/09/13 05:00 UTC
There's a variation of commit signing presently being actively discussed
on the Git mailing list. It's making a LOT more progress than previous
signing discussions. Rather than signing blobs or commits directly, it's
actually signing pushes (which include the SHA1's of commits and thus
blobs). I'm personally concerned it's going to still be vulnerable to
the collision/pre-image attacks, but it's much better than no signing
(one of the attacks suggested against my SHA1-workaround signing was to
subvert the note that my signature was being stored in).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] thin manifests

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 3907 bytes --]

On Tuesday, September 13, 2011 04:00:05 Robin H. Johnson wrote:
> On Thu, Aug 25, 2011 at 12:23:40AM -0400, Mike Frysinger wrote:
> > On Monday, August 22, 2011 15:28:57 Robin H. Johnson wrote:
> > > Unresolved items:
> > > - commit signing
> > > - thin Manifests
> > 
> > how exactly are these two supposed to interact ?  the previous discussion
> > seemed to miss signing.  if devs sign the thin manifests, when we go to
> > produce the full manifest for rsync, we invalidate the signature.
> 
> Thin Manifests are not going to be explicitly signed like the current
> signatures.
> 
> To summarize this better:
> 1. Thin Manifests contain DIST lines, and _nothing_ else.
> 1.1. Specifically: no signatures, and esp. not any other files that
>      appear in Git.

for the record, i dont have a problem with thin manifests by themselves

> 2. Commits (or pushes [1]) are signed going into Git.
> 2.1. Non-signed commits/pushes are REJECTED by git-receive-pack on the
>      server-side.

what and how is it being signed if it isn't the Manifest today ?  are you 
referring to signed tags ?  that's the only signing i'm aware of in git itself 
today, and signed tags would just cause a huge amount of noise ...

i'm not terribly interested in possible additions to git if they aren't 
available for the foreseeable future.

> > the other attack we want to prevent is MITM when people sync.  in this
> > case, someone who syncs over git:// is perpetually vulnerable with thin
> > manifests as the attacker can keep recomputing the collisions so that
> > the modified tree keeps ending up with the same digests as the public
> > one.  and the end user never notices without manually reviewing
> > everything themselves.
> 
> I don't follow this attack. The commits are signed, and the git:// user
> can verify them.

my point here was wrt the original thin manifest proposal as Funtoo is using 
it: Manifest is not signed, and they're relying on the SHA1's that git 
provides as the foundation of their verification.  moving on to the paranoid 
point that SHA1's are brute-forcible, Funtoo's git:// sync method is 
vulnerable to a perpetual MITM attack.

as for signing, if it's based on the SHA1, then it still doesn't matter.

> > well, it sort of does.  sha1 has been shown to be weaker than brute
> > forcing,
> ...
> > talking about migrating away from it.  and now in 2012, we want to talk
> > about migrating purely to it ?
> 
> RESO UPSTREAM(git). It looks like Git will probably migrate to whatever
> hash wins the SHA-3 contest.

that's fine as long as we don't espouse SHA1's (and any system built upon it) 
as being secure.  so if we do provide anonymous git:// for syncing, we include 
the aforementioned caveats.

as for the devs pushing their signed SHA1's over ssh to the machine which 
produces signed Manifest2 files (as we have today), that reduces the attack 
surface to that server machine (ignoring the obvious surface of every dev 
machine where people do their commit/sign/push).  which isnt as big a deal.

> [1] Current state of commit signing, 2011/09/13 05:00 UTC
> There's a variation of commit signing presently being actively discussed
> on the Git mailing list. It's making a LOT more progress than previous
> signing discussions. Rather than signing blobs or commits directly, it's
> actually signing pushes (which include the SHA1's of commits and thus
> blobs). I'm personally concerned it's going to still be vulnerable to
> the collision/pre-image attacks, but it's much better than no signing
> (one of the attacks suggested against my SHA1-workaround signing was to
> subvert the note that my signature was being stored in).

i'm not sure how signing a push which includes only a single changeset is any 
better than signing a single changeset.  but i'll let them worry about it :P.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-scm] thin manifests

[Robin H. Johnson]

On Tue, Sep 13, 2011 at 11:19:21AM -0400, Mike Frysinger wrote:
> On Tuesday, September 13, 2011 04:00:05 Robin H. Johnson wrote:
> > On Thu, Aug 25, 2011 at 12:23:40AM -0400, Mike Frysinger wrote:
> > > On Monday, August 22, 2011 15:28:57 Robin H. Johnson wrote:
> > > > Unresolved items:
> > > > - commit signing
> > > > - thin Manifests
> > > 
> > > how exactly are these two supposed to interact ?  the previous discussion
> > > seemed to miss signing.  if devs sign the thin manifests, when we go to
> > > produce the full manifest for rsync, we invalidate the signature.
> > 
> > Thin Manifests are not going to be explicitly signed like the current
> > signatures.
> > 
> > To summarize this better:
> > 1. Thin Manifests contain DIST lines, and _nothing_ else.
> > 1.1. Specifically: no signatures, and esp. not any other files that
> >      appear in Git.
> 
> for the record, i dont have a problem with thin manifests by themselves
> 
> > 2. Commits (or pushes [1]) are signed going into Git.
> > 2.1. Non-signed commits/pushes are REJECTED by git-receive-pack on the
> >      server-side.
> what and how is it being signed if it isn't the Manifest today?  
What: The commit or push
How: see below

> are you 
> referring to signed tags ?  that's the only signing i'm aware of in git itself 
> today, and signed tags would just cause a huge amount of noise ...
>
> i'm not terribly interested in possible additions to git if they aren't 
> available for the foreseeable future.
There are 3 paths for signing in Git:
1. Signed tags (core, in stable)
2. Signed commits w/ signatures stored in notes tree (hooks)
3. Signed pushes (core, patches on git ML)

#1 doesn't cover what we need to.
#2 can be subverted at the client end by failing to set up the hooks,
   and it's difficult to verify on the server when receiving the packfile
   from a push (not impossible, just difficult). Also apparently
   technical issues in having multiple signatures on a single commit.
#3 Easier to verify at the receiving end, setup is really simple, just
   GPG key and 'git push -s'





> 
> > > the other attack we want to prevent is MITM when people sync.  in this
> > > case, someone who syncs over git:// is perpetually vulnerable with thin
> > > manifests as the attacker can keep recomputing the collisions so that
> > > the modified tree keeps ending up with the same digests as the public
> > > one.  and the end user never notices without manually reviewing
> > > everything themselves.
> > 
> > I don't follow this attack. The commits are signed, and the git:// user
> > can verify them.
> my point here was wrt the original thin manifest proposal as Funtoo is using 
> it: Manifest is not signed, and they're relying on the SHA1's that git 
> provides as the foundation of their verification.  moving on to the paranoid 
> point that SHA1's are brute-forcible, Funtoo's git:// sync method is 
> vulnerable to a perpetual MITM attack.
> 
> as for signing, if it's based on the SHA1, then it still doesn't matter.
The MITM case is a variant of my collision attack I documented earlier
in the signing discussion since the kernel.org attack. The MITM case is
actually harder, because you could subvert the entire mechanism and just
hand over an attack tree WITHOUT any verification chain at all (or
worse, hand over a custom-crafted packfile to exploit git, see the
discussion about nulls recently).

> > RESO UPSTREAM(git). It looks like Git will probably migrate to whatever
> > hash wins the SHA-3 contest.
> that's fine as long as we don't espouse SHA1's (and any system built upon it) 
> as being secure.  so if we do provide anonymous git:// for syncing, we include 
> the aforementioned caveats.
I'm fine with that.

> as for the devs pushing their signed SHA1's over ssh to the machine which 
> produces signed Manifest2 files (as we have today), that reduces the attack 
> surface to that server machine (ignoring the obvious surface of every dev 
> machine where people do their commit/sign/push).  which isnt as big a deal.
1. devs push via SSH, reduced attack surface.
2. rsync users get full signed Manifest2 files, no MITM attack surface.
3. anongit users need to verify exposed blobs [1] are signed by a key in
   the Gentoo set of trusted keys (signed pushes cover commits, commits
   cover blobs). MITM potential reduced by signatures.

> i'm not sure how signing a push which includes only a single changeset is any 
> better than signing a single changeset.  but i'll let them worry about it :P.
> -mike
One of the variations is that two people should be able to sign the same
push/head/commit-range, and that's apparently difficult w/ signatures
per changeset.

Footnotes:
1. "exposed blobs": At a given point in time, the set of signatures
   that for every blob in the working directory copy.


-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-scm] Progress on cvs->git migration

[Alexey Shvetsov]

Hi all!

Any progress on this topic?

portage already has thin manifest and some king of changelog generation 
functionality.
what else we need to proceed with migration?

PS there were no updates in tracker bug
-- 
Best Regards,
Alexey 'Alexxy' Shvetsov
Petersburg Nuclear Physics Institute, Russia
Department of Molecular and Radiation Biophysics
Gentoo Team Ru
Gentoo Linux Dev
mailto:alexxyum@gmail.com
mailto:alexxy@gentoo.org
mailto:alexxy@omrb.pnpi.spb.ru

Re: [gentoo-scm] Progress on cvs->git migration

[Robin H. Johnson]

On Sat, Nov 05, 2011 at 10:31:20PM +0300, Alexey Shvetsov wrote:
> Hi all!
> 
> Any progress on this topic?
> 
> portage already has thin manifest and some king of changelog generation 
> functionality.
> what else we need to proceed with migration?
> 
> PS there were no updates in tracker bug
Pending from the last update:
- commit signing
- merge policy

Commit signing is locked up in discussion on the upstream Git list.

Merge policy looks like it might be dictated by how the commit signing
goes: some of the models for signing include having explicit merge
commits that are signed.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85