[gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4956 bytes --]

Hi,

I'm quite tired of promises and all that perfectionist non-sense which
locks us up with CVS for next 10 years of bikeshed. Therefore, I have
prepared a plan how to do git migration, and I believe it's doable in
less than 2 weeks (plus the testing). Of course, that assumes infra is
going to cooperate quickly or someone else is willing to provide the
infra for it.

I can provide some testing repos once someone is willing to provide
the hardware.


What needs to be done
---------------------

I can do most of the scripting. What I need others to do is provide
the hosting for git repos. We can't use public services like github
since they don't allow us to set our own update hook, so we can't
enforce signing policies etc.

Once basic infra is ready, I think the following is the best way to
switch:

1. send announcement to devs to explain how to use git,

2. lock CVS out to read-only,

3. create all the git repos, get hooks rolling,

4. enable R/W access to the repos.

With some luck, no more than 2 hours downtime.


The infra
---------

The general idea is based on 3-level structure that's extension of how
Funtoo works. The following ultimately pretty picture explains that:

  +----------------+
  | developer repo | - - - - - - - - - - -, 
  +----------------+                      v
          |               +------------------------------+
          |               | cache, DTDs and other extras |
          v               +------------------------------+
  +----------------+                      |
  | user sync repo | <--------------------'
  +----------------+ - - - - - - - - - - ,
          |                              v
          |               +-----------------------------+
          |               | ChangeLogs, thick Manifests |
          v               +-----------------------------+
  +----------------+                     |
  |    rsync       | <-------------------'
  +----------------+

Text version:

We have main developer repo where developers work & commit and are
relatively happy. For every push into developer repo, automated magic
thingie merges stuff into user sync repo and updates the metadata cache
there.

User sync repo is for power users than want to fetch via git. It's quite
fast and efficient for frequent updates, and also saves space by being free
of ChangeLogs.

On top of user sync repo rsync is propagated. The rsync tree is populated
with all old ChangeLogs copied from CVS (stored in 30M git repo), new
ChangeLogs are generated from git logs and Manifests are expanded.


Main developer repo
-------------------

I was able to create a start git repository that takes around 66M
as a git pack (this is how much you will have to fetch to start working
with it). The repository is stripped clean of history and ChangeLogs,
and has thin Manifests only.

This means we don't have to wait till someone figures out the perfect
way of converting the old CVS repository. You don't need that history
most of the time, and you can play with CVS to get it if you really do.
In any case, we would likely strip the history anyway to get a small
repo to work with.

I have prepared a basic git update hook that keeps master clean
and attached it to the bug [1]. It enforces basic policies, prevents
forced updates and checks GPG signatures on left-most history line. It
can also be extended to do more extensive tree checks.

For GPG signing, I relied upon gpg to do the right thing. That is, git
checks the signatures and we accept only trusted signatures. So
an external tool (gentoo-keys) need to play with gpg to import, trust
and revoke developer keys.

I think we should also merge gentoo-news & glsa & herds.xml into
the repository. They all reference Gentoo packages at a particular
state in time, and it would be much nicer to have them synced properly.

[1]:https://bugs.gentoo.org/show_bug.cgi?id=502060


User syncing repo
-----------------

IMO this will be the most useful syncing method. The user syncing repo
is updated automatically for developer repo commits, and afterwards
md5-cache is regenerated and committed. Also other repositories (like
DTDs, glsas and others if you dislike the previous idea) are merged
into it.

This repo is still free of ChangeLogs (since git logs are more
efficient) and has thin Manifests. It's the space-efficient Gentoo
variant. And commits are signed so users can verify the trust.


The rsync tree
--------------

We'd also propagate things to rsync. We'd have to populate it with old
ChangeLogs, new ChangeLog entries (autogenerated from git) and thick
Manifests. So users won't notice much of a change.

The remaining issue is signing of stuff. We could supposedly sign
Manifests but IMO it's a waste of resources considered how poor
the signing system is for non-git repos.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Jauhien Piatlicki]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

Hi,

14.09.14 14:03, Michał Górny написав(ла):
> Hi,
> 
> I'm quite tired of promises and all that perfectionist non-sense which
> locks us up with CVS for next 10 years of bikeshed. Therefore, I have
> prepared a plan how to do git migration, and I believe it's doable in
> less than 2 weeks (plus the testing). Of course, that assumes infra is
> going to cooperate quickly or someone else is willing to provide the
> infra for it.
> 

as always, nice effort, but I foresee lots of bikeshedding in this thread. )

> This means we don't have to wait till someone figures out the perfect
> way of converting the old CVS repository. You don't need that history
> most of the time, and you can play with CVS to get it if you really do.
> In any case, we would likely strip the history anyway to get a small
> repo to work with.
> 

Is it so difficult to convert CVS history?

> 
> The rsync tree
> --------------
> 
> We'd also propagate things to rsync. We'd have to populate it with old
> ChangeLogs, new ChangeLog entries (autogenerated from git) and thick
> Manifests. So users won't notice much of a change.
> 

How will user check the ebuild integrity with thick manifests using rsync?

> The remaining issue is signing of stuff. We could supposedly sign
> Manifests but IMO it's a waste of resources considered how poor
> the signing system is for non-git repos.
> 

Again, how will user check the integrity and authenticity if Manifests are unsigned?

Also, it would be a good idea to add automatic signature checking to portage for overlays that use signing (or is it already done?).

--
Jauhien



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Jauhien Piatlicki:
> 
> Again, how will user check the integrity and authenticity if Manifests are unsigned?
> 

While this is an issue to be solved, it shouldn't be a blocker for the
git migration.

There is no regression if this isn't solved. There is no sane automated
method for verifying signed Manifests yet (that should be on PM level)
and signing them isn't even enforced throughout the tree. Moreover I
highly doubt that there is any user who runs around ebuild directories
and checks Manifest signatures by hand.

People who really care use emerge-webrsync.
If we use the proposed solution, then there is an additional method via
the User syncing repo, so it's a win.

We can put more effort into solving this for rsync mirrors later, but
I'd rather focus on the git migration.

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

hasufell posted on Sun, 14 Sep 2014 13:50:32 +0000 as excerpted:

> Jauhien Piatlicki:
>> 
>> Again, how will user check the integrity and authenticity if Manifests
>> are unsigned?

> There is no regression if this isn't solved.

> People who really care use emerge-webrsync.
> If we use the proposed solution, then there is an additional method via
> the User syncing repo, so it's a win.

Absolutely.  emerge-webrsync is the current "I care enough to worry about 
it" method, and this already adds the user-sync git repo as an "I care" 
option.  Leaving standard rsync users where they already are isn't a 
regression and shouldn't be a blocker.  Don't let the perfect be the 
enemy of the imperfect but better! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2341 bytes --]

Dnia 2014-09-14, o godz. 15:09:25
Jauhien Piatlicki <jauhien@gentoo.org> napisał(a):

> 14.09.14 14:03, Michał Górny написав(ла):
> > Hi,
> > 
> > I'm quite tired of promises and all that perfectionist non-sense which
> > locks us up with CVS for next 10 years of bikeshed. Therefore, I have
> > prepared a plan how to do git migration, and I believe it's doable in
> > less than 2 weeks (plus the testing). Of course, that assumes infra is
> > going to cooperate quickly or someone else is willing to provide the
> > infra for it.
> > 
> 
> as always, nice effort, but I foresee lots of bikeshedding in this thread. )

Yes. I'm planning to ignore most of bikeshed and take only serious
answers into consideration. Otherwise, we will be stuck with CVS.

> > This means we don't have to wait till someone figures out the perfect
> > way of converting the old CVS repository. You don't need that history
> > most of the time, and you can play with CVS to get it if you really do.
> > In any case, we would likely strip the history anyway to get a small
> > repo to work with.
> 
> Is it so difficult to convert CVS history?

It may be difficult to convert it properly, especially considering
the splitting of ebuild+Manifest commit. Then we need to somehow check
if it was converted properly. I don't even want to waste my time on
this. IMO the history doesn't have such a great value.

> > The rsync tree
> > --------------
> > 
> > We'd also propagate things to rsync. We'd have to populate it with old
> > ChangeLogs, new ChangeLog entries (autogenerated from git) and thick
> > Manifests. So users won't notice much of a change.
> > 
> 
> How will user check the ebuild integrity with thick manifests using rsync?

The same way he currently does :).

> > The remaining issue is signing of stuff. We could supposedly sign
> > Manifests but IMO it's a waste of resources considered how poor
> > the signing system is for non-git repos.
> 
> Again, how will user check the integrity and authenticity if Manifests are unsigned?

As far as I'm concerned, user can use the user git tree to get proper
signatures or any other method that has proper signing support already.

If someone wants proper GPG support in rsync, he can work on that.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 400 bytes --]

>>>>> On Sun, 14 Sep 2014, Michał Górny wrote:

> I think we should also merge gentoo-news & glsa & herds.xml into the
> repository. They all reference Gentoo packages at a particular state
> in time, and it would be much nicer to have them synced properly.

Not a good idea, because we may want to grant commit access to these
repos for people who are not necessarily ebuild devs.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Johannes Huber]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

Am Sonntag 14 September 2014, 15:17:41 schrieb Ulrich Mueller:
> >>>>> On Sun, 14 Sep 2014, Michał Górny wrote:
> > I think we should also merge gentoo-news & glsa & herds.xml into the
> > repository. They all reference Gentoo packages at a particular state
> > in time, and it would be much nicer to have them synced properly.
> 
> Not a good idea, because we may want to grant commit access to these
> repos for people who are not necessarily ebuild devs.
> 
> Ulrich

This could be solved by a pull requests review tool (gerrit, reviewboard, 
gitlab etc).

-- 
Johannes Huber (johu)
Gentoo Linux Developer / KDE Team
GPG Key ID F3CFD2BD

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

>>>>> On Sun, 14 Sep 2014, Johannes Huber wrote:

> Am Sonntag 14 September 2014, 15:17:41 schrieb Ulrich Mueller:
>> >>>>> On Sun, 14 Sep 2014, Michał Górny wrote:
>> > I think we should also merge gentoo-news & glsa & herds.xml into the
>> > repository. They all reference Gentoo packages at a particular state
>> > in time, and it would be much nicer to have them synced properly.
>> 
>> Not a good idea, because we may want to grant commit access to
>> these repos for people who are not necessarily ebuild devs.

> This could be solved by a pull requests review tool (gerrit,
> reviewboard, gitlab etc).

Second argument is that gentoo-x86 is large enough as it is, and we
shouldn't make it even larger by merging in things that are not
strictly necessary. Especially glsa has a non negligible size.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

Dnia 2014-09-14, o godz. 15:17:41
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Sun, 14 Sep 2014, Michał Górny wrote:
> 
> > I think we should also merge gentoo-news & glsa & herds.xml into the
> > repository. They all reference Gentoo packages at a particular state
> > in time, and it would be much nicer to have them synced properly.
> 
> Not a good idea, because we may want to grant commit access to these
> repos for people who are not necessarily ebuild devs.

We may want to add metadata.xml access to those people too.

If you really are that distrustful of our contributors, I believe we
can do per-path filtering in the 'update' hook, or use pull request
or intermediate-repository based workflow.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Jauhien Piatlicki]

[-- Attachment #1: Type: text/plain, Size: 219 bytes --]

Another question: will it be possible to maintain a copy of tree on github to make contributions for users simpler (similarly to e.g. science overlay)? (Can it somehow be combined with proposed signing mechanism?)



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Jauhien Piatlicki]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]

14.09.14 15:23, Jauhien Piatlicki написав(ла):
> Another question: will it be possible to maintain a copy of tree on github to make contributions for users simpler (similarly to e.g. science overlay)? (Can it somehow be combined with proposed signing mechanism?)
> 
> 

Or well, have our own pull requests review tool.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

["C. Bergström"]

On 09/14/14 08:24 PM, Jauhien Piatlicki wrote:
> 14.09.14 15:23, Jauhien Piatlicki написав(ла):
>> Another question: will it be possible to maintain a copy of tree on github to make contributions for users simpler (similarly to e.g. science overlay)? (Can it somehow be combined with proposed signing mechanism?)
>>
>>
> Or well, have our own pull requests review tool.
NIH? What would be the benefit of that.. before going down this path.. I 
think there's some good tools around which may at least serve as a base 
to (fork) from before starting a ground up project.

Sorry to jump in the middle of the conversation, but I know 1st hand how 
much is involved here.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Jauhien Piatlicki]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

14.09.14 15:25, "C. Bergström" написав(ла):
> On 09/14/14 08:24 PM, Jauhien Piatlicki wrote:
>> 14.09.14 15:23, Jauhien Piatlicki написав(ла):
>>> Another question: will it be possible to maintain a copy of tree on github to make contributions for users simpler (similarly to e.g. science overlay)? (Can it somehow be combined with proposed signing mechanism?)
>>>
>>>
>> Or well, have our own pull requests review tool.
> NIH? What would be the benefit of that.. before going down this path.. I think there's some good tools around which may at least serve as a base to (fork) from before starting a ground up project.
> 
> Sorry to jump in the middle of the conversation, but I know 1st hand how much is involved here.
> 

I was not precise. By our own I mean hosted by us, not by github. )



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Jauhien Piatlicki:
> 
> Or well, have our own pull requests review tool.
> 
> 

Also only a secondary problem. Mirroring on github/bitbucket whatever
should be fairly straightforward to allow user contributions.

In addition the usual git workflow via e-mail/ML would become more
popular (either via git style patches or plain pull request information
with branch/commit/repository).

So I'd suggest to focus on the git migration first.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

Dnia 2014-09-14, o godz. 15:23:24
Jauhien Piatlicki <jauhien@gentoo.org> napisał(a):

> Another question: will it be possible to maintain a copy of tree on github to make contributions for users simpler (similarly to e.g. science overlay)? (Can it somehow be combined with proposed signing mechanism?)

Yes. I'm planning to have a mirror on github and bitbucket,
and auto-pushing to both.

However, I'm wondering if it would be possible to restrict people from
accidentally committing straight into github (e.g. merging pull
requests there instead of to our main server).

In fact, I would start my experiments straight into github if not
the fact that they don't allow us to set our own update hooks.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1113 bytes --]

On 15 September 2014 02:40, Michał Górny <mgorny@gentoo.org> wrote:

> However, I'm wondering if it would be possible to restrict people from
> accidentally committing straight into github (e.g. merging pull
> requests there instead of to our main server).
>


Easy.

Put the Gentoo repo in its own group.
Don't give anyone any kinds of permissions on it.
Have only one approved account for the purpose of pushing commits.
Have a post-push hook that replicates to github as that approved account

=> Github is just a read only mirror, any pull reqs submitted there will be
fielded and pushed to gentoo directly.

Only downside there is the way github pull reqs work is if the final SHA1's
that hit tree don't match, the pull req doesn't close.

Solutions:

- A) Have somebody tasked with reaping old pull reqs with permissions
granted. ( Uck )
- B) Always use a merge of some kind to mark the pull req as dead ( for
instance, an "ours" merge to mark the branch as deprecated )

Both of those options are kinda ugly.



-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 2210 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

Dnia 2014-09-15, o godz. 03:15:14
Kent Fredric <kentfredric@gmail.com> napisał(a):

> On 15 September 2014 02:40, Michał Górny <mgorny@gentoo.org> wrote:
> 
> > However, I'm wondering if it would be possible to restrict people from
> > accidentally committing straight into github (e.g. merging pull
> > requests there instead of to our main server).
>
> => Github is just a read only mirror, any pull reqs submitted there will be
> fielded and pushed to gentoo directly.
> 
> Only downside there is the way github pull reqs work is if the final SHA1's
> that hit tree don't match, the pull req doesn't close.
> 
> Solutions:
> 
> - A) Have somebody tasked with reaping old pull reqs with permissions
> granted. ( Uck )
> - B) Always use a merge of some kind to mark the pull req as dead ( for
> instance, an "ours" merge to mark the branch as deprecated )
> 
> Both of those options are kinda ugly.

If you merge a pull request, I suggest doing a proper 'git merge -S'
anyway to get a developer signature on top of all the changes.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1530 bytes --]

On Sun, Sep 14, 2014 at 05:40:30PM +0200, Michał Górny wrote:
> Dnia 2014-09-15, o godz. 03:15:14 Kent Fredric napisał(a):
> > Only downside there is the way github pull reqs work is if the
> > final SHA1's that hit tree don't match, the pull req doesn't
> > close.
> > 
> > Solutions:
> > 
> > - A) Have somebody tasked with reaping old pull reqs with
> > permissions granted. ( Uck )
> > - B) Always use a merge of some kind to mark the pull req as dead
> > ( for instance, an "ours" merge to mark the branch as deprecated )
> > 
> > Both of those options are kinda ugly.
> 
> If you merge a pull request, I suggest doing a proper 'git merge -S'
> anyway to get a developer signature on top of all the changes.

Some previous package-tree-in-Git efforts suggested that only
Gentoo-dev signatures were acceptable, and that those signatures would
be required on every commit (not just the first-parent line) [1,2].  I
don't see the point of that, so long as Gentoo devs are signing the
first-parent line, but if folks still want Gentoo-dev signatures on
every commit the ‘git merge -S’ approach will not work for closing
PRs.

Cheers,
Trevor

[1]: http://article.gmane.org/gmane.linux.gentoo.devel/77572
     id:CAGfcS_maNfikeVTj3cmcQ1OF-uQAVEbE2r1oKykYGwC5VOmvfw@mail.gmail.com
[2]: https://bugs.gentoo.org/show_bug.cgi?id=502060#c0

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

W. Trevor King:
> On Sun, Sep 14, 2014 at 05:40:30PM +0200, Michał Górny wrote:
>> Dnia 2014-09-15, o godz. 03:15:14 Kent Fredric napisał(a):
>>> Only downside there is the way github pull reqs work is if the
>>> final SHA1's that hit tree don't match, the pull req doesn't
>>> close.
>>>
>>> Solutions:
>>>
>>> - A) Have somebody tasked with reaping old pull reqs with
>>> permissions granted. ( Uck )
>>> - B) Always use a merge of some kind to mark the pull req as dead
>>> ( for instance, an "ours" merge to mark the branch as deprecated )
>>>
>>> Both of those options are kinda ugly.
>>
>> If you merge a pull request, I suggest doing a proper 'git merge -S'
>> anyway to get a developer signature on top of all the changes.
> 
> Some previous package-tree-in-Git efforts suggested that only
> Gentoo-dev signatures were acceptable, and that those signatures would
> be required on every commit (not just the first-parent line) [1,2].  I
> don't see the point of that, so long as Gentoo devs are signing the
> first-parent line, but if folks still want Gentoo-dev signatures on
> every commit the ‘git merge -S’ approach will not work for closing
> PRs.
> 
> Cheers,
> Trevor
> 
> [1]: http://article.gmane.org/gmane.linux.gentoo.devel/77572
>      id:CAGfcS_maNfikeVTj3cmcQ1OF-uQAVEbE2r1oKykYGwC5VOmvfw@mail.gmail.com
> [2]: https://bugs.gentoo.org/show_bug.cgi?id=502060#c0
> 

Yes, there is a possible attack vector mentioned in this comment
https://bugs.gentoo.org/show_bug.cgi?id=502060#c16

So we'd basically end up using either "git cherry-pick" or "git am" for
"pulling" user stuff, so that we also sign the blobs.

Regular merges would still be possible for developer pull requests, but
that's probably not the primary use case anyway.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]

On Sun, Sep 14, 2014 at 10:38:41PM +0000, hasufell wrote:
> Yes, there is a possible attack vector mentioned in this comment
> https://bugs.gentoo.org/show_bug.cgi?id=502060#c16

From that comment, the point 1.2 is highly unlikely [1]:

  1. Attacker constructs a init.d script, regular part at the start,
     malicious part at the end
  1.1. This would be fairly simple, just construct two start()
     functions, one of which is mundane, the other is malicious.
  1.2. Both variants of the script have the same SHA1...

> So we'd basically end up using either "git cherry-pick" or "git am"
> for "pulling" user stuff, so that we also sign the blobs.

Rebasing the original commits doesn't protect you from the birthday
attach either, because the vulnerable hash is likely going to still be
in the rebased commit's tree.  All rebasing does is swap the committer
and drop the initial signature.

Cheers,
Trevor

[1]: http://article.gmane.org/gmane.comp.version-control.git/210622

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

W. Trevor King:
> On Sun, Sep 14, 2014 at 10:38:41PM +0000, hasufell wrote:
>> Yes, there is a possible attack vector mentioned in this comment
>> https://bugs.gentoo.org/show_bug.cgi?id=502060#c16
> 
> From that comment, the point 1.2 is highly unlikely [1]:
> 
>   1. Attacker constructs a init.d script, regular part at the start,
>      malicious part at the end
>   1.1. This would be fairly simple, just construct two start()
>      functions, one of which is mundane, the other is malicious.
>   1.2. Both variants of the script have the same SHA1...
> 
>> So we'd basically end up using either "git cherry-pick" or "git am"
>> for "pulling" user stuff, so that we also sign the blobs.
> 
> Rebasing the original commits doesn't protect you from the birthday
> attach either, because the vulnerable hash is likely going to still be
> in the rebased commit's tree.  All rebasing does is swap the committer
> and drop the initial signature.
> 

According to Robin, it's not about rebasing, it's about signing all
commits so that messing with the blob (even if it has the same sha-1)
will cause signature verification failure.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Sun, Sep 14, 2014 at 6:56 PM, hasufell <hasufell@gentoo.org> wrote:
> According to Robin, it's not about rebasing, it's about signing all
> commits so that messing with the blob (even if it has the same sha-1)
> will cause signature verification failure.
>

The only thing that gets signed is the commit message, and the only
thing that ties the commit message to the code is the sha1 of the
top-level tree.  If you can attack sha1 either at any tree level or at
the blob level you can defeat the signature.

That is way better than nothing though - I think it is worth pursuing
until somebody comes up with a way to upgrade git to more secure
hashes.  Most projects don't gpg sign their trees at all, including
linux.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Rich Freeman:
> On Sun, Sep 14, 2014 at 6:56 PM, hasufell <hasufell@gentoo.org> wrote:
>> According to Robin, it's not about rebasing, it's about signing all
>> commits so that messing with the blob (even if it has the same sha-1)
>> will cause signature verification failure.
>>
> 
> The only thing that gets signed is the commit message, and the only
> thing that ties the commit message to the code is the sha1 of the
> top-level tree.  If you can attack sha1 either at any tree level or at
> the blob level you can defeat the signature.
> 

So can we get this clear now.

Robin said
> The Git commit-signing design explicitly signs the entire commit, including blob contents, to avoid this security problem.

Is this correct or not?

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

On Sun, Sep 14, 2014 at 11:25:33PM +0000, hasufell wrote:
> So can we get this clear now.
> 
> Robin said
>
> > The Git commit-signing design explicitly signs the entire commit,
> > including blob contents, to avoid this security problem.
> 
> Is this correct or not?

That is false.  The commit signature explicitly signs the commit,
which includes the root tree hash.  That is the only connection
between the signature and the tree contents.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

On 15 September 2014 11:25, hasufell <hasufell@gentoo.org> wrote:

> Robin said
> > The Git commit-signing design explicitly signs the entire commit,
> including blob contents, to avoid this security problem.
>
> Is this correct or not?
>

I can verify a commit by hand with only the commit object and gpg, but
without any of the trees or parents.

https://gist.github.com/kentfredric/8448fe55ffab7d314ecb


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1236 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Jauhien Piatlicki]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

Hi,

On 09/15/2014 01:37 AM, Kent Fredric wrote:
> On 15 September 2014 11:25, hasufell <hasufell@gentoo.org> wrote:
> 
>> Robin said
>>> The Git commit-signing design explicitly signs the entire commit,
>> including blob contents, to avoid this security problem.
>>
>> Is this correct or not?
>>
> 
> I can verify a commit by hand with only the commit object and gpg, but
> without any of the trees or parents.
> 
> https://gist.github.com/kentfredric/8448fe55ffab7d314ecb
> 
> 

So signing of git commits does not guarantee enough security (taking
that SHA1 is weak and can be broken), right? Could we than just use
usual (not thin) manifests?

--
Jauhien


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-dev] git security (SHA-1)

[hasufell]

Jauhien Piatlicki:
> Hi,
> 
> On 09/15/2014 01:37 AM, Kent Fredric wrote:
>> On 15 September 2014 11:25, hasufell <hasufell@gentoo.org> wrote:
>>
>>> Robin said
>>>> The Git commit-signing design explicitly signs the entire commit,
>>> including blob contents, to avoid this security problem.
>>>
>>> Is this correct or not?
>>>
>>
>> I can verify a commit by hand with only the commit object and gpg, but
>> without any of the trees or parents.
>>
>> https://gist.github.com/kentfredric/8448fe55ffab7d314ecb
>>
>>
> 
> So signing of git commits does not guarantee enough security (taking
> that SHA1 is weak and can be broken), right? Could we than just use
> usual (not thin) manifests?
> 

* there is no known SHA-1 collision afais
* calculating one isn't that hard. NSA might be able to do it in
reasonable time
* however, the algorithms to do that will come up with random garbage,
so it's a completely different thing to hide a useful vulnerability
behind a SHA-1 collision

Re: [gentoo-dev] git security (SHA-1)

[hasufell]

hasufell:
> 
> * there is no known SHA-1 collision afais
> * calculating one isn't that hard. NSA might be able to do it in
> reasonable time
> * however, the algorithms to do that will come up with random garbage,
> so it's a completely different thing to hide a useful vulnerability
> behind a SHA-1 collision
> 

That said... an attacker who has that much resources to calculate a
_random_ hash collision in reasonable time would certainly have a lot of
easier attack vectors than forging a _non-random_ hash collision that
contains actual working code (which, afaiu doesn't effectively work with
the current attack algorithms on SHA-1).

He could simply break into one of the ~200 developer computers. There's
a pretty high chance at least one of them is running windows or known
vulnerable versions of the kernel or other random packages.

No need to waste millions of dollars on SHA-1.

Re: [gentoo-dev] git security (SHA-1)

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

On Mon, Sep 15, 2014 at 7:02 AM, hasufell <hasufell@gentoo.org> wrote:

> hasufell:
> >
> > * there is no known SHA-1 collision afais
> > * calculating one isn't that hard. NSA might be able to do it in
> > reasonable time
> > * however, the algorithms to do that will come up with random garbage,
> > so it's a completely different thing to hide a useful vulnerability
> > behind a SHA-1 collision
> >
>
> That said... an attacker who has that much resources to calculate a
> _random_ hash collision in reasonable time would certainly have a lot of
> easier attack vectors than forging a _non-random_ hash collision that
> contains actual working code (which, afaiu doesn't effectively work with
> the current attack algorithms on SHA-1).
>
> He could simply break into one of the ~200 developer computers. There's
> a pretty high chance at least one of them is running windows or known
> vulnerable versions of the kernel or other random packages.
>
> No need to waste millions of dollars on SHA-1.
>

Even if you wanted to burn the money to find that magical collision that
actually contains working code, you've still got to somehow propagate that
to other repositories, since they'll just ignore it for having the same
hash as an already-existing object.

[-- Attachment #2: Type: text/html, Size: 1734 bytes --]

Re: [gentoo-dev] git security (SHA-1)

[Duy Nguyen]

On Tue, Sep 16, 2014 at 5:11 AM, Gordon Pettey <petteyg359@gmail.com> wrote:
> On Mon, Sep 15, 2014 at 7:02 AM, hasufell <hasufell@gentoo.org> wrote:
>>
>> hasufell:
>> >
>> > * there is no known SHA-1 collision afais
>> > * calculating one isn't that hard. NSA might be able to do it in
>> > reasonable time
>> > * however, the algorithms to do that will come up with random garbage,
>> > so it's a completely different thing to hide a useful vulnerability
>> > behind a SHA-1 collision
>> >
>>
>> That said... an attacker who has that much resources to calculate a
>> _random_ hash collision in reasonable time would certainly have a lot of
>> easier attack vectors than forging a _non-random_ hash collision that
>> contains actual working code (which, afaiu doesn't effectively work with
>> the current attack algorithms on SHA-1).
>>
>> He could simply break into one of the ~200 developer computers. There's
>> a pretty high chance at least one of them is running windows or known
>> vulnerable versions of the kernel or other random packages.
>>
>> No need to waste millions of dollars on SHA-1.
>
>
> Even if you wanted to burn the money to find that magical collision that
> actually contains working code, you've still got to somehow propagate that
> to other repositories, since they'll just ignore it for having the same hash
> as an already-existing object.

In the fetch/pull case, if you receive the "same" object that you
already have, git performs byte-to-byte comparison and warns loudly if
the "new"object does not match yours.
-- 
Duy

Re: [gentoo-dev] git security (SHA-1)

[Duy Nguyen]

On Tue, Sep 16, 2014 at 5:41 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>> Even if you wanted to burn the money to find that magical collision that
>> actually contains working code, you've still got to somehow propagate that
>> to other repositories, since they'll just ignore it for having the same hash
>> as an already-existing object.
>
> In the fetch/pull case, if you receive the "same" object that you
> already have, git performs byte-to-byte comparison and warns loudly if
> the "new"object does not match yours.

Correction, in the push case as well. You have to find another way to
inject this collided object (perhaps via a server that is not C Git).
-- 
Duy

Re: [gentoo-dev] git security (SHA-1)

[Rich Freeman]

On Mon, Sep 15, 2014 at 6:11 PM, Gordon Pettey <petteyg359@gmail.com> wrote:
>
> Even if you wanted to burn the money to find that magical collision that
> actually contains working code, you've still got to somehow propagate that
> to other repositories, since they'll just ignore it for having the same hash
> as an already-existing object.
>

Well, if you're willing to trust that nobody is able to tamper with
repositories, then you don't need gpg signatures in the first place.

I think that gpg signatures protected by an SHA1 hash provide fairly
little security - a chain is as strong as its weakest link and sha1
has been considered fairly weak for years now.

However, I think it does make sense to at least get gpg into the
workflow in the hopes that some day git will move to a stronger hash,
and since it isn't a huge hardship to do so.

I wouldn't make too light of the use of SHA1 though.  As you point out
simply exploiting it isn't enough, but the whole reason for having
signatures is to make an attack on a central repository useless.
Having gpg on top of ssh keys and all that is obviously redundant, but
that is the whole point of it.

--
Rich

Re: [gentoo-dev] git security (SHA-1)

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 15/09/14 07:59 PM, Rich Freeman wrote:
> On Mon, Sep 15, 2014 at 6:11 PM, Gordon Pettey
> <petteyg359@gmail.com> wrote:
>> 
>> Even if you wanted to burn the money to find that magical
>> collision that actually contains working code, you've still got
>> to somehow propagate that to other repositories, since they'll
>> just ignore it for having the same hash as an already-existing
>> object.
>> 
> 
> Well, if you're willing to trust that nobody is able to tamper
> with repositories, then you don't need gpg signatures in the first
> place.
> 
> I think that gpg signatures protected by an SHA1 hash provide
> fairly little security - a chain is as strong as its weakest link
> and sha1 has been considered fairly weak for years now.
> 
> However, I think it does make sense to at least get gpg into the 
> workflow in the hopes that some day git will move to a stronger
> hash, and since it isn't a huge hardship to do so.
> 
> I wouldn't make too light of the use of SHA1 though.  As you point
> out simply exploiting it isn't enough, but the whole reason for
> having signatures is to make an attack on a central repository
> useless. Having gpg on top of ssh keys and all that is obviously
> redundant, but that is the whole point of it.
> 
> -- Rich
> 

If the issue preventing protection is that the gpg signature only
signs the hash, couldn't we just make repoman automatically add to the
bottom of the comment a clearsign on the contents of the commit?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlQYPskACgkQ2ugaI38ACPDjowEAmfMQePUgmLSDrmKyXxdUfbil
g6KVaPkL1yfDwrLP7J8BAK+g5MMCMDgH9wDzEHIYerDi9ZIm39AfwazQF3mz3dPR
=slAr
-----END PGP SIGNATURE-----

Re: [gentoo-dev] git security (SHA-1)

[Rich Freeman]

On Tue, Sep 16, 2014 at 9:44 AM, Ian Stakenvicius <axs@gentoo.org> wrote:
>
> If the issue preventing protection is that the gpg signature only
> signs the hash, couldn't we just make repoman automatically add to the
> bottom of the comment a clearsign on the contents of the commit?
>

The gpg signature is on the entire contents of the "commit."  However,
the contents of the commit do not include the files that are being
committed - it includes hashes of the parent commit, the commit
message, other headers, and the hash of the tree being committed,
which is sha1.  That last hash is the only thing that ties the commit
to the files being committed, so you can modify the files all you like
as long as the sha1 is the same.

I don't think we should try to fix git.  It makes far more sense to
have upstream fix it.  I don't think we should hold up the migration
over it - NOBODY is holding off on adopting git over this stuff and
I'm not even aware of any projects that gpg sign their git commits.

Remember, the data model is:
commit --> tree --> [tree...] --> blob
The signature is against the commit, and sha1 hashes are what tie
everything else to it.

If you want to satisfy yourself I believe you can get git to dump the
contents of any object without formatting/etc.  You'll see that the
gpg signature matches the content of the commit (minus the gpg
signature header, of course).  If you directly access objects from the
filesystem I think git prepends a hash or something to the start of
every file.

--
Rich

Re: [gentoo-dev] git security (SHA-1)

[Michael Orlitzky]

On 09/16/2014 10:03 AM, Rich Freeman wrote:
> 
> The gpg signature is on the entire contents of the "commit."  However,
> the contents of the commit do not include the files that are being
> committed - it includes hashes of the parent commit, the commit
> message, other headers, and the hash of the tree being committed,
> which is sha1.  That last hash is the only thing that ties the commit
> to the files being committed, so you can modify the files all you like
> as long as the sha1 is the same.
> 

To put things in perspective, all I had to do was ask for commit access
and somebody eventually gave it to me. We should worry about this when
breaking SHA1 becomes less expensive than the ebuild quizzes.

Re: [gentoo-dev] git security (SHA-1)

[hasufell]

Michael Orlitzky:
> On 09/16/2014 10:03 AM, Rich Freeman wrote:
>>
>> The gpg signature is on the entire contents of the "commit."  However,
>> the contents of the commit do not include the files that are being
>> committed - it includes hashes of the parent commit, the commit
>> message, other headers, and the hash of the tree being committed,
>> which is sha1.  That last hash is the only thing that ties the commit
>> to the files being committed, so you can modify the files all you like
>> as long as the sha1 is the same.
>>
> 
> To put things in perspective, all I had to do was ask for commit access
> and somebody eventually gave it to me. We should worry about this when
> breaking SHA1 becomes less expensive than the ebuild quizzes.
> 
> 

Yep, that's what I'd try to do actually if I was working for NSA
(uh-oh). Try to get "collaborators" into every possible opensource project.

There are so many thing you can do... e.g. "fix" a security bug, but
reference a self-packaged tarball from your dev space (which still
contains the exploit) in the ebuild. No one will know.
And that's a pretty low hanging fruit.

Re: [gentoo-dev] git security (SHA-1)

[Aaron W. Swenson]

[-- Attachment #1: Type: text/plain, Size: 2190 bytes --]

On 2014-09-16 14:40, hasufell wrote:
> Michael Orlitzky:
> > To put things in perspective, all I had to do was ask for commit access
> > and somebody eventually gave it to me. We should worry about this when
> > breaking SHA1 becomes less expensive than the ebuild quizzes.
> 
> Yep, that's what I'd try to do actually if I was working for NSA
> (uh-oh). Try to get "collaborators" into every possible opensource project.
> 
> There are so many thing you can do... e.g. "fix" a security bug, but
> reference a self-packaged tarball from your dev space (which still
> contains the exploit) in the ebuild. No one will know.
> And that's a pretty low hanging fruit.
> 

This is what's been driving me batty. None of you verified my identity
before letting me be an official Gentoo Developer. Yet I have access to
the repo. All I had to do was complete the quizzes.

The real concern is restricting access to the master repository. If the
attacker has gained access, either by becoming a developer or some other
means, then we're only kind inconvenienced a little. We have to take the
system down for a bit, fix the problem, and replace the repo with a
trusted source or just roll it back to the last known good commit before
the good commit was made.

When Linus has talked about Git using SHA-1, the impression I got was
that it isn't a means of preventing attacks, but ensuring corruption
hasn't happened. When he talked about an attack to the kernel
repository, it was with BitKeeper, which used a much weaker hash, and
still thwarted an attack.

I also like what Pro Git has to say:
http://git-scm.com/book/ch6-1.html#A-SHORT-NOTE-ABOUT-SHA-1

It doesn't mention SHA-1 as a security feature, but that collissions are
effectively not a concern. Instead, we should be more concerned about us
all being dragged off into the night by wolves. Simultaneously.

Git hasn't promised to be secure against attacks. Just secure against
corruption. Two different things.

-- 
Mr. Aaron W. Swenson
Gentoo Linux Developer
PostgreSQL Herd Bull
Email : titanofold@gentoo.org
GnuPG FP : 2C00 7719 4F85 FB07 A49C 0E31 5713 AA03 D1BB FDA0
GnuPG ID : D1BBFDA0

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: [gentoo-dev] git security (SHA-1)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 264 bytes --]

On Wed, 17 Sep 2014 07:04:08 -0400
"Aaron W. Swenson" <titanofold@gentoo.org> wrote:
> This is what's been driving me batty. None of you verified my identity
> before letting me be an official Gentoo Developer.

Why does that matter?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] git security (SHA-1)

[Tim Boudreau]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

If someone wants to commit malicious code into Gentoo, they're far more
likely to take the ugly but pragmatic approach of, say, forcing someone to
commit malicious code at gunpoint and then shooting them, than to go to the
vast effort it would take to come up with malicious code that conveniently
has the same SHA-1 hash as an existing commit.

-Tim

[-- Attachment #2: Type: text/html, Size: 438 bytes --]

Re: [gentoo-dev] git security (SHA-1)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

On Wed, 17 Sep 2014 07:21:08 -0400
Tim Boudreau <niftiness@gmail.com> wrote:
> If someone wants to commit malicious code into Gentoo, they're far
> more likely to take the ugly but pragmatic approach of, say, forcing
> someone to commit malicious code at gunpoint and then shooting them,
> than to go to the vast effort it would take to come up with malicious
> code that conveniently has the same SHA-1 hash as an existing commit.

...or getting themselves recruited. Even easier.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] git security (SHA-1)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 746 bytes --]

Dnia 2014-09-17, o godz. 12:36:16
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> napisał(a):

> On Wed, 17 Sep 2014 07:21:08 -0400
> Tim Boudreau <niftiness@gmail.com> wrote:
> > If someone wants to commit malicious code into Gentoo, they're far
> > more likely to take the ugly but pragmatic approach of, say, forcing
> > someone to commit malicious code at gunpoint and then shooting them,
> > than to go to the vast effort it would take to come up with malicious
> > code that conveniently has the same SHA-1 hash as an existing commit.
> 
> ...or getting themselves recruited. Even easier.

I'm sorry but we are not allowed to go offtopic discussing
the recruitment problems on this list.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: git security (SHA-1)

[Duncan]

Ciaran McCreesh posted on Wed, 17 Sep 2014 12:36:16 +0100 as excerpted:

> On Wed, 17 Sep 2014 07:21:08 -0400 Tim Boudreau <niftiness@gmail.com>
> wrote:
>> If someone wants to commit malicious code into Gentoo, they're far more
>> likely to take the ugly but pragmatic approach of, say, forcing someone
>> to commit malicious code at gunpoint and then shooting them, than to go
>> to the vast effort it would take to come up with malicious code that
>> conveniently has the same SHA-1 hash as an existing commit.
> 
> ...or getting themselves recruited. Even easier.

Getting recruited, involves at minimum "playing the gentoo game" for 
several months, learning some otherwise esoteric knowledge, working up a 
number of patches that are found useful, and otherwise in general gaining 
the trust of existing devs over a period of some months.

The (arguably reasonable put potentially invalid?) assumption is that 
there's easier ways to do it, should someone feel it worth the trouble, 
so this hurdle is as high as is required.

OTOH, Gunpoint commits and then disappearing the committer involves a few 
days max, and with devs around the world, one will likely even be 
accessible within a local or near-local jurisdiction.


It's worth noting that even with air-tight gpg signing, what's validated 
is possession of the secret key, *NOT* the real identity of the signer, 
whatever key signing parties and key-holder ID verification may or may 
have occurred at some point in the past.

And I think most will agree that an argument that penetrating the 
security of /some/ dev, any one of the ~250-ish out there, and getting a 
copy of their secret signing key, is likely to be *WELL* within the 
resources of someone with rather less resources than the NSA, and is 
likely to be FAR less trouble than EITHER the gunpoint commit or "playing 
the gentoo game for N months" scenarios.  *THAT* is the one we have to 
worry about, as demonstrated by reason and recent history (the kernel.org 
hack a couple years ago, among others) both.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] git security (SHA-1)

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Wed, Sep 17, 2014 at 07:21:08AM -0400, Tim Boudreau wrote:
> If someone wants to commit malicious code into Gentoo, they're far more
> likely to take the ugly but pragmatic approach of, say, forcing someone to
> commit malicious code at gunpoint and then shooting them, than to go to the
> vast effort it would take to come up with malicious code that conveniently
> has the same SHA-1 hash as an existing commit.

But... what's the point? Upload ugly backdoor to all Gentoo users? (like
there're 0,01% of computer users out there?) It would be easier to just
gunpoint "the interesting user" or torture him for
keys/passwords/whatever in some creepy basement. This looks like someone
has a really bad fantasy about Gentoo ruling the world where every bad
guy/terrorist/younameit uses this super-secured-gpged-git-portage (and
looking at "Snowden files" all this is already exploited ;).


Piotr Szymaniak.
-- 
... wyobrazenie,  ze  ludzkosc  zmierza  ku  jakiemus  naprawde  milemu
przeznaczeniu,  jest bajka dla dzieci ponizej szostego roku zycia,  jak
Dobra Wrózka, Zajaczek Wielkanocny i Sw. Mikolaj.
  -- Kurt Vonnegut, "Hokus Pokus"

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] git security (SHA-1)

[Aaron W. Swenson]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

On 2014-09-17 12:08, Ciaran McCreesh wrote:
> On Wed, 17 Sep 2014 07:04:08 -0400
> "Aaron W. Swenson" <titanofold@gentoo.org> wrote:
> > This is what's been driving me batty. None of you verified my identity
> > before letting me be an official Gentoo Developer.
> 
> Why does that matter?

My argument is Git using SHA-1 for checksumming is not the weakest
part of our security model.

-- 
Mr. Aaron W. Swenson
Gentoo Linux Developer
PostgreSQL Herd Bull
Email : titanofold@gentoo.org
GnuPG FP : 2C00 7719 4F85 FB07 A49C 0E31 5713 AA03 D1BB FDA0
GnuPG ID : D1BBFDA0

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

[gentoo-dev] Re: git security (SHA-1)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

>>>>> On Wed, 17 Sep 2014, Aaron W. Swenson wrote:

> My argument is Git using SHA-1 for checksumming is not the weakest
> part of our security model.

I had always assumed that robbat2's series of GLEPs (57 to 61) would
be implemented at some point. So security from the developer to the
master repository would be ensured by using a secure channel for
commits, and distribution from the repository to users would use
secure hashes (SHA-256 or better) and gpg signatures.

I didn't see any mention of this in the discussion, though. Have these
plans been abandoned, and are we now planning to distribute the tree
to users via Git, where everything goes through the bottleneck of a
SHA-1 sum, which was never intended as a security feature? [1]

If this is so, why don't we abandon all those fancy SHA-512s and
WHIRLPOOLs in our Manifest files, and replace them by a single SHA-1?
Altogether, this would save about 50 MB of space in the tree. :)

Ulrich


[1] Tech Talk: Linus Torvalds on git: "[...] the point is the SHA-1,
as far as Git is concerned, isn't even a security feature. It's purely
a consistency check. [...] It has nothing at all to do with security,
it's just the best hash you can get."
https://www.youtube.com/watch?v=4XpnKHJAok8&t=56m20s

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[Rich Freeman]

On Sat, Sep 20, 2014 at 2:09 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Wed, 17 Sep 2014, Aaron W. Swenson wrote:
>
>> My argument is Git using SHA-1 for checksumming is not the weakest
>> part of our security model.
>
> I had always assumed that robbat2's series of GLEPs (57 to 61) would
> be implemented at some point. So security from the developer to the
> master repository would be ensured by using a secure channel for
> commits, and distribution from the repository to users would use
> secure hashes (SHA-256 or better) and gpg signatures.
>
> I didn't see any mention of this in the discussion, though. Have these
> plans been abandoned, and are we now planning to distribute the tree
> to users via Git, where everything goes through the bottleneck of a
> SHA-1 sum, which was never intended as a security feature? [1]
>
> If this is so, why don't we abandon all those fancy SHA-512s and
> WHIRLPOOLs in our Manifest files, and replace them by a single SHA-1?
> Altogether, this would save about 50 MB of space in the tree. :)

The issue is that we have lots of devs with lots of plans, and nothing
official whatsoever.  There is no formal plan to ever adopt those
GLEPs (I'm not saying it isn't likely to happen - just that there
isn't some kind of roadmap to actually implementing them).  There is
no plan to ever move to git, or not move to git.  There is no plan for
how any of this stuff is supposed to work a few years from now.

Gentoo isn't terribly big on planning - whether this is a good or bad
thing is perhaps debatable.  Typically whoever implements something
first tends to decide how it gets implemented.

It seems like most people involved in the git migration have planned
on moving to signed commits and thin manifests.  This would mean that
the chain of security would include dependence on sha1, since that is
the only thing that binds a singed commit to the data which was
committed (including the thin manifest file itself).

Then you get all the pile-on arguments that basically question whether
we care about the tree being signed anyway.

If this were some kind of corporation where we could make whatever
plans we wanted to make and pay people to make them happen whether
they liked the plans or not, then I would suggest the way we should go
about this is:
1.  Decide whether we really care about tree signing.
2.  If yes, no more arguing about whether we need tree signing.  Next
step is to decide what threat models we want to protect against.
3.  Then design something that actually provides security against the
targeted threat models.

The issue here is that we're basically starting at #3, with people all
over the map about #1-2, and thus it should be entirely unsurprising
that we invest in things like signed commits with nothing more than
sha1 to bind those signatures to files.

I'm personally in the camp that I'd rather see ANY git migration
happen sooner rather than later and I'd rather migrate first and then
fix any signature issues later.  Simple gpg signed commits secured
only with sha1 seems good enough to start with.

--
Rich

Re: [gentoo-dev] Re: git security (SHA-1)

[hasufell]

Ulrich Mueller:
> 
> Have these
> plans been abandoned, and are we now planning to distribute the tree
> to users via Git, where everything goes through the bottleneck of a
> SHA-1 sum, which was never intended as a security feature?
> 

This is a bug in git. Do you want us to wait until it is resolved?

Re: [gentoo-dev] Re: git security (SHA-1)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

>>>>> On Sat, 20 Sep 2014, hasufell  wrote:

>> Have these plans been abandoned, and are we now planning to
>> distribute the tree to users via Git, where everything goes through
>> the bottleneck of a SHA-1 sum, which was never intended as a
>> security feature?

> This is a bug in git. Do you want us to wait until it is resolved?

Not a bug. There are VCSs (like Subversion or Bazaar) that use simple
revision numbers to identify their commits. Git happens to use a hash,
which is perfectly fine as long as accidental collisions are unlikely.
Neither has to do anything with security, though.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[hasufell]

Ulrich Mueller:
>>>>>> On Sat, 20 Sep 2014, hasufell  wrote:
> 
>>> Have these plans been abandoned, and are we now planning to
>>> distribute the tree to users via Git, where everything goes through
>>> the bottleneck of a SHA-1 sum, which was never intended as a
>>> security feature?
> 
>> This is a bug in git. Do you want us to wait until it is resolved?
> 
> Not a bug. There are VCSs (like Subversion or Bazaar) that use simple
> revision numbers to identify their commits. Git happens to use a hash,
> which is perfectly fine as long as accidental collisions are unlikely.
> Neither has to do anything with security, though.
> 

Because there are other VCSs it is not a bug??

Of course it is a bug since it is in the gpg-signing chain and to use it
in a practical way is very unlikely.

So you are suggesting to not migrate at all or severely break the
workflow because someone might forge _working code_ with a specific
SHA1? There is no efficient algorithm for that afaik, those are just
about finding _any_ collision and even then it takes considerable
resources that can be used to break gentoo in much easier ways.

If you argue there might be someone who already found out more efficient
algorithms (and didn't publish them), then I hope you don't really
believe that using SHA256 will protect us from him.

Re: [gentoo-dev] Re: git security (SHA-1)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2230 bytes --]

On 21 September 2014 09:01, hasufell <hasufell@gentoo.org> wrote:

> Because there are other VCSs it is not a bug??
>
>
No, it just means "using SHA1 for making a repository work" is not a bug,
just like using "i am number 6, parent is number 5" is not a "security" bug.


> Of course it is a bug since it is in the gpg-signing chain and to use it
> in a practical way is very unlikely.
>
>
Its only a bug in that we're intending to use it for something it was not
designed for. SHA1s are not a security mechanism for Git.

GPGs are not entirely useless there, just we're taking more meaning from it
than it really supports.

I literally read GPG as being no more evidence than proof that, "yes, I
wrote that commit message, and I wrote that commit". It doesn't prove you
made any of those dependencies ( because some of those dependencies is gits
entire history of commits down the parent -> parent -> parent line )


> So you are suggesting to not migrate at all or severely break the
> workflow because someone might forge _working code_ with a specific
> SHA1? There is no efficient algorithm for that afaik, those are just
> about finding _any_ collision and even then it takes considerable
> resources that can be used to break gentoo in much easier ways.
>

He is proposing quite the opposite.  He's saying "git is not secure in this
way, but lets not let that stop us, migrate and fix that after the fact or
we'll never get around to it, because all this debate is the perfect being
the enemy of the good".

Git is still more than adequately secure without GPG to defend against a
whole bunch of attacks you'd need NSA grade stuff to attack as it is, and
GPG on the commits themselves basically rules out the easiest place
somebody *could* get things in without a GPG.

That is to say: without gpg, you can just create some random commit with
some arbitrary content and push it somewhere, and you can pretend you're a
gentoo dev and pretend you're writing commits as them.

GPG sufficiently prevents that from happening, and takes it from ameteur
grade imposter requirements to NSA grade imposter requirements. And that's
not a bad compromise for being imperfect.


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 3550 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[hasufell]

Kent Fredric:
> 
> He is proposing quite the opposite.  He's saying "git is not secure in this
> way, but lets not let that stop us, migrate and fix that after the fact or
> we'll never get around to it, because all this debate is the perfect being
> the enemy of the good".
> 

I didn't see him saying that. It rather sounds like we want to have
thick signed Manifests and break pull requests and whatnot.

Re: [gentoo-dev] Re: git security (SHA-1)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On 21 September 2014 09:18, hasufell <hasufell@gentoo.org> wrote:

> Kent Fredric:
> >
> > He is proposing quite the opposite.  He's saying "git is not secure in
> this
> > way, but lets not let that stop us, migrate and fix that after the fact
> or
> > we'll never get around to it, because all this debate is the perfect
> being
> > the enemy of the good".
> >
>
> I didn't see him saying that. It rather sounds like we want to have
> thick signed Manifests and break pull requests and whatnot.
>


<<<
I'm personally in the camp that I'd rather see ANY git migration
happen sooner rather than later and I'd rather migrate first and then
fix any signature issues later.  Simple gpg signed commits secured
only with sha1 seems good enough to start with.

--
Rich
>>>


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1605 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[hasufell]

You'v quoted the wrong guy.

Re: [gentoo-dev] Re: git security (SHA-1)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On 21 September 2014 09:18, hasufell <hasufell@gentoo.org> wrote:

> I didn't see him saying that. It rather sounds like we want to have
> thick signed Manifests and break pull requests and whatnot.
>

Those aren't the only options.

We could of course develop a custom commit signature system, either in the
commit itself, or using a custom ref protocol.

For instance, you could have an object in refs/signatures/<*> for every
blob in the tree, signed by the person who created that blob. But you'd
probably have to hook the git client somewhere low-level to make that
option work, and that way, those refs could be pulled only by people who
wanted them (Speed!), *and* they could be created after-the-fact.


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1548 bytes --]

[gentoo-dev] Re: git security (SHA-1)

[Duncan]

Kent Fredric posted on Sun, 21 Sep 2014 09:14:36 +1200 as excerpted:

> That is to say: without gpg, you can just create some random commit with
> some arbitrary content and push it somewhere, and you can pretend you're
> a gentoo dev and pretend you're writing commits as them.
> 
> GPG sufficiently prevents that from happening, and takes it from ameteur
> grade imposter requirements to NSA grade imposter requirements. And
> that's not a bad compromise for being imperfect.

I've seen this idea repeated several times in this thread and it bothers 
me.

In practice, gpg doesn't take it to NSA grade, even in theory it might.

The problem is this.  A gpg signature does *NOT* ensure that the person 
whose name is attached to a public/private key pair actually did the 
signature.  *ALL* it ensures is that someone with access to the 
particular private key in question signed the content.

Gpg doesn't know or care whether the person with that signing key is who 
they say they are or not.  All it knows/cares is that whoever they are, 
they have that key.  If the person who owned that key didn't keep the 
private half secure and someone else got ahold of it, game-over.  Until 
it's caught and the key revoked, that person can act with impunity as 
person-in-possession of that key.

Now realistically, gentoo has ~250 devs working in all sorts of different 
situations.  What is the change that NONE of those 250 people EVER lets 
someone else have access, whether due to letting them borrow the machine 
and then going to the restroom, or due to loss of laptop in a taxi or 
something, or due to malware?

IIRC the number of folks with kernel.org access was something similarly 
close to ~250 or so, before someone got their access creds stolen and 
kernel.org got hacked.

And as far as we know, that was *NOT* the NSA.  It was just some cracker 
group wanting access to good network bandwidth for their botnet, and they 
either didn't realize what they had or didn't know what to do with it 
once they realized it.

Basically, with ~250 devs, we can pretty much must assume that somebody's 
secret key is compromised at any point in time.  We don't know whose and 
we don't know whether it's even bad guys, not just some innocent that 
doesn't have the foggiest, and we might get lucky, but someone's key is 
either compromised at any particular point or relatively soon will be.  
With 250 devs out there living life, it's foolish to assume otherwise.


With 250 devs with signing keys and all of them having access to the 
entire tree, their humanity is the weak link, not SHA1.  SHA1 is a major 
exercise in unnecessary pain, compared to this weak link.  No NSA grade 
resources needed, and with 250 people out their spinning the roulette 
wheel of life every day that they aren't going to forget their laptop in 
a taxi somewhere, it's either already happened or it WILL happen.  That's 
a given.

Plus, even the NSA has their Edward Snowdens.  Perhaps it won't be some 
bad guy getting ahold of a key.  It's just as likely to be a "good" gentoo 
dev either turning bad, or never "good" in the first place.

So at least from here, all this worry about SHA1 is much ado about 
nothing.  The real worry is elsewhere.  Someone's has or will have 
unauthorized access to a signing key, and once they do it's simply a 
matter of chance whether they're a bad guy that knows what to do with 
it.  The real question is what systems we have in place to catch that and 
to stop-loss when we do detect it?  Because now or latter, it either has 
already happened or WILL happen.  We'd be foolish to assume anything else.

And git's not going to change that one bit.  Neither will all the signing 
and secure hashes in the world.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: git security (SHA-1)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1603 bytes --]

>>>>> On Sat, 20 Sep 2014, hasufell  wrote:

>>> This is a bug in git. Do you want us to wait until it is resolved?
>> 
>> Not a bug. There are VCSs (like Subversion or Bazaar) that use simple
>> revision numbers to identify their commits. Git happens to use a hash,
>> which is perfectly fine as long as accidental collisions are unlikely.
>> Neither has to do anything with security, though.

> Because there are other VCSs it is not a bug??

No, but with any other VCS we wouldn't have this discussion. Git using
SHA-1 obscures the fact that an additional security layer is needed.
This can be either a secure channel for accessing the repository
(developers pushing their commits to it), or signed Manifests that
ensure integrity of the tree distributed to users.

> Of course it is a bug since it is in the gpg-signing chain and to
> use it in a practical way is very unlikely.

> So you are suggesting to not migrate at all or severely break the
> workflow because someone might forge _working code_ with a specific
> SHA1? There is no efficient algorithm for that afaik, those are just
> about finding _any_ collision and even then it takes considerable
> resources that can be used to break gentoo in much easier ways.

Weakness of SHA-1 is discussed since several years, and it is
generally recommended that one should slowly move away from it.
Therefore I would find it strange if we (in 2014!) deployed a system
relying on it, while in our present Manifest files SHA-1 was already
abandoned long time ago, in favour of more secure hashes. It looks
like a move in the wrong direction.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[hasufell]

Ulrich Mueller:
>> So you are suggesting to not migrate at all or severely break the
>> workflow because someone might forge _working code_ with a specific
>> SHA1? There is no efficient algorithm for that afaik, those are just
>> about finding _any_ collision and even then it takes considerable
>> resources that can be used to break gentoo in much easier ways.
> 
> Weakness of SHA-1 is discussed since several years, and it is
> generally recommended that one should slowly move away from it.
> Therefore I would find it strange if we (in 2014!) deployed a system
> relying on it, while in our present Manifest files SHA-1 was already
> abandoned long time ago, in favour of more secure hashes. It looks
> like a move in the wrong direction.
> 

You are only talking about hashes, not about practical security.

Re: [gentoo-dev] Re: git security (SHA-1)

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

You're following the wrong train down the wrong tracks. Git [0-9a-f]{40} is
to CVS 1[.][1-9][0-9]+. You're arguing that CVS is more secure because its
commits are sequential numbers.

On Sat, Sep 20, 2014 at 4:20 PM, Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Sat, 20 Sep 2014, hasufell  wrote:
>
> >>> This is a bug in git. Do you want us to wait until it is resolved?
> >>
> >> Not a bug. There are VCSs (like Subversion or Bazaar) that use simple
> >> revision numbers to identify their commits. Git happens to use a hash,
> >> which is perfectly fine as long as accidental collisions are unlikely.
> >> Neither has to do anything with security, though.
>
> > Because there are other VCSs it is not a bug??
>
> No, but with any other VCS we wouldn't have this discussion. Git using
> SHA-1 obscures the fact that an additional security layer is needed.
> This can be either a secure channel for accessing the repository
> (developers pushing their commits to it), or signed Manifests that
> ensure integrity of the tree distributed to users.
>
> > Of course it is a bug since it is in the gpg-signing chain and to
> > use it in a practical way is very unlikely.
>
> > So you are suggesting to not migrate at all or severely break the
> > workflow because someone might forge _working code_ with a specific
> > SHA1? There is no efficient algorithm for that afaik, those are just
> > about finding _any_ collision and even then it takes considerable
> > resources that can be used to break gentoo in much easier ways.
>
> Weakness of SHA-1 is discussed since several years, and it is
> generally recommended that one should slowly move away from it.
> Therefore I would find it strange if we (in 2014!) deployed a system
> relying on it, while in our present Manifest files SHA-1 was already
> abandoned long time ago, in favour of more secure hashes. It looks
> like a move in the wrong direction.
>
> Ulrich
>

[-- Attachment #2: Type: text/html, Size: 2536 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[Rich Freeman]

On Sat, Sep 20, 2014 at 8:58 PM, Gordon Pettey <petteyg359@gmail.com> wrote:
> You're following the wrong train down the wrong tracks. Git [0-9a-f]{40} is
> to CVS 1[.][1-9][0-9]+. You're arguing that CVS is more secure because its
> commits are sequential numbers.

Ulrich is well-aware of that.  His argument is that with cvs there is
no security whatsoever in the scm, and so there is more interest in
layering security on-top.  With git there is more of a tendency to
rely on the less-than-robust commit signing system.

We could always just keep full manifests in the tree and be no worse
off than with cvs.

I sill think it makes more sense to start with a threat model and go
from there.  There are a lot of devs with a lot of keys and a lot of
steps on servers where the tree has to be manipulated.

One of the advantages of robust commit signing would be that in the
event there was a compromise it would be a lot easier to go back and
clean up the mess.

--
Rich

Re: [gentoo-dev] Re: git security (SHA-1)

[Peter Stuge]

Rich Freeman wrote:
> I sill think it makes more sense to start with a threat model and go
> from there.

I agree. It's a great project to tackle after the migration.

For migrating I agree with you that it's fine to do whatever.

Really. This year. As in before 2015. There are three more months.

I've so far gotten zero feedback on my hosting offer, intended to
help find some starting processes.


//Peter

Re: [gentoo-dev] Re: git security (SHA-1)

[Rich Freeman]

On Sat, Sep 20, 2014 at 9:27 PM, Peter Stuge <peter@stuge.se> wrote:
>
> I've so far gotten zero feedback on my hosting offer, intended to
> help find some starting processes.
>

hassufel's repository on github should be more than adequate:
https://github.com/gentoo/gentoo-gitmig

If we need history we can always substitute this with one with full history.

On a side note, my migration scripts take a few hours to run on a
fairly old Phenom II quad-core.  From the looks of things,
single-threaded performance seems to be what matters the most.  I did
try running a migration on EC2 for kicks, and this only reinforced the
need for good single-threaded performance (which is awful on EC2) -
with 32 cores it did make it through a majority of the tree in very
short order, but the longest categories (especially profiles) still
took something like 1.5 hours to convert and then the second stage of
the migration doesn't really use more than a core or two.  It was
faster than doing it on the old Phenom, but not by all that much.

If somebody has a beefy server they'd like to test a migration on let
me know.  I have a tarball of a chroot that is all set to run a
migration, and a squashfs of the cvs tree which just needs to be
mounted on top of it.  It is fairly trivial to transport, and in the
event of an actual migration it could be tested and then at the time
of the migration all you need is a current cvsroot to mount on top.

I suspect that more than 4 cores would be optimal, but I'd put the
highest priority on single-thread performance.  It didn't really seem
to be IO-bound on the writes (and this is on btrfs on spinning disks
which is less than stellar for performance), but if you have a large
number of cores IO might become a bottleneck.  I don't think IO reads
are going to be much of an issue at all - the starting tree is only a
few hundred MB in squashfs, and the disks didn't look busy for reads
in the later steps at all.

Even a few hours for such a huge migration isn't really that big a
deal I would think, but I've heard that on a good system this can be
cut down to under an hour (plus all the time moving trees around,
setting up back-end, etc).

--
Rich

Re: [gentoo-dev] Re: git security (SHA-1)

[Peter Stuge]

Rich Freeman wrote:
> > I've so far gotten zero feedback on my hosting offer, intended to
> > help find some starting processes.
> 
> hassufel's repository on github should be more than adequate:
> https://github.com/gentoo/gentoo-gitmig

The very first email in this thread pointed out that it is difficult
to do anything custom on github, so I offered to help because I'm
willing and able to help arrange hooks and scripting and whatever foo
may be needed to move on.


//Peter

Re: [gentoo-dev] Re: git security (SHA-1)

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 1242 bytes --]

On Sat, Sep 20, 2014 at 8:20 PM, Rich Freeman <rich0@gentoo.org> wrote:
On Sat, Sep 20, 2014 at 8:58 PM, Gordon Pettey <petteyg359@gmail.com> wrote:
> You're following the wrong train down the wrong tracks. Git [0-9a-f]{40}
is
> to CVS 1[.][1-9][0-9]+. You're arguing that CVS is more secure because its
> commits are sequential numbers.

Ulrich is well-aware of that.  His argument is that with cvs there is
no security whatsoever in the scm, and so there is more interest in
layering security on-top.  With git there is more of a tendency to
rely on the less-than-robust commit signing system.

On Sat, Sep 20, 2014 at 9:17 PM, Peter Stuge <peter@stuge.se> wrote:

> Rich Freeman wrote:
> > > I've so far gotten zero feedback on my hosting offer, intended to
> > > help find some starting processes.
> >
> > hassufel's repository on github should be more than adequate:
> > https://github.com/gentoo/gentoo-gitmig
>
> The very first email in this thread pointed out that it is difficult
> to do anything custom on github, so I offered to help because I'm
> willing and able to help arrange hooks and scripting and whatever foo
> may be needed to move on.
>

I can set up something in New York and/or Germany if additional hosts are
needed.

[-- Attachment #2: Type: text/html, Size: 1980 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

Dnia 2014-09-20, o godz. 21:20:34
Rich Freeman <rich0@gentoo.org> napisał(a):

> On Sat, Sep 20, 2014 at 8:58 PM, Gordon Pettey <petteyg359@gmail.com> wrote:
> > You're following the wrong train down the wrong tracks. Git [0-9a-f]{40} is
> > to CVS 1[.][1-9][0-9]+. You're arguing that CVS is more secure because its
> > commits are sequential numbers.  
> 
> Ulrich is well-aware of that.  His argument is that with cvs there is
> no security whatsoever in the scm, and so there is more interest in
> layering security on-top.  With git there is more of a tendency to
> rely on the less-than-robust commit signing system.
> 
> We could always just keep full manifests in the tree and be no worse
> off than with cvs.

And we would be no better off than with CVS. We'd have huge repository
with a lot of redundant space-eating data and the impossibility of sane
merges or rebases.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

>>>>> On Sun, 21 Sep 2014, Michał Górny wrote:

> Rich Freeman <rich0@gentoo.org> napisał(a):
>> Ulrich is well-aware of that.  His argument is that with cvs there
>> is no security whatsoever in the scm, and so there is more interest
>> in layering security on-top.  With git there is more of a tendency
>> to rely on the less-than-robust commit signing system.
>>
>> We could always just keep full manifests in the tree and be no
>> worse off than with cvs.

> And we would be no better off than with CVS. We'd have huge
> repository with a lot of redundant space-eating data and the
> impossibility of sane merges or rebases.

Not necessarily. As long as you keep write access to the repository
secure, you don't need anything special there. However, it's a
different story when the tree is distributed via a mirror system that
is not entirely under our control.

Full manifests could be generated automatically (and signed with an
infra key) when copying the tree from the repository to the master
mirror.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[hasufell]

Ulrich Mueller:
> 
> Full manifests could be generated automatically (and signed with an
> infra key) when copying the tree from the repository to the master
> mirror.
> 

Would you like to implement it?

Re: [gentoo-dev] Re: git security (SHA-1)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

Dnia 2014-09-21, o godz. 09:54:06
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Sun, 21 Sep 2014, Michał Górny wrote:
> 
> > Rich Freeman <rich0@gentoo.org> napisał(a):
> >> Ulrich is well-aware of that.  His argument is that with cvs there
> >> is no security whatsoever in the scm, and so there is more interest
> >> in layering security on-top.  With git there is more of a tendency
> >> to rely on the less-than-robust commit signing system.
> >>
> >> We could always just keep full manifests in the tree and be no
> >> worse off than with cvs.
> 
> > And we would be no better off than with CVS. We'd have huge
> > repository with a lot of redundant space-eating data and the
> > impossibility of sane merges or rebases.
> 
> Not necessarily. As long as you keep write access to the repository
> secure, you don't need anything special there. However, it's a
> different story when the tree is distributed via a mirror system that
> is not entirely under our control.
> 
> Full manifests could be generated automatically (and signed with an
> infra key) when copying the tree from the repository to the master
> mirror.

Do you really consider keeping a key open for machine signing somewhat
secure?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

>>>>> On Sun, 21 Sep 2014, Michał Górny wrote:

> Do you really consider keeping a key open for machine signing
> somewhat secure?

You mean, as compared to manifests (or commits) signed by 250
different developers' keys?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: git security (SHA-1)

[Alon Bar-Lev]

On Sun, Sep 21, 2014 at 2:13 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Sun, 21 Sep 2014, Michał Górny wrote:
>
>> Do you really consider keeping a key open for machine signing
>> somewhat secure?
>
> You mean, as compared to manifests (or commits) signed by 250
> different developers' keys?
>
> Ulrich

Unrelated to git discussion, in the past we discussed co-sign, so that
developer signs using short term key, and infra co-sign using long
term key if the developer sign is valid at that time. Portage infra
should relay on infra key signature, while tractability is available
up to developer.

I will take the opportunity of responding to write that my preference
is to keep the manifest signature detached from the version management
technology, with no git specific feature usage, nor git specific
development (signed hrefs). It will enable much easier use of each
technology, one for file management and the other for security, while
enabling rebase and reorg without effecting integrity. If we can
establish co-sign I will be very happy.

Regards,
Alon

Re: [gentoo-dev] Re: git security (SHA-1)

[hasufell]

Ulrich Mueller:
>>>>>> On Sun, 21 Sep 2014, Michał Górny wrote:
> 
>> Do you really consider keeping a key open for machine signing
>> somewhat secure?
> 
> You mean, as compared to manifests (or commits) signed by 250
> different developers' keys?
> 

That's the actual security problem in gentoo: 250 developers (which will
not be fixed by SHA256 and not by an infra key).

I think this discussion is derailing and unrelated to practical
security, but you keep talking _only_ about hashes instead of... well,
security which is not just about maths, but also about probability,
resources, configuration and project structure.

If you keep pushing into this direction without an implementation that
solves it then you will just have no one care about git migration any more.

Re: [gentoo-dev] Re: git security (SHA-1)

[Rich Freeman]

On Sat, Sep 20, 2014 at 4:40 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Sat, 20 Sep 2014, hasufell  wrote:
>
>>> Have these plans been abandoned, and are we now planning to
>>> distribute the tree to users via Git, where everything goes through
>>> the bottleneck of a SHA-1 sum, which was never intended as a
>>> security feature?
>
>> This is a bug in git. Do you want us to wait until it is resolved?
>
> Not a bug. There are VCSs (like Subversion or Bazaar) that use simple
> revision numbers to identify their commits. Git happens to use a hash,
> which is perfectly fine as long as accidental collisions are unlikely.
> Neither has to do anything with security, though.
>

Sure, but in that case why add gpg signatures to git at all?  I think
that just like Gentoo this is just the nature of FOSS - everybody has
their opinion of what everything is supposed to do, and the only thing
that really matters is what it can actually do and who writes code to
get it to do something different.

If Linus felt that git needed gpg signatures he'd have added them back
in the beginning (for commits, not tags).  Somebody else felt
differently, and added them later, but did not address the hash issue.
Maybe sometime down the road we'll see support for a different hash
functions added, or some other workaround.  All it would take is for
somebody to write the code and support it seriously (either as part of
git or a fork).

But, I doubt anybody here wants to maintain that fork, so we're left
with what we have now, which is a git which accepts signatures, but
those signatures are only bound to code by sha1.

With FOSS what is and isn't a bug is up to whoever wants to write the
code to fix it, or pay somebody else to write it for them.

--
Rich

Re: [gentoo-dev] git security (SHA-1)

[hasufell]

Aaron W. Swenson:
> 
> This is what's been driving me batty. None of you verified my identity
> before letting me be an official Gentoo Developer. Yet I have access to
> the repo. All I had to do was complete the quizzes.
> 

The only way to improve security in the sense of random collaborators is
to not grant them push access in the first place. This is almost going
offtopic since it still doesn't solve the attack vector this topic was
initially about. But our project model is definitely not up to date anymore.

Let me quote Bryan Østergaard in this context [0]:

> Other source based distributions follows a fairly closed development model that relies on a particular group of developers doing most, if not all the work and a somewhat complex organisation model that's supposed to help solve internal problems. The most common solution when technical problems (such as packages not getting timely updates) occurs is to add more developers to the organisation. Unfortunately this also tends to amplify any organisational problems.

Not just organisational problems, but also trust problems and QA
problems on top of that. If we want to improve this, we have to think
again and start a real review-based development model. This will mean
changing the whole gentoo project structure and use the benefits of git
to do it right. Anyone up for that? I guess not. You'd have to write up
10+ GLEPs to even try it, lol.


--
[0] https://archive.fosdem.org/2009/interview/bryan+ostergaard

Re: [gentoo-dev] git security (SHA-1)

[Peter Stuge]

Rich Freeman wrote:
> If you want to satisfy yourself I believe you can get git to dump
> the contents of any object without formatting/etc.

git ls-tree HEAD .
git show $blobhash
git show --pretty=raw HEAD


//Peter

Re: [gentoo-dev] git security (SHA-1)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

On 17 September 2014 01:44, Ian Stakenvicius <axs@gentoo.org> wrote:

> bottom of the comment a clearsign on the contents of the commit?
>


I don't see that being useful as written, because that's presently exactly
what git does.

the very best I think you could do is sign the commit *diff*, ie:
recursively compare TREE and PARENT->TREE and feed only *new* objects
through GPG. That's the most meaningful thing you'll get. ( And it really
is up-streams responsibility to implement this feature if they care,
bolting it on after the fact is horrible )

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1336 bytes --]

Re: [gentoo-dev] git security (SHA-1)

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 16/09/14 10:33 AM, Kent Fredric wrote:
> 
> On 17 September 2014 01:44, Ian Stakenvicius <axs@gentoo.org 
> <mailto:axs@gentoo.org>> wrote:
> 
>> bottom of the comment a clearsign on the contents of the commit?
> 
> [...] the very best I think you could do is sign the commit *diff*,
> 
Yes, that's what I meant by the contents of the commit.  Apologies for
not using proper terminology.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlQYTIMACgkQ2ugaI38ACPBQzQD8Cc/7i/Q7v0NVv0duVNkpDAER
1PDOT+TLIDbv9DG0YW4BAIGSSLb0dahPp6+bdbe9AUR63m9QIrw51TqL3lGpGsm8
=jEVx
-----END PGP SIGNATURE-----

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2509 bytes --]

On 15 September 2014 22:10, Jauhien Piatlicki <jauhien@gentoo.org> wrote:

> So signing of git commits does not guarantee enough security (taking
> that SHA1 is weak and can be broken), right? Could we than just use
> usual (not thin) manifests?
>

However, the attackability of SHA1 may be entirely immaterial, because
methods to exploit that require compromising other security strategies.

If somebody pushes signed commit 0x0001  with parents 0x0002 and 0x0003
with tree 0x0004 with files 0x0005 to 0x0010, those binary blobs are
pushed. And there is no way I know of to have those binary blobs replaced
with cuckoo blobs.

Once they're replicated, Git doesn't try re-replicating the same SHA1s.

So your attack vectors entail directly manipulating the git storage on
gentoo's servers, or poisoning a mirror under their control, or poisoning
the data *PRIOR* to it landing on gentoo servers, or being NSA and
poisoning it dynamically when a user attempts to fetch that specific SHA1.

None of these are impossible things, but they're much more complex than
"just make a dodgy commit and get somebody to pull it".

This basically means you could use CRC32 as your hash algorithm and still
pose a respectable problem for would-be attackers.

As such, I don't presently see git commit signing as a "Security" model,
merely a proof of authorship model. Anybody can forge commits with your
'author  = ' and 'committer = ', but you're the only person who can sign
the commit with your signature.

That is to say, you sign that you crafted the *commit*. But you're *not*
signing the creation of any of the dependencies.

For instance, two commits may have the same tree, but obviously only one
person forged that tree object.

And two trees may have the same file ( and indeed, this is an expected
element of how git works ), but you're not signing that you created all the
files in that tree. You may infer that from the chain of authority from the
commit itself, but it is not fact.

And parent objects are also dependencies, and nobody would ever consider
claiming they're a signing authority for all of that ;), that would be by
proxy signing the creation of the entire repository back to the first
commit ever forged! .... and its for that reason its probably good that git
doesn't presently recursively feed all dependencies of a commit into GPG. I
don't have 5 hours while every single blob in my repository is uncompressed
and fed through GPG :p


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 3579 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

On Mon, Sep 15, 2014 at 11:26:47PM +1200, Kent Fredric wrote:
> None of these are impossible things, but they're much more complex than
> "just make a dodgy commit and get somebody to pull it".

Much more simple would be to make a dodgy commit by one of the devs. Why
use users for that, if "the bad guy(/-s)" could be inside? Isn't that
the best way to poison stuff? Just look at the "let's move to git"
threads, there're bikeshedding and it's still cvs+rsync (maybe, just
maybe, this isn't a coincidence)


Piotr Szymaniak.
-- 
Mezczyzna  odlozyl gazete z powrotem na stojak i postapil krok w przod,
robiac  mine,  ktora nadala mu wyglad swinskiego pecherza rozpietego na
drucianym wieszaku do garniturow.
  -- Graham Masterton, "The Burning"

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1282 bytes --]

On Sun, Sep 14, 2014 at 07:13:21PM -0400, Rich Freeman wrote:
> The only thing that gets signed is the commit message, and the only
> thing that ties the commit message to the code is the sha1 of the
> top-level tree.  If you can attack sha1 either at any tree level or at
> the blob level you can defeat the signature.
> 
> That is way better than nothing though - I think it is worth pursuing
> until somebody comes up with a way to upgrade git to more secure
> hashes.  Most projects don't gpg sign their trees at all, including
> linux.

I'm not worried about the attack (as I explained earlier in this
thread).  I'm just arguing for signing first-parent commits to master,
and not worrying about signatures on any side-branch commits.  So long
as the merge gets signed, you've got all the security you're going to
get.  Leaving the side-branch commits unchanged allows you to preserve
any non-dev commit hashes, which makes it easier for contributors to
verify that their changes have landed (the same way that GitHub is
checking to know when to automatically close pull requests).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

On Sun, Sep 14, 2014 at 10:56:33PM +0000, hasufell wrote:
> W. Trevor King:
> > On Sun, Sep 14, 2014 at 10:38:41PM +0000, hasufell wrote:
> >> So we'd basically end up using either "git cherry-pick" or "git
> >> am" for "pulling" user stuff, so that we also sign the blobs.
> > 
> > Rebasing the original commits doesn't protect you from the
> > birthday attach either, because the vulnerable hash is likely
> > going to still be in the rebased commit's tree.  All rebasing does
> > is swap the committer and drop the initial signature.
> 
> According to Robin, it's not about rebasing, it's about signing all
> commits so that messing with the blob (even if it has the same
> sha-1) will cause signature verification failure.

All cherry-pick and am do is apply one commit's diff to a different
parent.  Changing the parent hash (which is stored in the commit body
[1]), so old signatures won't apply to the new commit.  If there have
been other tree changes between the initial parent and the new parent,
the tree hash will also change, which would also break old signatures.
None of that has anything to do with a malicious blob being pushed
into the tree disguised as a same-hashed good blob.  Such a blob will
*not* break any signatures, since GnuPG is *never hashing the blob
contents* when signing commits [1,2].  You're only signing the commit
object, not the tree and blob objects referenced by that commit.

Cheers,
Trevor

[1]: http://article.gmane.org/gmane.linux.gentoo.devel/77537
[2]: http://git.kernel.org/cgit/git/git.git/tree/commit.c?id=v2.1.0#n1076

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]

On 15 September 2014 11:15, W. Trevor King <wking@tremily.us> wrote:

> All cherry-pick and am do is apply one commit's diff to a different
> parent.  Changing the parent hash (which is stored in the commit body
> [1]), so old signatures won't apply to the new commit.  If there have
> been other tree changes between the initial parent and the new parent,
> the tree hash will also change, which would also break old signatures.
> None of that has anything to do with a malicious blob being pushed
> into the tree disguised as a same-hashed good blob.  Such a blob will
> *not* break any signatures, since GnuPG is *never hashing the blob
> contents* when signing commits [1,2].  You're only signing the commit
> object, not the tree and blob objects referenced by that commit.
>
> Cheers,
> Trevor
>


And given that the method of "security" against attacks is established by a
chain of custody from a signed commit, through multiple child unsigned SHA1
objects, having a parent being an unsigned commit is no *less* secure than
having a tree or file blob being unsigned, it doesn't make perfect sense to
me that "all" commits have to be signed.  ( Because doing so doesn't give
the benefit of security we think it does ).

Thus, a "I signed this commit, establishing a chain of trust relying on
SHA1 integrity to the previous signed commit" is all that seems truly
necessary. Anything else is decreased utility with no increase in security.


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 2299 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 609 bytes --]

On 15 September 2014 10:56, hasufell <hasufell@gentoo.org> wrote:

> According to Robin, it's not about rebasing, it's about signing all
> commits so that messing with the blob (even if it has the same sha-1)
> will cause signature verification failure.
>

Correct me if I'm wrong, but wouldn't a SHA1 attack on the tree object or
file blobs be completely invisible to the commit SHA1?

As the Signature only signs content of the commit object, not any of the
nodes it refers to.

Granted, getting a tree/file object to replicate might be interesting.

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1339 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Luca Barbato]

On 14/09/14 17:15, Kent Fredric wrote:
> On 15 September 2014 02:40, Michał Górny <mgorny@gentoo.org> wrote:
> 
>> However, I'm wondering if it would be possible to restrict people from
>> accidentally committing straight into github (e.g. merging pull
>> requests there instead of to our main server).
>>
> 
> 
> Easy.
> 
> Put the Gentoo repo in its own group.
> Don't give anyone any kinds of permissions on it.
> Have only one approved account for the purpose of pushing commits.
> Have a post-push hook that replicates to github as that approved account
> 
> => Github is just a read only mirror, any pull reqs submitted there will be
> fielded and pushed to gentoo directly.
> 
> Only downside there is the way github pull reqs work is if the final SHA1's
> that hit tree don't match, the pull req doesn't close.
> 
> Solutions:
> 
> - A) Have somebody tasked with reaping old pull reqs with permissions
> granted. ( Uck )
> - B) Always use a merge of some kind to mark the pull req as dead ( for
> instance, an "ours" merge to mark the branch as deprecated )

C) Ask nicely Github to have an application key and have a pull-request
bridge to avoid the problem completely.

I'd complete the migration first and discuss this kind of details later.

lu

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Davide Pesavento]

On Sun, Sep 14, 2014 at 2:03 PM, Michał Górny <mgorny@gentoo.org> wrote:
> We have main developer repo where developers work & commit and are
> relatively happy. For every push into developer repo, automated magic
> thingie merges stuff into user sync repo and updates the metadata cache
> there.

How long does the md5-cache regeneration process take? Are you sure it
will be able to keep up with the rate of pushes to the repo during
"peak hours"? If not, maybe we could use a time-based thing similar to
the current cvs->rsync synchronization.

[...]
> Main developer repo
> -------------------
>
> I was able to create a start git repository that takes around 66M
> as a git pack (this is how much you will have to fetch to start working
> with it). The repository is stripped clean of history and ChangeLogs,
> and has thin Manifests only.
>
> This means we don't have to wait till someone figures out the perfect
> way of converting the old CVS repository. You don't need that history
> most of the time, and you can play with CVS to get it if you really do.

+1

> In any case, we would likely strip the history anyway to get a small
> repo to work with.
>
> I have prepared a basic git update hook that keeps master clean
> and attached it to the bug [1]. It enforces basic policies, prevents
> forced updates and checks GPG signatures on left-most history line. It
> can also be extended to do more extensive tree checks.

Are we going to disallow merge commits and ask devs to rebase local
changes in order to keep the history "clean"?

Thanks a lot,
Davide

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Davide Pesavento:
>> Main developer repo
>> -------------------
>>
>> I was able to create a start git repository that takes around 66M
>> as a git pack (this is how much you will have to fetch to start working
>> with it). The repository is stripped clean of history and ChangeLogs,
>> and has thin Manifests only.
>>
>> This means we don't have to wait till someone figures out the perfect
>> way of converting the old CVS repository. You don't need that history
>> most of the time, and you can play with CVS to get it if you really do.
> 
> +1
> 

+1

>> In any case, we would likely strip the history anyway to get a small
>> repo to work with.
>>
>> I have prepared a basic git update hook that keeps master clean
>> and attached it to the bug [1]. It enforces basic policies, prevents
>> forced updates and checks GPG signatures on left-most history line. It
>> can also be extended to do more extensive tree checks.
> 
> Are we going to disallow merge commits and ask devs to rebase local
> changes in order to keep the history "clean"?
> 

I'd say it doesn't make sense to create merge commits for conflicts that
arise by someone having pushed earlier than you.

Merge commits should only be there if they give useful information.

Also... if you merge from a _user_ who is untrusted and allow a
fast-forward merge, then the signature verification fails. That means
for such pull requests you either have to use "git am" or "git merge
--no-ff".

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Davide Pesavento]

On Sun, Sep 14, 2014 at 3:55 PM, hasufell <hasufell@gentoo.org> wrote:
> Davide Pesavento:
>>> In any case, we would likely strip the history anyway to get a small
>>> repo to work with.
>>>
>>> I have prepared a basic git update hook that keeps master clean
>>> and attached it to the bug [1]. It enforces basic policies, prevents
>>> forced updates and checks GPG signatures on left-most history line. It
>>> can also be extended to do more extensive tree checks.
>>
>> Are we going to disallow merge commits and ask devs to rebase local
>> changes in order to keep the history "clean"?
>>
>
> I'd say it doesn't make sense to create merge commits for conflicts that
> arise by someone having pushed earlier than you.
>
> Merge commits should only be there if they give useful information.
>

I totally agree. But is there a way to automatically enforce this?

> Also... if you merge from a _user_ who is untrusted and allow a
> fast-forward merge, then the signature verification fails. That means
> for such pull requests you either have to use "git am" or "git merge
> --no-ff".
>

Right. In that case you can either sign the merge commit or amend the
user's commit and sign it yourself (re-signing could be needed anyway
if you have to rebase).

Thanks,
Davide

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2190 bytes --]

Dnia 2014-09-14, o godz. 15:40:06
Davide Pesavento <pesa@gentoo.org> napisał(a):

> On Sun, Sep 14, 2014 at 2:03 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > We have main developer repo where developers work & commit and are
> > relatively happy. For every push into developer repo, automated magic
> > thingie merges stuff into user sync repo and updates the metadata cache
> > there.
> 
> How long does the md5-cache regeneration process take? Are you sure it
> will be able to keep up with the rate of pushes to the repo during
> "peak hours"? If not, maybe we could use a time-based thing similar to
> the current cvs->rsync synchronization.

This strongly depends on how much data is there to update. A few
ebuilds are quite fast, eclass change isn't ;). I was thinking of
something along the lines of, in pseudo-code speaking:

  systemctl restart cache-regen

That is, we start the regen on every update. If it finishes in time, it
commits the new metadata. If another update occurs during regen, we
just restart it to let it catch the new data.

Of course, if we can't spare the resources to do intermediate updates,
we may as well switch to cron-based update method.

> [...]
> > In any case, we would likely strip the history anyway to get a small
> > repo to work with.
> >
> > I have prepared a basic git update hook that keeps master clean
> > and attached it to the bug [1]. It enforces basic policies, prevents
> > forced updates and checks GPG signatures on left-most history line. It
> > can also be extended to do more extensive tree checks.
> 
> Are we going to disallow merge commits and ask devs to rebase local
> changes in order to keep the history "clean"?

I don't think we should cripple git. Just to be clear, 'accidental'
merges won't happen because the automatic merges are unsigned
and the 'update' hook will refuse them.

The developers will have to either rebase and resign the commits, or
use a signed merge commit whichever makes more sense in particular
context.

Signed merge commits will also allow merging user-submitted changes
while preserving original history.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Tim Harder]

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]

On 2014-09-14 10:46, Michał Górny wrote:
> Dnia 2014-09-14, o godz. 15:40:06
> Davide Pesavento <pesa@gentoo.org> napisał(a):
> > How long does the md5-cache regeneration process take? Are you sure it
> > will be able to keep up with the rate of pushes to the repo during
> > "peak hours"? If not, maybe we could use a time-based thing similar to
> > the current cvs->rsync synchronization.
> 
> This strongly depends on how much data is there to update. A few
> ebuilds are quite fast, eclass change isn't ;). I was thinking of
> something along the lines of, in pseudo-code speaking:
> 
>   systemctl restart cache-regen
> 
> That is, we start the regen on every update. If it finishes in time, it
> commits the new metadata. If another update occurs during regen, we
> just restart it to let it catch the new data.
> 
> Of course, if we can't spare the resources to do intermediate updates,
> we may as well switch to cron-based update method.

I don't see per push metadata regen working entirely well in this case
if this is the only way we're generating the metadata cache for users to
sync. It's easy to imagine a plausible situation where a widely used
eclass change is made followed by commits less than a minute apart (or
shorter than however long it would take for metadata regen to occur) for
at least 30 minutes (rsync refresh period for most user-facing mirrors)
during a time of high activity.

I haven't run portage metadata regen on a beefy machine lately, but I
don't think it could keep up in all cases. Perhaps someone can prove me
wrong.

Anyway, things could definitely be sped up if portage merges a few speed
tweaks used in pkgcore. Specifically, I think using some of the weakref
and perhaps jitted attrs support along with the eclass caching hacks
would give a 2-4x metadata regen speedup. Otherwise pkgcore could
potentially be used to regen metadata as well or some other tuned regen
tool.

Tim

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

On 15 September 2014 13:30, Tim Harder <radhermit@gentoo.org> wrote:

> I haven't run portage metadata regen on a beefy machine lately, but I
> don't think it could keep up in all cases. Perhaps someone can prove me
> wrong.
>
> Anyway, things could definitely be sped up if portage merges a few speed
> tweaks used in pkgcore. Specifically, I think using some of the weakref
> and perhaps jitted attrs support along with the eclass caching hacks
> would give a 2-4x metadata regen speedup. Otherwise pkgcore could
> potentially be used to regen metadata as well or some other tuned regen
> tool.
>


I generate metadata for the perl-experimental overlay periodically as a
snapshotted variation of the same, and the performance isn't so bad.

However, what I suspect you *could* do with a push hook is regen metadata
for only things that were modified in that commit, because I believe
there's a way to regen metadata for only specific files now.

ie:
 modifications to cat/PN *would* trigger a metadata update, but only for
that cat/PN
 modifications to eclass/* would *NOT* trigger a metadata update as part of
the push.

And doing tree-wide "an eclass was changed" updates could be done with
lower priority in an asynchronous cron job or something so as not to block
workflow for several minutes/hours/whatever while some muppet sits there
watching "git push" do nothing.

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 2329 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Tim Harder]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On 2014-09-14 21:57, Kent Fredric wrote:
> I generate metadata for the perl-experimental overlay periodically as a
> snapshotted variation of the same, and the performance isn't so bad.

Overlays with few eclasses are much different than the main tree.
Anyway, egencache isn't bad it's just significantly slower than
alternatives so it could be sped up quite a lot if necessary.

> However, what I suspect you *could* do with a push hook is regen metadata
> for only things that were modified in that commit, because I believe
> there's a way to regen metadata for only specific files now.

> ie:
>  modifications to cat/PN *would* trigger a metadata update, but only for
> that cat/PN
>  modifications to eclass/* would *NOT* trigger a metadata update as part of
> the push.

> And doing tree-wide "an eclass was changed" updates could be done with
> lower priority in an asynchronous cron job or something so as not to block
> workflow for several minutes/hours/whatever while some muppet sits there
> watching "git push" do nothing.

If we need to do piecewise regen it seems we would be better off just
sticking with the current scheduled cron job approach. Otherwise it
sounds like one could pull updates without having the correct metadata
for a significant portion of the tree.

Tim

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2618 bytes --]

Dnia 2014-09-14, o godz. 21:30:36
Tim Harder <radhermit@gentoo.org> napisał(a):

> On 2014-09-14 10:46, Michał Górny wrote:
> > Dnia 2014-09-14, o godz. 15:40:06
> > Davide Pesavento <pesa@gentoo.org> napisał(a):
> > > How long does the md5-cache regeneration process take? Are you sure it
> > > will be able to keep up with the rate of pushes to the repo during
> > > "peak hours"? If not, maybe we could use a time-based thing similar to
> > > the current cvs->rsync synchronization.
> > 
> > This strongly depends on how much data is there to update. A few
> > ebuilds are quite fast, eclass change isn't ;). I was thinking of
> > something along the lines of, in pseudo-code speaking:
> > 
> >   systemctl restart cache-regen
> > 
> > That is, we start the regen on every update. If it finishes in time, it
> > commits the new metadata. If another update occurs during regen, we
> > just restart it to let it catch the new data.
> > 
> > Of course, if we can't spare the resources to do intermediate updates,
> > we may as well switch to cron-based update method.
> 
> I don't see per push metadata regen working entirely well in this case
> if this is the only way we're generating the metadata cache for users to
> sync. It's easy to imagine a plausible situation where a widely used
> eclass change is made followed by commits less than a minute apart (or
> shorter than however long it would take for metadata regen to occur) for
> at least 30 minutes (rsync refresh period for most user-facing mirrors)
> during a time of high activity.

For a metadata recheck (that is, egencache run with no changes):

a. cold cache ext4:

  real    3m54.321s
  user    0m44.413s
  sys     0m13.497s

b. warm cache ext4:

  real    0m40.672s
  user    0m35.087s
  sys     0m 4.687s

I will try to re-run that on btrfs or reiserfs to get a more meaningful
numbers.

Now, that results back up your claims. However, if we can get that to
<10s, I doubt we would have a major issue. My idea works like this:

1. first update is pushed,
1a. egencache starts rechecking and updating cache,
2. second update is pushed,
2a. previous egencache is terminated,
2b. egencache starts rechecking and updating cache,
2c. egencache finishes in time and commits.

The point is, nothing gets committed to the user-reachable location
before egencache finishes. And it goes quasi-incrementally, so if
another update happens before egencache finished, it only does
the 'slow' regen on changed metadata.

I will come back with more results soon.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Luca Barbato]

On 14/09/14 16:46, Michał Górny wrote:
> Of course, if we can't spare the resources to do intermediate updates,
> we may as well switch to cron-based update method.

The mirror have a sync time, so basically regenerating the cache and
pushing the tree with further toward the user can happen the same way
w/out impacting anybody.

We could just snapshot the tree when the regen starts and push that
commit to the user git and rsync.

People is still supposed to play nice and sync not every minute.

lu

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

Dnia 2014-09-16, o godz. 19:05:18
Luca Barbato <lu_zero@gentoo.org> napisał(a):

> On 14/09/14 16:46, Michał Górny wrote:
> > Of course, if we can't spare the resources to do intermediate updates,
> > we may as well switch to cron-based update method.
> 
> The mirror have a sync time, so basically regenerating the cache and
> pushing the tree with further toward the user can happen the same way
> w/out impacting anybody.
> 
> We could just snapshot the tree when the regen starts and push that
> commit to the user git and rsync.
> 
> People is still supposed to play nice and sync not every minute.

People don't have to sync. They will pull, and pulling often doesn't
really hurt servers like rsync does.

Of course, I'm considering the users switching to git there. However,
I don't think limitations of rsync should impact them.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Patrick Lauer]

On Sunday 14 September 2014 15:40:06 Davide Pesavento wrote:
> On Sun, Sep 14, 2014 at 2:03 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > We have main developer repo where developers work & commit and are
> > relatively happy. For every push into developer repo, automated magic
> > thingie merges stuff into user sync repo and updates the metadata cache
> > there.
> 
> How long does the md5-cache regeneration process take? Are you sure it
> will be able to keep up with the rate of pushes to the repo during
> "peak hours"? If not, maybe we could use a time-based thing similar to
> the current cvs->rsync synchronization.

Best case only one package is affected - a few seconds
Worst case someone touches an eclass like eutils, then it expands to something 
on the order of one or two CPU-hours.
 
> Are we going to disallow merge commits and ask devs to rebase local
> changes in order to keep the history "clean"?

Is that going to be sane with our commit frequency?

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Patrick Lauer:
>> Are we going to disallow merge commits and ask devs to rebase local
>> changes in order to keep the history "clean"?
> 
> Is that going to be sane with our commit frequency?
> 

You have to merge or rebase anyway in case of a push conflict, so the
only difference is the method and the effect on the history.

Currently... CVS allows you to run repoman on an outdated tree and push
broken ebuilds with repoman being happy. Git will not allow this.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Sun, Sep 14, 2014 at 11:42 AM, hasufell <hasufell@gentoo.org> wrote:
> Patrick Lauer:
>>> Are we going to disallow merge commits and ask devs to rebase local
>>> changes in order to keep the history "clean"?
>>
>> Is that going to be sane with our commit frequency?
>>
>
> You have to merge or rebase anyway in case of a push conflict, so the
> only difference is the method and the effect on the history.
>
> Currently... CVS allows you to run repoman on an outdated tree and push
> broken ebuilds with repoman being happy. Git will not allow this.
>

Repoman is going to be a challenge here.  With cvs every package is
its own private repository with its own private history and cvs only
cares if there is a collision within the scope of a single file.

With git your commit is against the whole tree.  So, even though it is
trivial to merge, independent commits against two different packages
do collide and need to be rebased or merged.

Repoman can run against a single package fairly quickly, so assuming
we still allow that we could do a pull/rebase/repman/push workflow
even if people are doing commits every few minutes.  On the other
hand, if you're doing a package move or eclass change or some other
change that affects 300 packages, just doing the rebase might cost you
a few minutes (due to actual collisions), and running repoman against
the whole thing before doing a push isn't going to be practical.
Somebody doing a tree-wide commit would almost certainly have to run
repoman before the final rebase/merge, push that out, and then maybe
do another repoman after-the-fact and maybe clean up any issues.  For
all intents in purposes that is what we're doing today anyway, since
repoman+cvs doesn't offer any kind of tree-wide consistency guarantees
unless you're checking out based on a timestamp or something like
that.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Patrick Lauer]

On Sunday 14 September 2014 15:42:15 hasufell wrote:
> Patrick Lauer:
> >> Are we going to disallow merge commits and ask devs to rebase local
> >> changes in order to keep the history "clean"?
> > 
> > Is that going to be sane with our commit frequency?
> 
> You have to merge or rebase anyway in case of a push conflict, so the
> only difference is the method and the effect on the history.
> 
> Currently... CVS allows you to run repoman on an outdated tree and push
> broken ebuilds with repoman being happy. Git will not allow this.

iow, git doesn't allow people to work on more than one item at a time?

That'd mean I need half a dozen checkouts just to emulate cvs, which somehow 
doesn't make much sense to me ...

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

On 15 September 2014 11:21, Patrick Lauer <patrick@gentoo.org> wrote:

> iow, git doesn't allow people to work on more than one item at a time?
>
> That'd mean I need half a dozen checkouts just to emulate cvs, which
> somehow
> doesn't make much sense to me ...
>

Use the Stash. Or just commit items, then swap branches, and then discard
the commits sometime later before pushing.

Unlike CVS, git doesn't force you to work in "Keep millions of files in
uncommitted states" mode just to work on a codebase, due to the commit <->
replicate seperation.


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1324 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Patrick Lauer]

On Monday 15 September 2014 11:27:34 Kent Fredric wrote:
> On 15 September 2014 11:21, Patrick Lauer <patrick@gentoo.org> wrote:
> > iow, git doesn't allow people to work on more than one item at a time?
> > 
> > That'd mean I need half a dozen checkouts just to emulate cvs, which
> > somehow
> > doesn't make much sense to me ...
> 
> Use the Stash. Or just commit items, then swap branches, and then discard
> the commits sometime later before pushing.
> 
> Unlike CVS, git doesn't force you to work in "Keep millions of files in
> uncommitted states" mode just to work on a codebase, due to the commit <->
> replicate seperation.
But that's the feature!

I can work on bumping postgresql (takes about 1h walltime to compile and test 
all versions) *and* work on a few tiny python packages while doing that. 
Without breaking either process. Without multiple checkouts.

I doubt stash would allow things to progress ... but it's a cute idea.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Patrick Lauer:
> On Monday 15 September 2014 11:27:34 Kent Fredric wrote:
>> On 15 September 2014 11:21, Patrick Lauer <patrick@gentoo.org> wrote:
>>> iow, git doesn't allow people to work on more than one item at a time?
>>>
>>> That'd mean I need half a dozen checkouts just to emulate cvs, which
>>> somehow
>>> doesn't make much sense to me ...
>>
>> Use the Stash. Or just commit items, then swap branches, and then discard
>> the commits sometime later before pushing.
>>
>> Unlike CVS, git doesn't force you to work in "Keep millions of files in
>> uncommitted states" mode just to work on a codebase, due to the commit <->
>> replicate seperation.
> But that's the feature!
> 
> I can work on bumping postgresql (takes about 1h walltime to compile and test 
> all versions) *and* work on a few tiny python packages while doing that. 
> Without breaking either process. Without multiple checkouts.
> 
> I doubt stash would allow things to progress ... but it's a cute idea.
> 

Please read up about git branches.

I don't see anything particularly broken. People use git to work on 10+
different feature at a time. It works.

Also, let's not derail this thread to git vs CVS, thanks.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Peter Stuge]

Patrick Lauer wrote:
> > > That'd mean I need half a dozen checkouts just to emulate cvs, which
> > > somehow doesn't make much sense to me ...
> > 
> > Unlike CVS, git doesn't force you to work in "Keep millions of files in
> > uncommitted states" mode just to work on a codebase, due to the commit <->
> > replicate seperation.
> 
> But that's the feature!

You can have millions of uncommitted files with git too. The person
who creates a commit always decides what changes in what files should
be included in that commit. (You don't even have to commit all the
changes within one file at the same time.)

There are some shortcuts for committing all uncommitted changes at
once but you don't have to do that. I frequently only commit little
bits of my currently uncommitted changes.


> I can work on bumping postgresql (takes about 1h walltime to compile and test 
> all versions) *and* work on a few tiny python packages while doing that. 
> Without breaking either process. Without multiple checkouts.

Same with git.


//Peter

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Patrick Lauer:
> On Sunday 14 September 2014 15:42:15 hasufell wrote:
>> Patrick Lauer:
>>>> Are we going to disallow merge commits and ask devs to rebase local
>>>> changes in order to keep the history "clean"?
>>>
>>> Is that going to be sane with our commit frequency?
>>
>> You have to merge or rebase anyway in case of a push conflict, so the
>> only difference is the method and the effect on the history.
>>
>> Currently... CVS allows you to run repoman on an outdated tree and push
>> broken ebuilds with repoman being happy. Git will not allow this.
> 
> iow, git doesn't allow people to work on more than one item at a time?
> 

Completely the opposite. You can work on 400 packages, accumulate the
changes, commit them and push them in one blow instead of writing
fragile scripts or Makefiles that do >400 pushes, fail at some point in
the middle because of a conflict and then try to figure out what you
already pushed and what not.

> That'd mean I need half a dozen checkouts just to emulate cvs, which somehow 
> doesn't make much sense to me ...
> 

checkouts? You probably mean that you have to rebase your changes in
case someone pushed before you. That makes perfect sense, because the
ebuild you just wrote might be broken by now, because someone changed
profiles/.

We are talking about a one-liner in the shell that will work in the
majority of the cases. If it doesn't work (as in: merge conflict), then
that means there is something REALLY wrong and 2 people are working
uncoordinated on the same file at a time.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Sun, Sep 14, 2014 at 7:21 PM, Patrick Lauer <patrick@gentoo.org> wrote:
>
> iow, git doesn't allow people to work on more than one item at a time?
>
> That'd mean I need half a dozen checkouts just to emulate cvs, which somehow
> doesn't make much sense to me ...
>

Well, you can work on as many things as you like in git, but it
doesn't keep track of what changes have to do with what things if you
don't commit in-between.  So, you'll have a big list of changes in
your index, and you'll have to pick-and-choose what you commit at any
one time.

If you really want to work on many things "at once" the better way to
do it is to do a temporary branch per-thing, and when you switch
between things you switch between branches, and then move into master
things as they are done.

I assume you mean working on things that will take a while to
complete.  If you just want to do 15 standalone commits before you
push you can do those sequentially easily enough.  A branch would be
more appropriate for some kind of mini-project.

You can work on branches without pushing those to the master repo.
Or, if appropriate a project team might choose to push their branch to
master, or to some other repo (like an overlay).  This would allow
collaborative work on a large commit, with a quick final merge into
the main tree.  That is the beauty of git - branches are really cheap.
So are repositories - if somebody wants to do all their work in github
and then push to the main tree, they can do that.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Peter Stuge]

Rich Freeman wrote:
> If you just want to do 15 standalone commits before you push you can
> do those sequentially easily enough.  A branch would be more
> appropriate for some kind of mini-project.
..
> That is the beauty of git - branches are really cheap.
> So are repositories

And commits.

Not only are branches cheap, they are also very easy to create, and
maybe most importantly they can be created at any time, even after
the commits.

It's quick and painless to create a bunch of commits which aren't
really closely related in sequence, and only later clean the whole
series of commits up while creating different branches for commits
which should actually be grouped rather than mixed all together.


//Peter

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On 15 September 2014 13:06, Peter Stuge <peter@stuge.se> wrote:

> even after
> the commits.
>

I've even made branches in "detached head" state ( that is, without a
branch ) and given them branches after the fact.

After all, branches aren't really "things", they're just pointers to SHA1s,
that get repointed to new sha1's as part of "git commit".

Tags are also simply pointers, they just don't get updated by default.

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1188 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 14/09/14 09:06 PM, Peter Stuge wrote:
> Rich Freeman wrote:
>> If you just want to do 15 standalone commits before you push you
>> can do those sequentially easily enough.  A branch would be more 
>> appropriate for some kind of mini-project.
> ..
>> That is the beauty of git - branches are really cheap. So are
>> repositories
> 
> And commits.
> 
> Not only are branches cheap, they are also very easy to create,
> and maybe most importantly they can be created at any time, even
> after the commits.
> 
> It's quick and painless to create a bunch of commits which aren't 
> really closely related in sequence, and only later clean the whole 
> series of commits up while creating different branches for commits 
> which should actually be grouped rather than mixed all together.
> 

Ahh, so the secret here would then be just to git add files on a
related per-package basis, leaving the other files out of the commit.
 that makes sense.  There would still be the issue of untracked files
in the repo and the ability to switch back to the 'master' branch to
cherry-pick a commit for pushing, though...  I guess we'd just have to
deal with the delay there and try and push all of the changes at once?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlQXJQkACgkQ2ugaI38ACPBFBQD/Z1SYvajcf/WORxknJGu1VfI0
f8CFhMTdE34Bk0Zd+GoA/iJtwsYBUQQHXhRjs7AzQDxaIEuFRgzyUgee4BICKaiq
=8fbP
-----END PGP SIGNATURE-----

Re: [gentoo-dev] git basics

[hasufell]

Ian Stakenvicius:
> On 14/09/14 09:06 PM, Peter Stuge wrote:
>> Rich Freeman wrote:
>>> If you just want to do 15 standalone commits before you push you
>>> can do those sequentially easily enough.  A branch would be more
>>> appropriate for some kind of mini-project.
>> ..
>>> That is the beauty of git - branches are really cheap. So are
>>> repositories
> 
>> And commits.
> 
>> Not only are branches cheap, they are also very easy to create,
>> and maybe most importantly they can be created at any time, even
>> after the commits.
> 
>> It's quick and painless to create a bunch of commits which aren't
>> really closely related in sequence, and only later clean the whole
>> series of commits up while creating different branches for commits
>> which should actually be grouped rather than mixed all together.
> 
> 
> Ahh, so the secret here would then be just to git add files on a
> related per-package basis, leaving the other files out of the commit.
>  that makes sense.  There would still be the issue of untracked files
> in the repo and the ability to switch back to the 'master' branch to
> cherry-pick a commit for pushing, though...  I guess we'd just have to
> deal with the delay there and try and push all of the changes at once?
> 

First, I don't think this is related to mgornys migration plans anymore.

Second: yes, in contrary to CVS it will _never_ happen on git that you
accidentally commit random files because of a "dirty" tree (and that
happened to me as well on CVS... I didn't even know until I ran repoman
on a whole category by accident! oops).

There are numerous of ways to work on multiple things at once in git:
1) Just use different branches and sync them if necessary.

2) Don't stage everything you have modified... I guess a lot of people
don't really know that it's possible? As pointed out before, you can
even say which hunks/lines of a single file you want to have staged.

Furthermore, you don't even need to stash anything, even if you switch
branches, as long as the (un)staged changes don't collide with the
branch you are switching to (and if that is the case... then you have
bigger problems).

3) Although this is anti-git, you could still just copy paste the
repository directory. Every git repository is consistent in itself and
fully functional. You don't directly depend on any server. And you can
even pull from local git repositories. If you find this silly, do 2) or 1).

Re: [gentoo-dev] git basics

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 15/09/14 02:35 PM, hasufell wrote:
> Ian Stakenvicius:
>> On 14/09/14 09:06 PM, Peter Stuge wrote:
>>> Rich Freeman wrote:
>>>> If you just want to do 15 standalone commits before you push
>>>> you can do those sequentially easily enough.  A branch would
>>>> be more appropriate for some kind of mini-project.
>>> ..
>>>> That is the beauty of git - branches are really cheap. So
>>>> are repositories
>> 
>>> And commits.
>> 
>>> Not only are branches cheap, they are also very easy to
>>> create, and maybe most importantly they can be created at any
>>> time, even after the commits.
>> 
>>> It's quick and painless to create a bunch of commits which
>>> aren't really closely related in sequence, and only later clean
>>> the whole series of commits up while creating different
>>> branches for commits which should actually be grouped rather
>>> than mixed all together.
>> 
>> 
>> Ahh, so the secret here would then be just to git add files on a 
>> related per-package basis, leaving the other files out of the
>> commit. that makes sense.  There would still be the issue of
>> untracked files in the repo and the ability to switch back to the
>> 'master' branch to cherry-pick a commit for pushing, though...  I
>> guess we'd just have to deal with the delay there and try and
>> push all of the changes at once?
>> 
> 
> First, I don't think this is related to mgornys migration plans
> anymore.
> 

Completely agree; it may not even be on-topic for the git workflow
thread..

> Second: yes, in contrary to CVS it will _never_ happen on git that
> you accidentally commit random files because of a "dirty" tree (and
> that happened to me as well on CVS... I didn't even know until I
> ran repoman on a whole category by accident! oops).
> 
> There are numerous of ways to work on multiple things at once in
> git: 1) Just use different branches and sync them if necessary.
> 
> 2) Don't stage everything you have modified... I guess a lot of
> people don't really know that it's possible? As pointed out before,
> you can even say which hunks/lines of a single file you want to
> have staged.
> 
> Furthermore, you don't even need to stash anything, even if you
> switch branches, as long as the (un)staged changes don't collide
> with the branch you are switching to (and if that is the case...
> then you have bigger problems).
> 

My concern/question is not so much how to deal with different things
concurrently, as it is how to deal with having to do distinct git
operations concurrently.

Repeating my example, say i'm working on a new release of firefox, it
takes ~40 mins to compile and there's some stuff it needs to do with
files in ${FILESDIR} during src_install.  So i'm 'ebuild ...
install'ing that.  In the meantime, there's a high-priority fix that
came up that I have to do in say, eudev.  Unless i'm doing my firefox
work in the master branch locally (and probably am running these tests
on modified-but-not-committed files), i can't just switch branches to
do/test/commit/push while ebuild does its thing in the background.  Or
can I?  I'd have to resort to (3), right?

> 3) Although this is anti-git, you could still just copy paste the 
> repository directory. Every git repository is consistent in itself
> and fully functional. You don't directly depend on any server. And
> you can even pull from local git repositories. If you find this
> silly, do 2) or 1).
> 

If it were possible to just start up a new shell, cd /usr/portage.git;
git checkout master ; [..do stuff..]  while the current workspace in
my old shell remains consistent, all would be fine.  But so far as i
(and everyone currently active in #-dev) are aware there's no built-in
way to do that right now; the 'git' way would be to clone local repos,
yes?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlQXPgcACgkQ2ugaI38ACPCJFQD+JfDz8KfuRb5+Ybij11fAOBQv
75jah41LD1o2F7AvH4oA/3UAqtO90XS5nin+3o1qU8gfDlL6FPn5E9YGDEPtnQ6f
=Eo0Y
-----END PGP SIGNATURE-----

Re: [gentoo-dev] git basics

[hasufell]

Ian Stakenvicius:
> 
> Repeating my example, say i'm working on a new release of firefox, it
> takes ~40 mins to compile and there's some stuff it needs to do with
> files in ${FILESDIR} during src_install.  So i'm 'ebuild ...
> install'ing that.  In the meantime, there's a high-priority fix that
> came up that I have to do in say, eudev.  Unless i'm doing my firefox
> work in the master branch locally (and probably am running these tests
> on modified-but-not-committed files), i can't just switch branches to
> do/test/commit/push while ebuild does its thing in the background.  Or
> can I?  I'd have to resort to (3), right?
> 

1. git checkout -b eudev-tmp (create new branch which has the firefox
changes you just made and switch to it)
2. git reset --mixed HEAD~1 (dump the firefox patch back to the unstaged
area, the file is still the same!)
3. fiddle around with eudev, commit and push
4. git checkout -f master (switch back to the master branch and throw
away the unstaged changes in eudev-tmp, however... file is still the same)

Alternatives:
Copy over the firefox ebuild to your local overlay before doing
extensive tests and fiddling. That's what I'd do anyway instead of
working directly on the tree.

Re: [gentoo-dev] git basics

[hasufell]

hasufell:
> Ian Stakenvicius:
>>
>> Repeating my example, say i'm working on a new release of firefox, it
>> takes ~40 mins to compile and there's some stuff it needs to do with
>> files in ${FILESDIR} during src_install.  So i'm 'ebuild ...
>> install'ing that.  In the meantime, there's a high-priority fix that
>> came up that I have to do in say, eudev.  Unless i'm doing my firefox
>> work in the master branch locally (and probably am running these tests
>> on modified-but-not-committed files), i can't just switch branches to
>> do/test/commit/push while ebuild does its thing in the background.  Or
>> can I?  I'd have to resort to (3), right?
>>
> 
> 1. git checkout -b eudev-tmp (create new branch which has the firefox
> changes you just made and switch to it)
> 2. git reset --mixed HEAD~1 (dump the firefox patch back to the unstaged
> area, the file is still the same!)
> 3. fiddle around with eudev, commit and push
> 4. git checkout -f master (switch back to the master branch and throw
> away the unstaged changes in eudev-tmp, however... file is still the same)
> 
> Alternatives:
> Copy over the firefox ebuild to your local overlay before doing
> extensive tests and fiddling. That's what I'd do anyway instead of
> working directly on the tree.
> 

That said... I find that workflow a bit weird. Why is the firefox ebuild
already committed if it hasn't even been tested?

And if it's not committed... then we don't even have a problem here,
because switching branches will just work.

Re: [gentoo-dev] git basics

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 15/09/14 04:31 PM, hasufell wrote:
> hasufell:
>> Ian Stakenvicius:
>>> 
>>> Repeating my example, say i'm working on a new release of
>>> firefox, it takes ~40 mins to compile and there's some stuff it
>>> needs to do with files in ${FILESDIR} during src_install.  So
>>> i'm 'ebuild ... install'ing that.  In the meantime, there's a
>>> high-priority fix that came up that I have to do in say, eudev.
>>> Unless i'm doing my firefox work in the master branch locally
>>> (and probably am running these tests on
>>> modified-but-not-committed files), i can't just switch branches
>>> to do/test/commit/push while ebuild does its thing in the
>>> background.  Or can I?  I'd have to resort to (3), right?
>>> 
>> 
>> 1. git checkout -b eudev-tmp (create new branch which has the
>> firefox changes you just made and switch to it) 2. git reset
>> --mixed HEAD~1 (dump the firefox patch back to the unstaged area,
>> the file is still the same!) 3. fiddle around with eudev, commit
>> and push 4. git checkout -f master (switch back to the master
>> branch and throw away the unstaged changes in eudev-tmp,
>> however... file is still the same)
>> 

It's generally considered safe to push to origin/master a commit from
a temporary local branch?  how would that end up working with the
repoman->push->[fail]->pull cycle due to possible non-fast-forward
conflicts?  how difficult will it be to ensure local history isn't all
messed up, later on?

>> Alternatives: Copy over the firefox ebuild to your local overlay
>> before doing extensive tests and fiddling. That's what I'd do
>> anyway instead of working directly on the tree.
>> 

Yes, that's always a possibility as well -- having a second copy of
the tree, pulling (local) changes from the working copy and
merge-testing from that would reduce most of these concurrency
issues..  Probably worthwhile suggesting this in the git workflow
guide, as I expect there are plenty of people that test ebuilds
straight from CVS right now.


> 
> That said... I find that workflow a bit weird. Why is the firefox
> ebuild already committed if it hasn't even been tested?
> 

That's just how i tend to work with git in what little i've been
exposed to:  git checkout -b tmpbranch ; develop; commit; test &&
{squash,checkout master,cherry-pick,push} || develop [....]

It lets me roll back to earlier commits if i want to undo something,
or revisit some code that i discarded, or whatnot...  Since committing
!= pushing, generally there isn't much harm, and i squash all these
misc commits into one after everything's ready, so it doesn't clutter
the history.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlQXUkoACgkQ2ugaI38ACPAmvQD9GeqZu6dHi5BSd2lpArMvHES/
whHLDOoPdXUeb3Uh99kBAKorCvYZFmKEDHWgvbnCuk/vfua58ItwOjtQIY/WG1Az
=pNOG
-----END PGP SIGNATURE-----

Re: [gentoo-dev] git basics

[hasufell]

Ian Stakenvicius:
> 
> It's generally considered safe to push to origin/master a commit from
> a temporary local branch?

Why not?

Even if you have to rebase/merge, nothing will happen with your unstaged
local changes as long as no one has messed with the firefox ebuild in
the meantime... and then you have bigger problems.

> 
> as I expect there are plenty of people that test ebuilds
> straight from CVS right now.
> 

I remember people telling me to never do that, but anyway.

> 
> 
>> That said... I find that workflow a bit weird. Why is the firefox
>> ebuild already committed if it hasn't even been tested?
> 
> 
> That's just how i tend to work with git in what little i've been
> exposed to:  git checkout -b tmpbranch ; develop; commit; test &&
> {squash,checkout master,cherry-pick,push} || develop [....]
> 
> It lets me roll back to earlier commits if i want to undo something,
> or revisit some code that i discarded, or whatnot...  Since committing
> != pushing, generally there isn't much harm, and i squash all these
> misc commits into one after everything's ready, so it doesn't clutter
> the history.
> 

Maybe you could check if you can do some of that stuff with git stashes.
You can save them with a message indicating what it is about, apply a
particular one from the stash (ofc you can have multiple ones), drop
others, create branches based on the stash etc.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Mon, Sep 15, 2014 at 1:42 PM, Ian Stakenvicius <axs@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 14/09/14 09:06 PM, Peter Stuge wrote:
>> Rich Freeman wrote:
>>> If you just want to do 15 standalone commits before you push you
>>> can do those sequentially easily enough.  A branch would be more
>>> appropriate for some kind of mini-project.
>> ..
>>> That is the beauty of git - branches are really cheap. So are
>>> repositories
>>
>> And commits.
>>
>> Not only are branches cheap, they are also very easy to create,
>> and maybe most importantly they can be created at any time, even
>> after the commits.
>>
>> It's quick and painless to create a bunch of commits which aren't
>> really closely related in sequence, and only later clean the whole
>> series of commits up while creating different branches for commits
>> which should actually be grouped rather than mixed all together.
>>
>
> Ahh, so the secret here would then be just to git add files on a
> related per-package basis, leaving the other files out of the commit.
>  that makes sense.  There would still be the issue of untracked files
> in the repo and the ability to switch back to the 'master' branch to
> cherry-pick a commit for pushing, though...  I guess we'd just have to
> deal with the delay there and try and push all of the changes at once?

If you really like to keep a lot of things going at once, I'd strongly
recommend doing it in another branch (or more than one), so that you
can cherry-pick/merge/etc stuff into master and not have a mess of an
index while doing it.  You need not publish that branch, and if you do
publish it you need not do it on Gentoo's infra (a git repository can
be synced with multiple other repositories, and the various heads
don't need to stay in sync).

In general you want each commit to represent a single "change."  That
might be a revbump in a single package, or it might be a package move
that involves touching 300 packages in a single commit.  The idea
though is that every commit should stand on its own, so that they can
be reverted on their own, etc.  That's just good practice, and should
be what we're trying to do with cvs with the huge limitation that
multi-file changes in cvs aren't atomic.

There are a lot of guides/tools/etc out there for using git.  A
popular one is the git-flow workflow.  I'm not suggesting that it is
really necessary for one person, but anybody not familiar with git
should probably read up on it just so that you have some sense of how
it can be used.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Diamond]

On Mon, 15 Sep 2014 14:51:56 -0400
Rich Freeman <rich0@gentoo.org> wrote:

> 
> In general you want each commit to represent a single "change."  That
> might be a revbump in a single package, or it might be a package move
> that involves touching 300 packages in a single commit.

Is it right that you are going to move portage packages to
git/github/..? How are you going to make "revbump" with git? Especially 
if you need to see the diff between packagename-0.1-r1 and
packagename-0.1-r2 ebuilds? Git doesn't do this by default and it
will might be a nightmare to compare such revbumps by hand.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

On 18 September 2014 08:02, Diamond <diamond@hi-net.ru> wrote:

> Git doesn't do this by default and it
> will might be a nightmare to compare such revbumps by hand.
>

git diff -M1 -C1

^ is usually sufficient to show new files as differences between similar
files that were already there, including revbumps.



-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1084 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Wed, Sep 17, 2014 at 4:02 PM, Diamond <diamond@hi-net.ru> wrote:
> On Mon, 15 Sep 2014 14:51:56 -0400
> Rich Freeman <rich0@gentoo.org> wrote:
>>
>> In general you want each commit to represent a single "change."  That
>> might be a revbump in a single package, or it might be a package move
>> that involves touching 300 packages in a single commit.
>
> Is it right that you are going to move portage packages to
> git/github/..?

The intent is to move to git.  Git is an scm.  Github is a service
that hosts git repositories with some other value-adds.  There is
interest in mirroring gentoo-x86 on github to help entice more to
contribute (assuming they like working with github), but the nature of
git makes it very easy to mirror a repository on as many sites as you
care to.  That is one of the reasons that it is so popular with FOSS.

> How are you going to make "revbump" with git?

With cvs a revbump is:
cp foo-1.ebuild foo-2.ebuild
cvs add foo-2.ebuild
cvs commit

With git a revbump is:
cp foo-1.ebuild foo-2.ebuild
git add foo-2.ebuild
git commit

(I left out changelogs, repoman, etc, since there is no change with
any of these, and I left out syncing the git repo.)

There really is nothing new here.

>  Especially
> if you need to see the diff between packagename-0.1-r1 and
> packagename-0.1-r2 ebuilds? Git doesn't do this by default and it
> will might be a nightmare to compare such revbumps by hand.
>

cvs doesn't do anything to compare the contents of different files.
So, there really is no loss here.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1114 bytes --]

On 18 September 2014 13:01, Rich Freeman <rich0@gentoo.org> wrote:

> With git a revbump is:
> cp foo-1.ebuild foo-2.ebuild
> git add foo-2.ebuild
> git commit
>
> (I left out changelogs, repoman, etc, since there is no change with
> any of these, and I left out syncing the git repo.)
>
> There really is nothing new here.
>
> >  Especially
> > if you need to see the diff between packagename-0.1-r1 and
> > packagename-0.1-r2 ebuilds? Git doesn't do this by default and it
> > will might be a nightmare to compare such revbumps by hand.
> >
>
> cvs doesn't do anything to compare the contents of different files.
> So, there really is no loss here.
>

What's more, you can in fact do:

git mv foo-1.ebuild foo-2.ebuild
git commit

and you can still easily tell git to show that as a difference in a log.

Example script to emulate this and example output:
https://gist.github.com/kentfredric/10e93e9aac875e9edb93

( In fact, you don't even have to use 'git mv', as long as you change the
tree state completely, git is smart enough to track most changes )

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 2133 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Diamond]

On Thu, 18 Sep 2014 17:04:55 +1200
Kent Fredric <kentfredric@gmail.com> wrote:

> What's more, you can in fact do:
> 
> git mv foo-1.ebuild foo-2.ebuild
> git commit
> 
> and you can still easily tell git to show that as a difference in a
> log.
> 
> Example script to emulate this and example output:
> https://gist.github.com/kentfredric/10e93e9aac875e9edb93
> 
> ( In fact, you don't even have to use 'git mv', as long as you change
> the tree state completely, git is smart enough to track most changes )
> 
Lets assume, that I don't want to scrap old ebuild yet. There's no git
cp command. git mv is just git rm + git add. That's what does it look
like (usual revbump with git add in reality):
https://github.com/cerebrum/dr/commit/311df9b04d876f5847416fe5ba699edfab50adb6
I think that git (at least with default config is a pain in the ass for
packages at all and we should probably think about better platform for
portage).

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Thu, Sep 18, 2014 at 3:33 PM, Diamond <diamond@hi-net.ru> wrote:
> Lets assume, that I don't want to scrap old ebuild yet. There's no git
> cp command. git mv is just git rm + git add. That's what does it look
> like (usual revbump with git add in reality):
> https://github.com/cerebrum/dr/commit/311df9b04d876f5847416fe5ba699edfab50adb6
> I think that git (at least with default config is a pain in the ass for
> packages at all and we should probably think about better platform for
> portage).
>

What would you propose?  The problem you raise is just as much an
issue with cvs.  I don't get a continuous history across revbumps in
cvs today, so I don't really see a problem with moving to git.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Diamond]

On Thu, 18 Sep 2014 16:00:59 -0400
Rich Freeman <rich0@gentoo.org> wrote:

> What would you propose?  The problem you raise is just as much an
> issue with cvs.  I don't get a continuous history across revbumps in
> cvs today, so I don't really see a problem with moving to git.
I don't know what to propose. I stumbled over this problem when started
to use git for packages. At least there are other SCM systems too.
Haven't investigated them yet for that issue. Facebook uses even it's
own one.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Peter Stuge]

Diamond wrote:
> I stumbled over this problem when started to use git for packages.

Use git show -M to unstumble yourself.


//Peter

[gentoo-dev] Git copy detection (was: My masterplan for git migration...)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

On Thu, Sep 18, 2014 at 11:33:40PM +0400, Diamond wrote:
> Lets assume, that I don't want to scrap old ebuild yet. There's no git
> cp command. git mv is just git rm + git add. That's what does it look
> like (usual revbump with git add in reality):
> https://github.com/cerebrum/dr/commit/311df9b04d876f5847416fe5ba699edfab50adb6

Git can check for copies if you like:

  $ git clone git://github.com/cerebrum/dr.git
  $ cd dr/
  $ git show --find-copies-harder 311df9b04
  commit 311df9b04d876f5847416fe5ba699edfab50adb6
  …
  diff --git a/games-strategy/openra/openra-20140608.ebuild b/games-strategy/openra/openra-20140722.ebuild
  similarity index 100%
  copy from games-strategy/openra/openra-20140608.ebuild
  copy to games-strategy/openra/openra-20140722.ebuild

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Git copy detection (was: My masterplan for git migration...)

[Diamond]

On Thu, 18 Sep 2014 13:08:11 -0700
"W. Trevor King" <wking@tremily.us> wrote:

> Git can check for copies if you like:
> 
>   $ git clone git://github.com/cerebrum/dr.git
>   $ cd dr/
>   $ git show --find-copies-harder 311df9b04
>   …
>   copy from games-strategy/openra/openra-20140608.ebuild
>   copy to games-strategy/openra/openra-20140722.ebuild

--find-copies-harder is better but it isn't shown on github in a
convenient way for package management.

Re: [gentoo-dev] Git copy detection (was: My masterplan for git migration...)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1004 bytes --]

On Fri, Sep 19, 2014 at 01:01:13AM +0400, Diamond wrote:
> On Thu, 18 Sep 2014 13:08:11 -0700 W. Trevor King wrote:
> > Git can check for copies if you like:
> > 
> >   $ git clone git://github.com/cerebrum/dr.git
> >   $ cd dr/
> >   $ git show --find-copies-harder 311df9b04
> >   …
> >   copy from games-strategy/openra/openra-20140608.ebuild
> >   copy to games-strategy/openra/openra-20140722.ebuild
> 
> --find-copies-harder is better but it isn't shown on github in a
> convenient way for package management.

So Git works great, and GitHub's web UI doesn't support all of Git's
bells and whistles, so… switch to a different VCS?  Personally, my
conclusion is “just use Git from the command line”.  It's not like
you're abandoning long years of experience using CVSHub with some
miracle UI ;).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Git copy detection (was: My masterplan for git migration...)

[Diamond]

On Thu, 18 Sep 2014 14:29:41 -0700
"W. Trevor King" <wking@tremily.us> wrote:


> 
> So Git works great, and GitHub's web UI doesn't support all of Git's
> bells and whistles, so… switch to a different VCS?  Personally, my
> conclusion is “just use Git from the command line”.  It's not like
> you're abandoning long years of experience using CVSHub with some
> miracle UI ;).

--find-copies-harder
           For performance reasons, by default, -C option finds copies
only if the original file of the copy was modified in the same
changeset. This flag makes the command inspect unmodified files as
candidates for the source of copy. This is a very expensive operation
for large projects, so use it with caution. Giving more than one -C
option has the same effect.

P.S. As you see from description this option affects git performance.
So, we probably can arrive at a conclusion that git itself isn't good
at all for package management. Maybe there are better architectural
decisions for that.

Re: [gentoo-dev] Git copy detection (was: My masterplan for git migration...)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 692 bytes --]

On 19 September 2014 10:13, Diamond <diamond@hi-net.ru> wrote:

> P.S. As you see from description this option affects git performance.
> So, we probably can arrive at a conclusion that git itself isn't good
> at all for package management. Maybe there are better architectural
> decisions for that.
>

Well. Yes. You're doing more because you want to get more out.

Or did you expect git to find potential copies from your whole repository
for /no/ increase in performance cost?

That's why it defaults to off, for performance.

( And because not everyone wants it on by default, because its somewhat a
subjective aesthetic choice )


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1459 bytes --]

Re: [gentoo-dev] Git copy detection (was: My masterplan for git migration...)

[Peter Stuge]

Diamond wrote:
> we probably can arrive at a conclusion that git itself isn't good
> at all for package management.

I don't think we can. Please stop the nonsense, at least until you
have created something superior to git.


//Peter

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]

On 19 September 2014 07:33, Diamond <diamond@hi-net.ru> wrote:

> Lets assume, that I don't want to scrap old ebuild yet. There's no git
> cp command. git mv is just git rm + git add. That's what does it look
> like (usual revbump with git add in reality):
>
> https://github.com/cerebrum/dr/commit/311df9b04d876f5847416fe5ba699edfab50adb6
> I think that git (at least with default config is a pain in the ass for
> packages at all and we should probably think about better platform for
> portage).
>

Not necessarily. It tracks copies too, -C

Also, don't rely on githubs presentation of things as being gospel, its
better than nothing, but it falls short of git.

git log -p --find-copies-harder games-strategy/openra-bin/*.ebuild

You can quite easily convince git to pretend vaguely similar files in the
log are sources of each other with the right options.

Throwing "-M1 -C1" in that command will let git find more

Just because "I think git cant" doesn't mean "git cant", especially if
you've not asked "how can I do .... " :)

For instance:

 > git log --stat -C1 -M1 --find-copies-harder

In your repo finds you these interesting "copies" if you look far enough

>.../files/opentracker.init.d =>
net-misc/twonky/files/twonky.initd          | 40 +++++++++----------
>{www-apps/rutorrent =>
net-misc/minidlna}/metadata.xml                      |  16 ++-
>{games-strategy/openra-bin =>
dev-util/mono-debugger}/metadata.xml          |   8 +--
>media-video/{webcamstudio =>
webcamstudio-module}/ChangeLog                 |  42 +++++----


-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 2806 bytes --]

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Martin Vaeth]

Diamond <diamond@hi-net.ru> wrote:
>
> There's no git cp command.
> git mv is just git rm + git add.

I think there is a misunderstanding about how git works.

If you are used to e.g. svn (I suppose with CVS it is similar)
then it makes an important difference whether you use
"svn cp A B" or "/bin/cp A B", because svn is file-centric:
The latter loses history and duplicates data in the repository,
while the former de-duplicates.

git, OTOH is "blob"-centric: It cares only about content of the
file, not where this content comes from; the information that
you copy A to B is not stored, and it does not need to be stored:
De-duplication happens automatically.

In other words, there is no way for git to "see" that B originally
was a copy of A, and normally there is also no such need.
If you think that this information is important enough to be
stored in the history, you have to make a commit where B is
completely identical to A before you make further
modifications for B (identical files have the same checksum and
thus are immediately identified).
If you make no such commit, git can guess only by similarities
of the content that B originally was as a copy of A; this,
of course, needs slightly longer than omitting such comparisons.

For this reason, there is absolutely no reason to use git for
file operations like cp or mv (as long as you add/remove the
corresponding files from the index before you commit, of course).

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 14/09/14 08:57 PM, Rich Freeman wrote:
> On Sun, Sep 14, 2014 at 7:21 PM, Patrick Lauer <patrick@gentoo.org>
> wrote:
>> 
>> iow, git doesn't allow people to work on more than one item at a
>> time?
>> 
>> That'd mean I need half a dozen checkouts just to emulate cvs,
>> which somehow doesn't make much sense to me ...
>> 
> 
> Well, you can work on as many things as you like in git, but it 
> doesn't keep track of what changes have to do with what things if
> you don't commit in-between.  So, you'll have a big list of changes
> in your index, and you'll have to pick-and-choose what you commit
> at any one time.
> 
> If you really want to work on many things "at once" the better way
> to do it is to do a temporary branch per-thing, and when you
> switch between things you switch between branches, and then move
> into master things as they are done.
> 
> I assume you mean working on things that will take a while to 
> complete.  If you just want to do 15 standalone commits before you 
> push you can do those sequentially easily enough.  A branch would
> be more appropriate for some kind of mini-project.
> 
> You can work on branches without pushing those to the master repo. 
> Or, if appropriate a project team might choose to push their branch
> to master, or to some other repo (like an overlay).  This would
> allow collaborative work on a large commit, with a quick final
> merge into the main tree.  That is the beauty of git - branches are
> really cheap. So are repositories - if somebody wants to do all
> their work in github and then push to the main tree, they can do
> that.
> 

Actually i see what Patrick's getting at -- I have similar issues when
working with mozilla stuff.  if you're using local (temporary)
branches, the whole tree is in the state of that current checkout,
right?  IE, while I have my firefox-update branch active and working
for an 'ebuild ... install', I can't be doing work in my
'freewrl-update' branch unless I make multiple separate repo trees,
one for each independently-separate workflow i want to do concurrently.

Ideal here would be the ability to have separate active checkouts of a
repo on a per-shell basis, ie each shell invocation would be able to
work concurrently and distinctly on distinct branches; anyone done
that before?  Does git do it already?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlQXJAAACgkQ2ugaI38ACPBREAD/YnsyY+fAK1TEXgzNYHBCq138
Q5Bj+J6pNGX8aBDjjHoA/iyy5CWxhyAYE3buSOXkEvFfhm/716DsQIptpX7JpS0m
=YrIG
-----END PGP SIGNATURE-----

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

Dnia 2014-09-15, o godz. 07:21:35
Patrick Lauer <patrick@gentoo.org> napisał(a):

> On Sunday 14 September 2014 15:42:15 hasufell wrote:
> > Patrick Lauer:
> > >> Are we going to disallow merge commits and ask devs to rebase local
> > >> changes in order to keep the history "clean"?
> > > 
> > > Is that going to be sane with our commit frequency?
> > 
> > You have to merge or rebase anyway in case of a push conflict, so the
> > only difference is the method and the effect on the history.
> > 
> > Currently... CVS allows you to run repoman on an outdated tree and push
> > broken ebuilds with repoman being happy. Git will not allow this.
> 
> iow, git doesn't allow people to work on more than one item at a time?
> 
> That'd mean I need half a dozen checkouts just to emulate cvs, which somehow 
> doesn't make much sense to me ...

I'd appreciate if you reduced FUD to minimum.

What hasufell meant is that you normally don't have three year-old
files lying around in checkout because you did 'cvs up -dP' in another
directory. With git, you update everything.

What you do locally, is totally unrelated.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 14/09/14 07:21 PM, Patrick Lauer wrote:
> On Sunday 14 September 2014 15:42:15 hasufell wrote:
>> Patrick Lauer:
>>>> Are we going to disallow merge commits and ask devs to rebase
>>>> local changes in order to keep the history "clean"?
>>> 
>>> Is that going to be sane with our commit frequency?
>> 
>> You have to merge or rebase anyway in case of a push conflict, so
>> the only difference is the method and the effect on the history.
>> 
>> Currently... CVS allows you to run repoman on an outdated tree
>> and push broken ebuilds with repoman being happy. Git will not
>> allow this.
> 
> iow, git doesn't allow people to work on more than one item at a
> time?
> 
> That'd mean I need half a dozen checkouts just to emulate cvs,
> which somehow doesn't make much sense to me ...
> 

I think you'd just need to have a (large) handful of different
branches, and squash+cherry-pick into master when you're ready to push.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlQXH08ACgkQ2ugaI38ACPAZ1gD/WfiMZnu3qesaILhPYEKYy2BP
MUS2zWJVqYJ8lKp16nUA/1ng1mMxX6pNKZVYIaT/BFuERKz3g0BcLck+XILs3Hth
=Ul7F
-----END PGP SIGNATURE-----

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Luca Barbato]

On 15/09/14 01:21, Patrick Lauer wrote:
> On Sunday 14 September 2014 15:42:15 hasufell wrote:
>> Patrick Lauer:
>>>> Are we going to disallow merge commits and ask devs to rebase local
>>>> changes in order to keep the history "clean"?
>>>
>>> Is that going to be sane with our commit frequency?
>>
>> You have to merge or rebase anyway in case of a push conflict, so the
>> only difference is the method and the effect on the history.
>>
>> Currently... CVS allows you to run repoman on an outdated tree and push
>> broken ebuilds with repoman being happy. Git will not allow this.
> 
> iow, git doesn't allow people to work on more than one item at a time?

It does.

> That'd mean I need half a dozen checkouts just to emulate cvs, which somehow 
> doesn't make much sense to me ...

Your statement sounds strange to me.

commands you need to know:

git rebase -i

git add (-p)

git commit (-p)

git branch/checkout

Examples

edit cat/pkg/foo.ebuild

edit cat2/pkg/bar.ebuild

edit profile

git add -p    # to select by line what you want in the commit

git commit    # and you write down the commit message

git commit -p # to do both at the same time.

git commit -p # again to lump other changes line by line

OR

edit cat/pkg/foo.ebuild

git commit -a # everything (that's tracked) you edited gets in a commit

edit cat/pkg/bar.ebuild

git commit -a # everything (that's tracked) you edited gets in again

...

git rebase -i # sort out what you want commit merge, edit, drop etc

git push.




Git let you do whatever you do in cvs, but in a _much_ saner and faster way.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Luca Barbato:
> On 15/09/14 01:21, Patrick Lauer wrote:
>> On Sunday 14 September 2014 15:42:15 hasufell wrote:
>>> Patrick Lauer:
>>>>> Are we going to disallow merge commits and ask devs to rebase local
>>>>> changes in order to keep the history "clean"?
>>>>
>>>> Is that going to be sane with our commit frequency?
>>>
>>> You have to merge or rebase anyway in case of a push conflict, so the
>>> only difference is the method and the effect on the history.
>>>
>>> Currently... CVS allows you to run repoman on an outdated tree and push
>>> broken ebuilds with repoman being happy. Git will not allow this.
>>
>> iow, git doesn't allow people to work on more than one item at a time?
> 
> It does.
> 

I think we really have to write up a step-by-step guide (not just
workflow policies) for people who have never seriously worked with git.

On the other hand... there are thousands of tutorials on the net already.

For the workflow model, I already have created a draft which is in no
way finished or even correct and there are still some controversially
discussed issues.
https://wiki.gentoo.org/wiki/Gentoo_git_workflow

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Daniel Campbell]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

On 09/16/2014 01:56 PM, hasufell wrote:
> Luca Barbato:
>> On 15/09/14 01:21, Patrick Lauer wrote:
>>> On Sunday 14 September 2014 15:42:15 hasufell wrote:
>>>> Patrick Lauer:
>>>>>> Are we going to disallow merge commits and ask devs to rebase local
>>>>>> changes in order to keep the history "clean"?
>>>>>
>>>>> Is that going to be sane with our commit frequency?
>>>>
>>>> You have to merge or rebase anyway in case of a push conflict, so the
>>>> only difference is the method and the effect on the history.
>>>>
>>>> Currently... CVS allows you to run repoman on an outdated tree and push
>>>> broken ebuilds with repoman being happy. Git will not allow this.
>>>
>>> iow, git doesn't allow people to work on more than one item at a time?
>>
>> It does.
>>
> 
> I think we really have to write up a step-by-step guide (not just
> workflow policies) for people who have never seriously worked with git.
> 
> On the other hand... there are thousands of tutorials on the net already.
> 
> For the workflow model, I already have created a draft which is in no
> way finished or even correct and there are still some controversially
> discussed issues.
> https://wiki.gentoo.org/wiki/Gentoo_git_workflow
> 
As a prospective Gentoo developer, having a guide around meant
specifically for Gentoo's practices would be incredibly helpful. I use
git in my own hobby development and learned from Pro Git, et al, but
it'd still be really nice to have, and give developers a place to
point to if a new developer is having troubles.

Just my 2¢.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 538 bytes --]

Re: [gentoo-dev] gentoo git step by step guide

[hasufell]

Daniel Campbell:
> As a prospective Gentoo developer, having a guide around meant
> specifically for Gentoo's practices would be incredibly helpful. I use
> git in my own hobby development and learned from Pro Git, et al, but
> it'd still be really nice to have, and give developers a place to
> point to if a new developer is having troubles.
> 
> Just my 2¢.
> 

First draft and only the basic ebuild workflow.

1. do the work (including removing files)

2. run 'repoman manifest'

3. stage files via 'git add <files>' (including removed ones)

4. run 'repoman full' and check for errors
=> if errors occur, fix them and continue from point 2

5. commit files via 'repoman commit -m "<message>"'

6. push to the dev repository via 'git push origin master'
=> if updates were rejected because of non-fast-forward push, try 'git
pull --rebase=preserve origin master' first, then run 'repoman full' and
continue from point 6.

NOTE: you may skip running repoman another time if you have manually
verified that the commits you are missing are totally unrelated to your
work (e.g. only affect a package that is not in the dependency chain of
yours). You can do so via: 'git diff $(git merge-base master
remotes/origin/master) remotes/origin/master'

==> if 'git pull --rebase=preserve origin master' fails or yields
complicated conflicts, continue with a regular merge (you might have to
abort the rebase via 'git rebase --abort' first): 'git merge
remotes/origin/master'

===> if merge conflicts occur, fix them via 'git mergetool' and continue
from point 2.

===> if no merge conflicts occur, run 'repoman full' and continue from
point 6.


TODO: guide for merging user pull requests

Re: [gentoo-dev] gentoo git step by step guide

[Rich Freeman]

On Wed, Sep 17, 2014 at 11:28 AM, hasufell <hasufell@gentoo.org> wrote:
> NOTE: you may skip running repoman another time if you have manually
> verified that the commits you are missing are totally unrelated to your
> work (e.g. only affect a package that is not in the dependency chain of
> yours). You can do so via: 'git diff $(git merge-base master
> remotes/origin/master) remotes/origin/master'

This seems like a reasonable compromise.  99% of the time this will be
the case for single-package commits, and for stuff that is tree-wide
everybody should be using care anyway.  I don't think I've ever made a
single package commit in cvs and run into a collision between the time
I updated cvs and the time I finished making my changes and doing a
commit.

--
Rich

Re: [gentoo-dev] gentoo git step by step guide

[hasufell]

hasufell:
> 
> First draft and only the basic ebuild workflow.
> 

I'm working out the rest here
https://wiki.gentoo.org/wiki/Gentoo_git_workflow#step_by_step_example_.28ebuild.29

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Luca Barbato]

On 14/09/14 17:30, Patrick Lauer wrote:
>> Are we going to disallow merge commits and ask devs to rebase local
>> changes in order to keep the history "clean"?
> 
> Is that going to be sane with our commit frequency?
> 

Which is our commit frequency? Worst case we can aggregate changes and
push them in bulk.

lu

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Tue, Sep 16, 2014 at 1:07 PM, Luca Barbato <lu_zero@gentoo.org> wrote:
> On 14/09/14 17:30, Patrick Lauer wrote:
>>> Are we going to disallow merge commits and ask devs to rebase local
>>> changes in order to keep the history "clean"?
>>
>> Is that going to be sane with our commit frequency?
>>
>
> Which is our commit frequency? Worst case we can aggregate changes and
> push them in bulk.
>

I don't think commit frequency will be a problem other than maybe
causing repoman issues if you're doing tree-wide changes (since
repoman takes a while to run).

See:
https://github.com/rich0/gentoo-gitmig-2014-09-15/graphs/commit-activity

Our tree gets 50-150 commits/day on average it seems.  I have no idea
how far back the punchcard view goes, but that should give a relative
sense of distribution.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 868 bytes --]

On 15 September 2014 00:03, Michał Górny <mgorny@gentoo.org> wrote:

> This means we don't have to wait till someone figures out the perfect
> way of converting the old CVS repository. You don't need that history
> most of the time, and you can play with CVS to get it if you really do.
>

Once somebody works this out, you can also simply make it available as a
"replacement" ref.

See 'git replace'

This would mean, essentially, you could push a ref called
'refs/replace/oldcvs' of value "firstsha1 oldcvssha1" and anyone who wanted
it  could manually fetch it, and any one who did fetch it would get the
full history in all of its glory, and then git would transparently pretend
that history was always there anyway.

No rebasing required, and available on a need-to-know basis :)

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

[-- Attachment #2: Type: text/html, Size: 1638 bytes --]

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Sun, Sep 14, 2014 at 8:03 AM, Michał Górny <mgorny@gentoo.org> wrote:
>
> I'm quite tired of promises and all that perfectionist non-sense which
> locks us up with CVS for next 10 years of bikeshed.

While I tend to agree with the sentiment, I don't think you're
actually targeting the problems that aren't already solved here.

> Of course, that assumes infra is
> going to cooperate quickly or someone else is willing to provide the
> infra for it.

The infra components to a git infrastructure are one of the main
blockers at this point.  I don't really see cooperation as the issue -
just lack of manpower or interest.

>
> I can provide some testing repos once someone is willing to provide
> the hardware.

We already have plenty of testing repos (well, minus all the back-end stuff).

>
> 1. send announcement to devs to explain how to use git,

This is one of the blockers.  We haven't actually decided how we want
to use git.

Sure, everybody knows how to use git.  The problem is that there are a
dozen different ways we COULD use git, and nobody has picked the ONE
way we WILL use it.

This isn't as trivial as you might think.  We have a fairly high
commit rate and with a single repository that means that in-between a
pull-merge/rebase-push there is a decent chance of another commit that
will make the resulting push a non-fast-forward.

People love to point out linux and its insane commit rate.  The thing
is, the mainline git repo with all those commits has exactly one
committer - Linus himself.  They don't have one big repo with one
master branch that everybody pushes to.  At least, that is my
understanding (and there are certainly others here who are more
involved with kernel development).

>
> 2. lock CVS out to read-only,
>
> 3. create all the git repos, get hooks rolling,
>
> 4. enable R/W access to the repos.
>
> With some luck, no more than 2 hours downtime.

I agree that the actual conversion should be able to done quickly.

> On top of user sync repo rsync is propagated. The rsync tree is populated
> with all old ChangeLogs copied from CVS (stored in 30M git repo), new
> ChangeLogs are generated from git logs and Manifests are expanded.

So, I don't really have a problem with your design.  I still question
whether we still need to be generating changelogs - they seem
incredibly redundant.  But, if people really want a redundant copy of
the git log, whatever...

> Main developer repo
> -------------------
>
> I was able to create a start git repository that takes around 66M
> as a git pack (this is how much you will have to fetch to start working
> with it). The repository is stripped clean of history and ChangeLogs,
> and has thin Manifests only.
>
> This means we don't have to wait till someone figures out the perfect
> way of converting the old CVS repository. You don't need that history
> most of the time, and you can play with CVS to get it if you really do.
> In any case, we would likely strip the history anyway to get a small
> repo to work with.

We already have a migration process that coverts the old CVS
repository, generating both a shallow repository that lacks history
and a full repository that contains all of history. Additionally,
these two are consistent - that is the last branch of the full
repository has the same commit ID as the base of the shallow
repository.  Basically we generate the full history and then trim out
99% of it so that the commit in the shallow repository points to a
parent that isn't in the packed repository.

Actually doing the conversion is basically a solved problem.  If this
were actually the blocker I'd be all for just sticking the history in
a different repo and starting from scratch with a new one.

>
> I think we should also merge gentoo-news & glsa & herds.xml into
> the repository. They all reference Gentoo packages at a particular
> state in time, and it would be much nicer to have them synced properly.
>

I can see the pros/cons here, but I don't personally have an issue
with merging them.  As has been brought up elsewhere herds.xml may
just go away.

If somebody can come up with a set of hooks/scripts that will create
the various trees and the only thing that is left is to get infra to
host them, I think we can make real progress.  I don't think this is
something that needs to take a long time.  The pieces are mostly there
- they just have to be assembled.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4996 bytes --]

Dnia 2014-09-14, o godz. 10:33:03
Rich Freeman <rich0@gentoo.org> napisał(a):

> > Of course, that assumes infra is
> > going to cooperate quickly or someone else is willing to provide the
> > infra for it.
> 
> The infra components to a git infrastructure are one of the main
> blockers at this point.  I don't really see cooperation as the issue -
> just lack of manpower or interest.

By 'cooperating' I simply meant offering the necessary resources
in a reasonable time.

> >
> > 1. send announcement to devs to explain how to use git,
> 
> This is one of the blockers.  We haven't actually decided how we want
> to use git.
> 
> Sure, everybody knows how to use git.  The problem is that there are a
> dozen different ways we COULD use git, and nobody has picked the ONE
> way we WILL use it.
> 
> This isn't as trivial as you might think.  We have a fairly high
> commit rate and with a single repository that means that in-between a
> pull-merge/rebase-push there is a decent chance of another commit that
> will make the resulting push a non-fast-forward.
> 
> People love to point out linux and its insane commit rate.  The thing
> is, the mainline git repo with all those commits has exactly one
> committer - Linus himself.  They don't have one big repo with one
> master branch that everybody pushes to.  At least, that is my
> understanding (and there are certainly others here who are more
> involved with kernel development).

It's hard to talk about commit rate when we combine crippled CVS with
awfully stupid two-part repoman committing. This forces us to commit
everything immediately, and makes some of us not committing anything
at all anymore...

With git, we can finally do stuff like preparing everything and pushing
in one go. Rebasing or merging will be much easier then, since
the effective push rate will be smaller than current commit rate.

> > On top of user sync repo rsync is propagated. The rsync tree is populated
> > with all old ChangeLogs copied from CVS (stored in 30M git repo), new
> > ChangeLogs are generated from git logs and Manifests are expanded.
> 
> So, I don't really have a problem with your design.  I still question
> whether we still need to be generating changelogs - they seem
> incredibly redundant.  But, if people really want a redundant copy of
> the git log, whatever...

I don't want them too. However, I'm pretty sure people will bikeshed
this to death if we kill them... Especially that rsync has no git log.
Not that many users make real use of ChangeLogs, esp. considering
how useless messages often are there...

> > Main developer repo
> > -------------------
> >
> > I was able to create a start git repository that takes around 66M
> > as a git pack (this is how much you will have to fetch to start working
> > with it). The repository is stripped clean of history and ChangeLogs,
> > and has thin Manifests only.
> >
> > This means we don't have to wait till someone figures out the perfect
> > way of converting the old CVS repository. You don't need that history
> > most of the time, and you can play with CVS to get it if you really do.
> > In any case, we would likely strip the history anyway to get a small
> > repo to work with.
> 
> We already have a migration process that coverts the old CVS
> repository, generating both a shallow repository that lacks history
> and a full repository that contains all of history. Additionally,
> these two are consistent - that is the last branch of the full
> repository has the same commit ID as the base of the shallow
> repository.  Basically we generate the full history and then trim out
> 99% of it so that the commit in the shallow repository points to a
> parent that isn't in the packed repository.
> 
> Actually doing the conversion is basically a solved problem.  If this
> were actually the blocker I'd be all for just sticking the history in
> a different repo and starting from scratch with a new one.

Was the resulting tree actually verified? How long does the conversion
take? Can it be incremental, i.e. convert most of it, lock CVS, convert
the remaining new commits?

> > I think we should also merge gentoo-news & glsa & herds.xml into
> > the repository. They all reference Gentoo packages at a particular
> > state in time, and it would be much nicer to have them synced properly.
> >
> 
> I can see the pros/cons here, but I don't personally have an issue
> with merging them.  As has been brought up elsewhere herds.xml may
> just go away.
> 
> If somebody can come up with a set of hooks/scripts that will create
> the various trees and the only thing that is left is to get infra to
> host them, I think we can make real progress.  I don't think this is
> something that needs to take a long time.  The pieces are mostly there
> - they just have to be assembled.

Are you willing to champion that, then? :)

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Sun, Sep 14, 2014 at 10:56 AM, Michał Górny <mgorny@gentoo.org> wrote:
> Dnia 2014-09-14, o godz. 10:33:03
>
> With git, we can finally do stuff like preparing everything and pushing
> in one go. Rebasing or merging will be much easier then, since
> the effective push rate will be smaller than current commit rate.

While I agree that the ability to consolidate commits will definitely
help with the commit rate, I'm not sure it will make a big difference.
It will turn a kde stablereq from 300 commits into 1, and do the same
for things like package moves and such.  However, I suspect that the
vast majority of our commits are things like bumps on individual
packages that will always be individual commits.  Maybe insofar as one
person does a bunch of them they can be pushed at the same time,
but...

Looking at https://github.com/rich0/gentoo-gitmig-2014-02-21 it seems
like we get about 150 commits/day on busy days.  I suspect that isn't
evenly distributed, but you may be right that it will just work out.

>>
>> Actually doing the conversion is basically a solved problem.  If this
>> were actually the blocker I'd be all for just sticking the history in
>> a different repo and starting from scratch with a new one.
>
> Was the resulting tree actually verified? How long does the conversion
> take? Can it be incremental, i.e. convert most of it, lock CVS, convert
> the remaining new commits?

The tree has been verified.  The verification approaches so far are
neither 100% thorough nor realtime in operation.  However, I think we
have a working migration process and I don't really see the need to do
a double-check at the time of the actual migration.

ferringb was able to do conversions in about 20min with a decent SSD
and a 32-core system.  His migration scripts can migrate categories in
parallel.  I haven't personally tried to run them myself, but I
believe robbat2 and patrick have experimented with them.  If there is
revived interested I can see if I can set them up to run in a chroot
with some documentation so that anybody can run it and satisfy
themselves that it works, assuming somebody else doesn't have such a
chroot ready to go.  If finding a host to run it on is a problem I'm
sure we could get the Trustees to spring for some time on EC2 or
whatever.  There is no reason that this couldn't be as simple as
extracting a tarball, bind-mounting a cvs repo inside, and firing off
the scripts.

I do not believe it can be made to be incremental.  But, the runtime
should be in keeping with your hour-or-two of downtime suggestion.  I
suspect a fair bit of the downtime will taken just to transfer the
copy of the cvroot to the migration server, and transfer the resulting
git tree to wherever it needs to go and get all the back-end scripts
running/etc.

>
> Are you willing to champion that, then? :)
>

Well, I'm in for what it matters.  I don't have root on any infra
boxes if that is what you're looking for.  :)

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Rich Freeman:
> On Sun, Sep 14, 2014 at 10:56 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> Dnia 2014-09-14, o godz. 10:33:03
>>
>> With git, we can finally do stuff like preparing everything and pushing
>> in one go. Rebasing or merging will be much easier then, since
>> the effective push rate will be smaller than current commit rate.
> 
> While I agree that the ability to consolidate commits will definitely
> help with the commit rate, I'm not sure it will make a big difference.
> It will turn a kde stablereq from 300 commits into 1, and do the same
> for things like package moves and such.  However, I suspect that the
> vast majority of our commits are things like bumps on individual
> packages that will always be individual commits.  Maybe insofar as one
> person does a bunch of them they can be pushed at the same time,
> but...
> 
> Looking at https://github.com/rich0/gentoo-gitmig-2014-02-21 it seems
> like we get about 150 commits/day on busy days.  I suspect that isn't
> evenly distributed, but you may be right that it will just work out.
> 

If the push frequency becomes so high that people barely get stuff
pushed because of conflicts, then we simply have to say goodbye to the
central repository workflow and have to establish a hierarchy where only
a handful of people have direct push access and the rest is worked out
through pull requests to project leads or dedicated reviewers.

So the merging and rebasing work would then be done by fewer people
instead of every single developer.

But given that currently project leads may or may not be active I'm not
sure that I'd vote for such a workflow. And I don't think we need that
yet (although enforced review workflow is ofc superior in many ways).

Let's try it with push access for every developer.

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Dirkjan Ochtman]

On Sun, Sep 14, 2014 at 6:10 PM, hasufell <hasufell@gentoo.org> wrote:
> Let's try it with push access for every developer.

+1.

I'm pretty strongly opposed to leaving the history behind. I'd tend to
agree with Rich when he says that history conversion is pretty much a
solved problem, anyway.

Cheers,

Dirkjan

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

On 14-09-2014 16:56:24 +0200, Michał Górny wrote:
> Rich Freeman <rich0@gentoo.org> napisał(a):
> > So, I don't really have a problem with your design.  I still question
> > whether we still need to be generating changelogs - they seem
> > incredibly redundant.  But, if people really want a redundant copy of
> > the git log, whatever...
> 
> I don't want them too. However, I'm pretty sure people will bikeshed
> this to death if we kill them... Especially that rsync has no git log.
> Not that many users make real use of ChangeLogs, esp. considering
> how useless messages often are there...

Council had some discussions on this topic:
http://www.gentoo.org/proj/en/council/meeting-logs/20111108-summary.txt
http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt

Conclusion back then was that ChangeLog files need to stay.


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

On Mon, Sep 15, 2014 at 09:53:43AM +0200, Fabian Groffen wrote:
> On 14-09-2014 16:56:24 +0200, Michał Górny wrote:
> > Rich Freeman <rich0@gentoo.org> napisał(a):
> > > So, I don't really have a problem with your design.  I still question
> > > whether we still need to be generating changelogs - they seem
> > > incredibly redundant.  But, if people really want a redundant copy of
> > > the git log, whatever...
> > 
> > I don't want them too. However, I'm pretty sure people will bikeshed
> > this to death if we kill them... Especially that rsync has no git log.
> > Not that many users make real use of ChangeLogs, esp. considering
> > how useless messages often are there...
> 
> Council had some discussions on this topic:
> http://www.gentoo.org/proj/en/council/meeting-logs/20111108-summary.txt
> http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt
> 
> Conclusion back then was that ChangeLog files need to stay.

I would have no problem with the council revisiting/changing this.

I tend to agree that the ChangeLogs in the portage tree will be
obsoleted when we switch to git because git's logging facilities are
much easier to use than those in CVS. Not to mention how much smaller
the portage tree would be without ChangeLogs.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Anthony G. Basile]

On 09/15/14 15:30, William Hubbs wrote:
> On Mon, Sep 15, 2014 at 09:53:43AM +0200, Fabian Groffen wrote:
>> On 14-09-2014 16:56:24 +0200, Michał Górny wrote:
>>> Rich Freeman <rich0@gentoo.org> napisał(a):
>>>> So, I don't really have a problem with your design.  I still question
>>>> whether we still need to be generating changelogs - they seem
>>>> incredibly redundant.  But, if people really want a redundant copy of
>>>> the git log, whatever...
>>> I don't want them too. However, I'm pretty sure people will bikeshed
>>> this to death if we kill them... Especially that rsync has no git log.
>>> Not that many users make real use of ChangeLogs, esp. considering
>>> how useless messages often are there...
>> Council had some discussions on this topic:
>> http://www.gentoo.org/proj/en/council/meeting-logs/20111108-summary.txt
>> http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt
>>
>> Conclusion back then was that ChangeLog files need to stay.
> I would have no problem with the council revisiting/changing this.
>
> I tend to agree that the ChangeLogs in the portage tree will be
> obsoleted when we switch to git because git's logging facilities are
> much easier to use than those in CVS. Not to mention how much smaller
> the portage tree would be without ChangeLogs.
>
> William
>

If the argument is that there are no Changelogs in rsync, then let's 
write git hooks to generate them when the repository is mirrored to the 
rsync host.  The only problem I see is with this is then adding 
ChangeLog to the manifest and gpg signing it which has to be done at the 
developer's side.  But, I think the tree that users get from rsync 
should have the logs.  Having *both* a ChangeLog file and git log is 
redundant.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Mon, Sep 15, 2014 at 3:55 PM, Anthony G. Basile <blueness@gentoo.org> wrote:
> On 09/15/14 15:30, William Hubbs wrote:
>> I would have no problem with the council revisiting/changing this.
>>
>> I tend to agree that the ChangeLogs in the portage tree will be
>> obsoleted when we switch to git because git's logging facilities are
>> much easier to use than those in CVS. Not to mention how much smaller
>> the portage tree would be without ChangeLogs.
>>
>> William
>>
>
> If the argument is that there are no Changelogs in rsync, then let's write
> git hooks to generate them when the repository is mirrored to the rsync
> host.  The only problem I see is with this is then adding ChangeLog to the
> manifest and gpg signing it which has to be done at the developer's side.
> But, I think the tree that users get from rsync should have the logs.
> Having *both* a ChangeLog file and git log is redundant.
>

I'll add this to the next Council agenda.  I think this is ripe for
discussion.  The last discussion of this really wasn't aimed at git
anyway.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

On 15-09-2014 15:58:00 -0400, Rich Freeman wrote:
> > If the argument is that there are no Changelogs in rsync, then let's write
> > git hooks to generate them when the repository is mirrored to the rsync
> > host.  The only problem I see is with this is then adding ChangeLog to the
> > manifest and gpg signing it which has to be done at the developer's side.
> > But, I think the tree that users get from rsync should have the logs.
> > Having *both* a ChangeLog file and git log is redundant.
> >
> 
> I'll add this to the next Council agenda.  I think this is ripe for
> discussion.  The last discussion of this really wasn't aimed at git
> anyway.

Not sure if you've read the discussions that were done on this topic.
The Council decided (due to git and auto-generation of ChangeLogs) that
ChangeLogs need to be amended, updated or changed.  That to fix
misleading typos, and more.  ChangeLogs are meant for our users.  For
this reason repoman was changed to update the ChangeLog automatically on
commit, if no changes to this file had been made.


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

>>>>> On Mon, 15 Sep 2014, Rich Freeman wrote:

> I'll add this to the next Council agenda.  I think this is ripe for
> discussion.  The last discussion of this really wasn't aimed at git
> anyway.

Some of the arguments back then were that a) ChangeLogs are aimed at
users, so they don't necessarily contain the same information as the
commit log (i.e. could be seen as NEWS files) and b) that it should be
possible to edit them, for example, to correct typos or wrong bug#
references. 

I fail to see how anything of this would depend on the underlying VCS
used.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Ulrich Mueller:
> 
> ChangeLogs are aimed at users

Did any1 ask them if they care?

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Alan McKinnon]

On 16/09/2014 12:18, hasufell wrote:
> Ulrich Mueller:
>>
>> ChangeLogs are aimed at users
> 
> Did any1 ask them if they care?
> 
> 
> 



I'm a user and I don't care.

I use diff. I only go to the Changelog when I can't determine the
maintainers intent from diff and the ebuild content. That happens maybe
once a year.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

Dnia 2014-09-16, o godz. 12:30:59
Alan McKinnon <alan.mckinnon@gmail.com> napisał(a):

> On 16/09/2014 12:18, hasufell wrote:
> > Ulrich Mueller:
> >>
> >> ChangeLogs are aimed at users
> > 
> > Did any1 ask them if they care?
> 
> I'm a user and I don't care.
> 
> I use diff. I only go to the Changelog when I can't determine the
> maintainers intent from diff and the ebuild content. That happens maybe
> once a year.

And if you need to do that, it usually means that someone didn't add
a useful comment to the ebuild :).

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Tue, Sep 16, 2014 at 6:18 AM, hasufell <hasufell@gentoo.org> wrote:
> Ulrich Mueller:
>>
>> ChangeLogs are aimed at users
>
> Did any1 ask them if they care?
>

I'm sure somebody will reply and say that they care.

It still seems like a lot of overhead to me for a very one-off
workflow.  Maybe if portage automatically output the relevant
changelog entries in pretend mode we could pretend that they're news
or something like that.  Most likely, if you stick something important
in the changelog it will be read by maybe 0.1% of our users before
emerging the package.  Maybe if you're lucky 20% of people running
into some kind of breakage will read the changelog after the fact.  I
imagine that 19.5% of those 20% would check the git log if the
changelog didn't exist.

If we actually move to a model where many users actually sync their
trees from git, then I'd expect the changelogs to be even less useful.
After all, git will actually tell you what changed since your last
sync.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Rich Freeman:
> On Tue, Sep 16, 2014 at 6:18 AM, hasufell <hasufell@gentoo.org> wrote:
>> Ulrich Mueller:
>>>
>>> ChangeLogs are aimed at users
>>
>> Did any1 ask them if they care?
>>
> 
> I'm sure somebody will reply and say that they care.
> 
> It still seems like a lot of overhead to me for a very one-off
> workflow.  Maybe if portage automatically output the relevant
> changelog entries in pretend mode we could pretend that they're news
> or something like that.  Most likely, if you stick something important
> in the changelog it will be read by maybe 0.1% of our users before
> emerging the package.  Maybe if you're lucky 20% of people running
> into some kind of breakage will read the changelog after the fact.  I
> imagine that 19.5% of those 20% would check the git log if the
> changelog didn't exist.
> 
> If we actually move to a model where many users actually sync their
> trees from git, then I'd expect the changelogs to be even less useful.
> After all, git will actually tell you what changed since your last
> sync.
> 

And git allows you to _properly_ check for changes, because all changes
are in one history, so you don't have to grep 3+ ChangeLogs (e.g. in
eclasses, profiles and licenses) in order to know what happened.
Even easier... related changes might just go in one commit and when you
look for it you'll also see the other files that have been modified as
part of a version bump (e.g. a useflag mask or whatever).

The only place I actually look for changes is the gentoo-commits ML
which is kind of the poor version of a git history.

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Pacho Ramos]

El mar, 16-09-2014 a las 07:26 -0400, Rich Freeman escribió:
> On Tue, Sep 16, 2014 at 6:18 AM, hasufell <hasufell@gentoo.org> wrote:
> > Ulrich Mueller:
> >>
> >> ChangeLogs are aimed at users
> >
> > Did any1 ask them if they care?
> >
> 
> I'm sure somebody will reply and say that they care.
> 
> It still seems like a lot of overhead to me for a very one-off
> workflow.  Maybe if portage automatically output the relevant
> changelog entries in pretend mode we could pretend that they're news
> or something like that.  Most likely, if you stick something important
> in the changelog it will be read by maybe 0.1% of our users before
> emerging the package.  Maybe if you're lucky 20% of people running
> into some kind of breakage will read the changelog after the fact.  I
> imagine that 19.5% of those 20% would check the git log if the
> changelog didn't exist.
> 
> If we actually move to a model where many users actually sync their
> trees from git, then I'd expect the changelogs to be even less useful.
> After all, git will actually tell you what changed since your last
> sync.
> 
> --
> Rich
> 

Maybe one option would be to kill Changelogs and provide a script to let
people get git messages and reformat them in a way similar as current
ChangeLog files, that way people will still be able to save this
information for the future (if they won't have internet conection later
for example) and read it simply with "less" for example. With this
option, we won't need to provide Changelogs and distribute them but
people wanting to have them will still be able to generate them if
wanted (for example, just after updating portage tree)

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Tue, Sep 16, 2014 at 9:44 AM, Pacho Ramos <pacho@gentoo.org> wrote:
>
> Maybe one option would be to kill Changelogs and provide a script to let
> people get git messages and reformat them in a way similar as current
> ChangeLog files, that way people will still be able to save this
> information for the future (if they won't have internet conection later
> for example) and read it simply with "less" for example. With this
> option, we won't need to provide Changelogs and distribute them but
> people wanting to have them will still be able to generate them if
> wanted (for example, just after updating portage tree)

Or they could just clone the git tree, and they can look at per-file
logs anytime they want to.

I mean, sure, we COULD do this stuff.  But, why?

It isn't like kernel.org has some tool that lets kernel users generate
per-file changelog histories just in case they don't want to use git.

If somebody wants to build a tool like this by all means go ahead and
do it. I just don't see it as something that should be a migration
pre-requisite.  That's just my opinion though.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Pacho Ramos]

El mar, 16-09-2014 a las 09:55 -0400, Rich Freeman escribió:
> On Tue, Sep 16, 2014 at 9:44 AM, Pacho Ramos <pacho@gentoo.org> wrote:
> >
> > Maybe one option would be to kill Changelogs and provide a script to let
> > people get git messages and reformat them in a way similar as current
> > ChangeLog files, that way people will still be able to save this
> > information for the future (if they won't have internet conection later
> > for example) and read it simply with "less" for example. With this
> > option, we won't need to provide Changelogs and distribute them but
> > people wanting to have them will still be able to generate them if
> > wanted (for example, just after updating portage tree)
> 
> Or they could just clone the git tree, and they can look at per-file
> logs anytime they want to.
> 
> I mean, sure, we COULD do this stuff.  But, why?
> 
> It isn't like kernel.org has some tool that lets kernel users generate
> per-file changelog histories just in case they don't want to use git.
> 
> If somebody wants to build a tool like this by all means go ahead and
> do it. I just don't see it as something that should be a migration
> pre-requisite.  That's just my opinion though.
> 
> --
> Rich
> 

I don't consider it a pre-requisite either, was only trying to give an
option to still tell people how to get a ChangeLog similar to current
ones easily (as looks like they are used a lot per the past
discussion :/) I remember something similar was done in the past when
gnome stuff moved to git:
https://wiki.gnome.org/Git/ChangeLog

But I guess once we get habituated to simply review something equivalent
to https://git.gnome.org/browse/ not many people will miss the old
Changelogs ;)

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

Rich Freeman posted on Tue, 16 Sep 2014 09:55:31 -0400 as excerpted:

> Or they could just clone the git tree, and they can look at per-file
> logs anytime they want to.

Give me ro access to a current git repo and I'll *VERY* happily leave 
changelogs to history along with 8-track tapes and 5.25-inch floppies! 
=:^)

I was strongly in favor of keeping changelogs (and mandating proper add/
change/deletion entries) the last time the topic came up, but that was in 
the context of (web)?rsync being the only viable user sync method and 
thus changelogs being the only user-local-accessible record.  With user-
git-repo access, I'll /very/ (very, very, very...) happily leave rsync 
behind for git, and changelogs along with it! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[vivo75]

Il 16/09/2014 20:02, Duncan ha scritto:
> Rich Freeman posted on Tue, 16 Sep 2014 09:55:31 -0400 as excerpted:
>
>> Or they could just clone the git tree, and they can look at per-file
>> logs anytime they want to.
> Give me ro access to a current git repo and I'll *VERY* happily leave 
> changelogs to history along with 8-track tapes and 5.25-inch floppies! 
> =:^)
>
> I was strongly in favor of keeping changelogs (and mandating proper add/
> change/deletion entries) the last time the topic came up, but that was in 
> the context of (web)?rsync being the only viable user sync method and 
> thus changelogs being the only user-local-accessible record.  With user-
> git-repo access, I'll /very/ (very, very, very...) happily leave rsync 
> behind for git, and changelogs along with it! =:^)
>
yes, this probably is the same for everyone, and if it's not it should
be anyway.

[gentoo-dev] Re: git migration

[Steven J. Long]

On Tue, Sep 16, 2014 at 07:26:06AM -0400, Rich Freeman wrote:
> On Tue, Sep 16, 2014 at 6:18 AM, hasufell <hasufell@gentoo.org> wrote:
> > Ulrich Mueller:
> >>
> >> ChangeLogs are aimed at users
> >
> > Did any1 ask them if they care?
> >
> 
> I'm sure somebody will reply and say that they care.

Yup, mainly because of this:

> It still seems like a lot of overhead to me for a very one-off
> workflow.  Maybe if portage automatically output the relevant
> changelog entries in pretend mode we could pretend that they're news
> or something like that.

..which seems to me the job of a consuming script. We[1] present the
last entry from changelogs when, for instance, a package is coming
through as a Downgrade from emerge.

It's counter-productive to overload the plumbing layer, with porcelain.
Equally the plumbing-layer has to provide enough info for the porcelain
to make decisions. That info has to be tailored for the admin, not the
ebuild developer, eclass author, or proxy-maintainer.

It's not needed when nothing's going wrong, but when something weird is
happening, the short message usually provides assurance and/or info
on options. (see next)

> If we actually move to a model where many users actually sync their
> trees from git, then I'd expect the changelogs to be even less useful.
> After all, git will actually tell you what changed since your last
> sync.

VCS commit messages are very different to the ebuild Changelogs ime.
However if you're switching to git, have the equivalent info in git
logs, and the user has a git clone by definition, that /can/ work.

I would ask that you consider the different purpose of VCS commit
messages, vs ebuild Changelogs. The latter are terse summaries from
the admin perspective, and the former naturally tend to be more
code-oriented, especially when applied to bigger projects, and
especially with the ncremental-commit model that git fosters. (don't
rebase: just push, and use add -p if you have to break it up[2].)

Either way, I don't think the discussion about Changelogs should
*at all* affect the move to git; you can change Policy after
switching, very easily, and there will be other things to iron out
which will take much more effort, so I don't think the switch
should get sidetracked, by this or other post-switch topics raised.

You've mentioned scripts a couple of times, and wanting help with
them; can I suggest you post a (git preferably) url of something
people can see and help you with? And tell us what IRC channel,
if you prefer that to forums, or the list for code review.

#git are excellent, btw, eg if you want to rewrite emails on import,
in your case from LDAP, or get advice on large-scale deployment
from the infra side.

[1] http://weaver.gentooexperimental.org/update.html
[2] http://forums.gentoo.org/viewtopic-p-7587838.html#7587838
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: git migration

[hasufell]

Steven J. Long:
> 
> VCS commit messages are very different to the ebuild Changelogs ime.

Yes, but this does not even apply to the current gentoo CVS workflow
(compare the cvs commit messages with the ChangeLog entries, e.g. for
sys-apps/portage).

> I would ask that you consider the different purpose of VCS commit
> messages, vs ebuild Changelogs. The latter are terse summaries from
> the admin perspective, and the former naturally tend to be more
> code-oriented, especially when applied to bigger projects, and
> especially with the ncremental-commit model that git fosters. (don't
> rebase: just push, and use add -p if you have to break it up[2].)

The difference between admin perspective and developer perspective is
very small in the context of ebuilds. Ebuild code is trivial and the
rate of changes that don't affect the users directly (such as optimizing
syntax stuff, improving code style etc) is pretty low when I look at my
own ebuilds.
The rest of the messages that do affect users are usually just "added
missing dependency on sys-apps/foo wrt #12345" or similar which don't
really need any sort of conversion to be understood by users/admins.

> Either way, I don't think the discussion about Changelogs should *at all* affect the move to git;

Correct, because this wouldn't even be a regression to the current CVS
workflow.

Re: [gentoo-dev] Re: git migration

[Rich Freeman]

On Sat, Sep 20, 2014 at 12:48 PM, hasufell <hasufell@gentoo.org> wrote:
> Steven J. Long:
>
>> Either way, I don't think the discussion about Changelogs should *at all* affect the move to git;
>
> Correct, because this wouldn't even be a regression to the current CVS
> workflow.
>

That depends a great deal on how the move is implemented.

Most proposals involve dropping Changelogs from the tree, so devs
would not write to them.  Having Changelogs in the tree that users
sync requires the creation of a script that will copy the git commit
message into the Changelog.

If we continued to maintain Changelogs as text files in the git tree
then there would not be any change in workflow.  Devs would continue
to use whatever scripts they're currently using to copy the commit
message into the Changelog, or vice-versa.

--
Rich

Re: [gentoo-dev] Re: git migration

[hasufell]

Rich Freeman:
> On Sat, Sep 20, 2014 at 12:48 PM, hasufell <hasufell@gentoo.org> wrote:
>> Steven J. Long:
>>
>>> Either way, I don't think the discussion about Changelogs should *at all* affect the move to git;
>>
>> Correct, because this wouldn't even be a regression to the current CVS
>> workflow.
>>
> 
> That depends a great deal on how the move is implemented.
> 

I was more referring to the proposal of separating git commits from
ChangeLogs which we don't even do now and I think is mostly pointless in
the context of ebuilds.

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Daniel Campbell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/16/2014 05:18 AM, hasufell wrote:
> Ulrich Mueller:
>> 
>> ChangeLogs are aimed at users
> 
> Did any1 ask them if they care?
> 

If the tree switches to git and there's an option within
Portage/emerge to fetch via git instead of rsync, then I'd rather rely
on `git log` than a bunch of scattered files.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJUGRVnAAoJEJUrb08JgYgH1pQH/RiBmtnUvewDY+dm1cdbAdWb
A8YXcHDHhYnVtll3x7hB+YphKLNYBN+baLLiKXHAR4LaWIfc+Z0NHMpfN3pNQTwZ
o3XjzShWMhZ9Z5mTafPuFgR1f+sAuqSG0lOhMm3tHwKmBEHt3fh2bnAZVkGtnJRE
L/xDCU5sniGPJCLhXBaPfU3om99xeEQtahXWR+rVHj64h93t9Cb1hHIlWRvjPzDT
M5kC9Rz/BS1wO4mwPqi/jW5mbQnLUhcy7y4OSszQeAMyroCIhkxwwKLeWES62XQr
bo6AKqv1SKMFVYIgYVRei0iTXbQ2/pWzlpatM11G6djqMtTvDlMR7f3wPbAiw2U=
=EIKj
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

Dnia 2014-09-16, o godz. 10:18:35
hasufell <hasufell@gentoo.org> napisał(a):

> Ulrich Mueller:
> > 
> > ChangeLogs are aimed at users
> 
> Did any1 ask them if they care?

A bit off-topic but asking such a question usually makes some
developers point out that they are users too and they do care :).

However, in this particular context it could be helpful to do some
research. As people have pointed out, users may actually prefer using
some git web-interface or git clone to get the details.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 2847 bytes --]

>>>>> On Wed, 17 Sep 2014, Michał Górny wrote:

> Dnia 2014-09-16, o godz. 10:18:35
> hasufell <hasufell@gentoo.org> napisał(a):

>> Ulrich Mueller:
>> >
>> > ChangeLogs are aimed at users
>>
>> Did any1 ask them if they care?

> A bit off-topic but asking such a question usually makes some
> developers point out that they are users too and they do care :).

> However, in this particular context it could be helpful to do some
> research. As people have pointed out, users may actually prefer using
> some git web-interface or git clone to get the details.

It is two questions that need to be answered. The first question is if
we want to keep a ChangeLog that is separate from commit messages. The
second question (and it can only be answered after the first one) is
how the information should be transmitted to users. It doesn't make
sense to mix the two questions, or to answer the second one before the
first.

The original intention back in 2002 was having a ChangeLog containing
information somewhat complementary to commit messages. This can still
be found in skel.ChangeLog:

| This changelog is targeted to users. This means that the comments
| should be well explained and written in clean English.

And:

| Any details about what exactly changed in the code should be added
| as a message when the changes are committed to cvs, not in this
| file.

That different messages can be used was acknowledged in the November
2011 council meeting (and yours truly had voted in favour of it):

| The Council agreed that developers are free to use different
| messages for ChangeLog and commit, but they are responsible for the
| messages, and the Council still expects appropriate messages to be
| used.

So the research that needs to be done first is to find out how often
our ChangeLog entries differ from the commit log. If it turns out that
they are identical in 99 % of all cases, then it obviously makes no
sense to maintain the same information in two places, and ChangeLogs
should be abandoned. (For my own commits, I would estimate that
messages are different for 20 % of commits.)

Only when this has been answered, we should discuss how the
information should be formatted and how users should obtain it.
Some ideas:
- We could have an echangelog replacement (integrated into repoman?)
  for nice formatting of commit messages.
- If we abandon separately maintained ChangeLog files, then there
  should be some means for correcting mistakes in commit messages.
  Maybe "git notes" could be used?
- There is certainly room for improvement how to communicate news
  about a package to users, apart from elog messages and GLEP 42 news
  items. Maybe readme.gentoo.eclass could be extended to optionally
  install a NEWS.gentoo file along with README.gentoo?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Wed, Sep 17, 2014 at 5:56 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
>
> So the research that needs to be done first is to find out how often
> our ChangeLog entries differ from the commit log. If it turns out that
> they are identical in 99 % of all cases, then it obviously makes no
> sense to maintain the same information in two places, and ChangeLogs
> should be abandoned. (For my own commits, I would estimate that
> messages are different for 20 % of commits.)

When I do commits the commit message is scripted to be identical to
the changelog message.  I doubt I have a single commit where they
differed, unless I went back to modify a changelog to fix a typo or
something.  They're all intended to be readable by anybody.

>
> Only when this has been answered, we should discuss how the
> information should be formatted and how users should obtain it.

So, this will be on the Council agenda.  By all means go out and dig
up whatever info you think will be useful for making a decision, but I
don't want to put this off hoping that somebody else will do it.  I
don't think it is essential to determine whether changelog messages
are different from commit messages in practice.  If a majority of
council members disagree we can defer the decision.

> Some ideas:

No objection to any of your ideas per se, but I don't want to make any
of them blockers for a git migration.  I think getting off of cvs is
orthogonal to improving our documentation and communications.

There are a million ways we could be spending our effort on Gentoo,
but I don't think that making our commit messages more nicely
formatted/etc is something worthy of a rule (ie something all devs are
forced to contribute to).  95% of it is noise, so if there is a
message that really needs to get out to users it should go into
something like a news item that is distributed BEFORE the change is
made.

If documentation improvements are built into a new echangelog-like
tool, I think that will greatly help adoption, but again I don't want
to hold up the git migration for this.  The git migration has been a
moving target forever because there has always been just one more
thing that needs to be done, and most who have gotten involved have
gotten frustrated/bored/whatever and moved on.  How many FOSS projects
of our scale are still using cvs anyway?

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2075 bytes --]

Dnia 2014-09-15, o godz. 15:55:35
"Anthony G. Basile" <blueness@gentoo.org> napisał(a):

> On 09/15/14 15:30, William Hubbs wrote:
> > On Mon, Sep 15, 2014 at 09:53:43AM +0200, Fabian Groffen wrote:
> >> On 14-09-2014 16:56:24 +0200, Michał Górny wrote:
> >>> Rich Freeman <rich0@gentoo.org> napisał(a):
> >>>> So, I don't really have a problem with your design.  I still question
> >>>> whether we still need to be generating changelogs - they seem
> >>>> incredibly redundant.  But, if people really want a redundant copy of
> >>>> the git log, whatever...
> >>> I don't want them too. However, I'm pretty sure people will bikeshed
> >>> this to death if we kill them... Especially that rsync has no git log.
> >>> Not that many users make real use of ChangeLogs, esp. considering
> >>> how useless messages often are there...
> >> Council had some discussions on this topic:
> >> http://www.gentoo.org/proj/en/council/meeting-logs/20111108-summary.txt
> >> http://www.gentoo.org/proj/en/council/meeting-logs/20111011-summary.txt
> >>
> >> Conclusion back then was that ChangeLog files need to stay.
> > I would have no problem with the council revisiting/changing this.
> >
> > I tend to agree that the ChangeLogs in the portage tree will be
> > obsoleted when we switch to git because git's logging facilities are
> > much easier to use than those in CVS. Not to mention how much smaller
> > the portage tree would be without ChangeLogs.
> 
> If the argument is that there are no Changelogs in rsync, then let's 
> write git hooks to generate them when the repository is mirrored to the 
> rsync host.  The only problem I see is with this is then adding 
> ChangeLog to the manifest and gpg signing it which has to be done at the 
> developer's side.  But, I think the tree that users get from rsync 
> should have the logs.  Having *both* a ChangeLog file and git log is 
> redundant.

Can't we just kill rsync then? The whole ChangeLog seems to take more
effort than the actual benefit it gives.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

On Mon, Sep 15, 2014 at 10:18:39PM +0200, Michał Górny wrote:
> Dnia 2014-09-15, o godz. 15:55:35 Anthony G. Basile napisał(a):
> > If the argument is that there are no Changelogs in rsync, then
> > let's write git hooks to generate them when the repository is
> > mirrored to the rsync host.  The only problem I see is with this
> > is then adding ChangeLog to the manifest and gpg signing it which
> > has to be done at the developer's side.  But, I think the tree
> > that users get from rsync should have the logs.  Having *both* a
> > ChangeLog file and git log is redundant.
> 
> Can't we just kill rsync then? The whole ChangeLog seems to take
> more effort than the actual benefit it gives.

I'm +1 for killing rsync and having everyone use Git.  With --shallow
clones for folks who don't care about the history, and deep clones for
those who do (and you can change your mind both ways), I think
everyone gets what they want without messing around with a Git → rsync
conversion layer.  Of course, it would be nice if the CSV → Git
migration added any ChangeLog notes to the associated commit message
to avoid losing information, but I imagine it would be hard to
automate that and still get readable commit messages ;).

I don't see any benefit to using rsync vs. a shallow clone as the
transmission protocol.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Mon, Sep 15, 2014 at 01:29:44PM -0700, W. Trevor King wrote:
> I don't see any benefit to using rsync vs. a shallow clone as the
> transmission protocol.

Other than the fact that before you dropped it you'd need to push a
‘emerge sync’ that could handle either rsync or Git, stabilize that
Portage, and then wait for folks to adopt it.  That's going to slow
down your migration a bit ;).  I think an rsync-able version is a
better choice for the migration, but since it's not destined to live
long (in my view), I don't think it really matters what goes into it.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

W. Trevor King posted on Mon, 15 Sep 2014 13:33:46 -0700 as excerpted:

> On Mon, Sep 15, 2014 at 01:29:44PM -0700, W. Trevor King wrote:
>> I don't see any benefit to using rsync vs. a shallow clone as the
>> transmission protocol.
> 
> Other than the fact that before you dropped it you'd need to push a
> ‘emerge sync’ that could handle either rsync or Git, stabilize that
> Portage, and then wait for folks to adopt it.

Portage already handles it. =:^)

Quoting from the emerge (1) manpage (listed as v2.2.12, dated Mar 2014):

--sync
  Updates  repositories, for which sync-type and sync-uri
  attributes are set in repos.conf. See portage(5) for more
  information.


Then in the portage (5) manpage:

repos.conf
  Specifies site-specific repository configuration information.
  Configuration specified in repos.conf can be overriden by
  PORTAGE_REPOSITORIES environmental variable, which has the
  same format as repos.conf.
                     
  [...]
                       
  Attributes supported in sections of repositories:

    [...]

    sync-type
      Specifies type of synchronization performed by `emerge
      --sync`. Valid non-empty values: cvs, git, rsync
      This  attribute  can be set to empty value to disable
      synchronization of given repository. Empty value is
      default.
                                   
    sync-uri
      Specifies URI of repository used for synchronization
      performed by `emerge --sync`.  This attribute can be
      set to empty value to disable synchronization of given
      repository. Empty value is default.
                                   
      Syntax:
        cvs: [cvs://]:access_method:[username@]hostname[:port]:/path
        git: (git|git+ssh|http|https)://[username@]hostname[:port]/path
        rsync: (rsync|ssh)://[username@]hostname[:port]/(module|path)
                                          

So portage already handles it. =:^)

Tho it's possible there are still bugs as surely that doesn't have the 
extremely broad long-term testing that rsync does, and I'm ~arch of 
course and don't know if that's stable, yet.  But it's there.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

On Tue, Sep 16, 2014 at 05:35:08PM +0000, Duncan wrote:
> W. Trevor King posted on Mon, 15 Sep 2014 13:33:46 -0700 as excerpted:
> > On Mon, Sep 15, 2014 at 01:29:44PM -0700, W. Trevor King wrote:
> >> I don't see any benefit to using rsync vs. a shallow clone as the
> >> transmission protocol.
> > 
> > Other than the fact that before you dropped it you'd need to push a
> > ‘emerge sync’ that could handle either rsync or Git, stabilize that
> > Portage, and then wait for folks to adopt it.
> 
> Portage already handles it. =:^)

Oh, lovely :).  Looks like that landed in 2.2.0 with 47e8d22d (Add
support for multiple repositories in `emerge --sync`, 2013-07-23).
There are older Portages in the tree though (back to 2.1.6.7_p1), so
you'd still want to wait until those were gone before dropping rsync.

Also, I don't see a way to say “use Git to sync, but keep a shallow
repository”.  Ideally, we'd want:

  $ git clone --depth=1 git://git.gentoo.org/gentoo-portage.git

for the initial clone (modulo whatever URI), and:

  $ git pull --depth=1

for subsequent syncs.  pym/_emerge/actions.py currently hardcodes ‘git
pull’ for the latter, and doesn't seem to have any code for the
former.  On the other hand, it wouldn't be too terrible to force users
to shallow their history manually whenever they felt like it.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

On Tue, Sep 16, 2014 at 10:52:13AM -0700, W. Trevor King wrote:
> Oh, lovely :).  Looks like that landed in 2.2.0 with 47e8d22d (Add
> support for multiple repositories in `emerge --sync`, 2013-07-23).

Actually, ‘git pull’ support in one form or another dates back to
ba797c11 (Add --sync support for `git pull`, 2008-12-11), which landed
in v2.2_rc18.

> There are older Portages in the tree though (back to 2.1.6.7_p1), so
> you'd still want to wait until those were gone before dropping rsync.

The ‘git pull’ support was also backported to the 2.1.6.7_p1 series
with d3c42937 (Add --sync support for `git pull`, 2008-12-11), which
landed in v2.1.6.1, so I doubt any Portage users lack pull support.
I'm not sure about folks using other package managers though.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

W. Trevor King posted on Tue, 16 Sep 2014 10:52:13 -0700 as excerpted:

> Also, I don't see a way to say “use Git to sync, but keep a shallow
> repository”.

Presumably at some point we'd get the PORTAGE_GIT* equivalent of the 
PORTAGE_RSYNC* settings from make.conf, but settable in both make.conf 
for global git-type settings and repos.conf for individual repo settings.

The big step, basic git support, is already there, but it doesn't have 
the broad usage of rsync and thus people haven't yet run into all the 
corner-cases that triggered the development of all those rich rsync 
option settings we have access to these days.

Tho I don't think it should be a blocker to the git migration, as I guess 
it'll be pretty mandatory that we keep an rsync backport for at least the 
gentoo-standard year of support.  Immediately dropping stuff like 
changelogs from that rsync backport should make that simpler.  If people 
want them, they can arrange to use the user-git repo.

> On the other hand, it wouldn't be too terrible to force users to shallow
> their history manually whenever they felt like it.

True.

Which brings up another point.  A user-focused git migration guide 
document would be very nice and very gentoo. =:^)  Once we have the git 
thing up and running for the early-adopters who can more or less find 
their own way, I'd call that a blocker to the possibility of eventually 
turning off rsync entirely, for our stable users.  Again, *NOT* a blocker 
to turning on git for those who can find their own way, but to eventually 
turning off the rsync backward compatibility machinery for stable.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2540 bytes --]

On Tue, 16 Sep 2014 10:52:13 -0700
"W. Trevor King" <wking@tremily.us> wrote:

> On Tue, Sep 16, 2014 at 05:35:08PM +0000, Duncan wrote:
> > W. Trevor King posted on Mon, 15 Sep 2014 13:33:46 -0700 as
> > excerpted:
> > > On Mon, Sep 15, 2014 at 01:29:44PM -0700, W. Trevor King wrote:
> > >> I don't see any benefit to using rsync vs. a shallow clone as the
> > >> transmission protocol.
> > > 
> > > Other than the fact that before you dropped it you'd need to push
> > > a ‘emerge sync’ that could handle either rsync or Git, stabilize
> > > that Portage, and then wait for folks to adopt it.
> > 
> > Portage already handles it. =:^)
> 
> Oh, lovely :).  Looks like that landed in 2.2.0 with 47e8d22d (Add
> support for multiple repositories in `emerge --sync`, 2013-07-23).
> There are older Portages in the tree though (back to 2.1.6.7_p1), so
> you'd still want to wait until those were gone before dropping rsync.
> 
> Also, I don't see a way to say “use Git to sync, but keep a shallow
> repository”.  Ideally, we'd want:
> 
>   $ git clone --depth=1 git://git.gentoo.org/gentoo-portage.git
> 
> for the initial clone (modulo whatever URI), and:
> 
>   $ git pull --depth=1
> 
> for subsequent syncs.  pym/_emerge/actions.py currently hardcodes ‘git
> pull’ for the latter, and doesn't seem to have any code for the
> former.  On the other hand, it wouldn't be too terrible to force users
> to shallow their history manually whenever they felt like it.
> 
> Cheers,
> Trevor
> 

The depth option will be added to the new portage plugin-sync system in
final stages of development now.  

There will be 2 new repos.conf options 

1 for new repo install
	eg. git clone: --depth=1

another for sync options:
	eg. git pull: --rebase origin master
	this will allow changes to a repo in a different branch be
	updated from the master branch.

they will be repo specific options, not global.


The new system will allow
emerge-webrync type repos to be synced via emerge --sync instead of the
emerge-webrsync command.  Plus it will add an svn type and the ability
for a layman type which layman already has code for.  Layman is just
waiting for the new sync system to land in portage's master branch
before enabling it to be installed.  It will also allow for third party
sync modules to be created and easily installed.  So a squashfs sync
type could be created and installed for those repos where that a
squashfs tree is offered.

-- 
Brian Dolbec <dolsen>


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 605 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

Dnia 2014-09-16, o godz. 10:52:13
"W. Trevor King" <wking@tremily.us> napisał(a):

>   $ git pull --depth=1
> 
> for subsequent syncs.  pym/_emerge/actions.py currently hardcodes ‘git
> pull’ for the latter, and doesn't seem to have any code for the
> former.  On the other hand, it wouldn't be too terrible to force users
> to shallow their history manually whenever they felt like it.

This isn't a good idea at all :). For git, --depth=1 fetching
is the same thing as --depth=1 clone. This way, you refetch everything
rather than just getting the update.

Instead, plain 'pull' is more appropriate to just get the new objects.
However, we may want to strip the history afterwards to reduce
the clone size.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 3930 bytes --]

On Wed, Sep 17, 2014 at 10:36:45AM +0200, Michał Górny wrote:
> Dnia 2014-09-16, o godz. 10:52:13
> W. Trevor King napisał(a):
> >   $ git pull --depth=1
> > 
> > for subsequent syncs.  pym/_emerge/actions.py currently hardcodes
> > ‘git pull’ for the latter, and doesn't seem to have any code for
> > the former.  On the other hand, it wouldn't be too terrible to
> > force users to shallow their history manually whenever they felt
> > like it.
> 
> This isn't a good idea at all :). For git, --depth=1 fetching is the
> same thing as --depth=1 clone. This way, you refetch everything
> rather than just getting the update.

You don't refetch everything, but the pull fails because it doesn't
know how the original and new shallow commits are related (so it can't
(ff-)merge them).  It works if you fetch and reset (skipping the
merge).  Here's my test script:

  #!/bin/sh

  rm -rf upstream local-1 local-2 &&
  mkdir upstream &&
  (
    cd upstream &&
    git init &&
    echo 'Some project' >README &&
    git add README &&
    git commit -m 'Start the project' &&
    for X in 1 2 3 4 5 6 7 8 9
    do
				echo "${X}" >"${X}" &&
    		git add "${X}"
    done &&
    git commit -m 'Add a bunch of dummy files'
  ) &&
  git clone --depth 1 "file://${PWD}/upstream" local-1 &&
  (
    cd upstream &&
    echo 'Build with ...' >>README &&
    git commit -am 'Add building instructions' &&
    echo 'Test with ...' >>README &&
    git commit -am 'Add testing instructions'
  ) &&
  (
    cd local-1 &&
    git fetch --depth 1 &&
    git reset --hard origin/master &&
    git --no-pager log --oneline --decorate &&
    du -s . &&
    git reflog expire --expire=now --all &&
    git gc --aggressive --prune=now &&
    du -s .
  ) &&
  git clone --depth 1 "file://${PWD}/upstream" local-2 &&
  (
    cd local-2 &&
    du -s .
  )

and here are some excerpts of it's output:

* The shallow fetch only pulls in three objects (the new README, tree,
  and commit):

  remote: Counting objects: 3, done.
  remote: Compressing objects: 100% (2/2), done.
  remote: Total 3 (delta 1), reused 0 (delta 0)
  Unpacking objects: 100% (3/3), done.
  From file:///tmp/upstream
   + 73f6253...5abbe64 master     -> origin/master  (forced update)
  HEAD is now at 5abbe64 Add testing instructions

* After the shallow fetch and reset, local-1 has a shallow history:

  5abbe64 (grafted, HEAD, origin/master, origin/HEAD, master) Add testing instructions

* But it still has references to the old master in the reflog, and
  that takes up some space:

  168     .

* Expiring the reflog and garbage collectiong gets us that space back
  (although in practice, I'd just let Git expire these automatically
  in the course of time):

  Counting objects: 12, done.
  Delta compression using up to 2 threads.
  Compressing objects: 100% (2/2), done.
  Writing objects: 100% (12/12), done.
  Total 12 (delta 0), reused 9 (delta 0)
  140     .

* A fresh shallow clone gets all 12 objects (not just the three new
  ones):

  Cloning into 'local-2'...
  remote: Counting objects: 12, done.
  remote: Compressing objects: 100% (2/2), done.
  remote: Total 12 (delta 0), reused 0 (delta 0)
  Receiving objects: 100% (12/12), done.
  Checking connectivity... done.

* And takes up as much space as our garbage-collected local-1:

  140     .

Again, I'm happy to leave it to users to manually

  $ git fetch --depth 1
  $ git reset --hard origin/master

to shorten their history, but I expect many will not bother, and then
get annoyed as that unpurged history takes up more and more space ;).
In any case, we don't have to resolve this before the transition.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Mon, Sep 15, 2014 at 4:18 PM, Michał Górny <mgorny@gentoo.org> wrote:
>
> Can't we just kill rsync then? The whole ChangeLog seems to take more
> effort than the actual benefit it gives.
>

I'm not sure ditching rsync entirely is necessary - it might be more
trouble than it is worth as it is a very effective simple way to
distribute the tree.  However, I'm not really opposed to it either.

However, I do really question whether we need changelogs in rsync.  It
seems like many projects are going away from these - or doing what the
kernel is doing and just dumping a git log into them.  I don't think
we need to try to shoehorn the old changelogs into our git history -
I'd just leave them in the tree for migration and then prune then
post-migration.

Oh, in case it is useful to know, a full historical git bundle is
about 1.2GB, and a clone+checkout of the bundle uses about 2.1GB of
space.  A compressed cvs tarball with the full history is about 575MB
in comparison, though I see it has grown by about 50MB in the last six
months.  Bottom line is that non-shallow checkouts will need a decent
amount of space.  Then again, my tmpfs /usr/portage uses 735M just by
itself.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Rich Freeman:
> 
> I'm not sure ditching rsync entirely is necessary - it might be more
> trouble than it is worth as it is a very effective simple way to
> distribute the tree.  However, I'm not really opposed to it either.
> 

The few people I personally know who use gentoo never use rsync for
syncing, because I told them to never use it, unless they want random
backdoors in their tree.

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Anthony G. Basile]

On 09/15/14 16:49, Rich Freeman wrote:
> On Mon, Sep 15, 2014 at 4:18 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> Can't we just kill rsync then? The whole ChangeLog seems to take more
>> effort than the actual benefit it gives.
>>
> I'm not sure ditching rsync entirely is necessary - it might be more
> trouble than it is worth as it is a very effective simple way to
> distribute the tree.  However, I'm not really opposed to it either.

I can live with git only but I'm not sure what would happen if we tried 
this?  There are lots of users and scripts out there that assume rsync.  
That's one cold shower.

>
> However, I do really question whether we need changelogs in rsync.  It
> seems like many projects are going away from these - or doing what the
> kernel is doing and just dumping a git log into them.  I don't think
> we need to try to shoehorn the old changelogs into our git history -
> I'd just leave them in the tree for migration and then prune then
> post-migration.

We could just push out the word that ChangeLogs are going away and they 
have to read the git repo.  That might be the easiest solution.  I do 
have users that quote my ChangeLogs though.

>
> Oh, in case it is useful to know, a full historical git bundle is
> about 1.2GB, and a clone+checkout of the bundle uses about 2.1GB of
> space.  A compressed cvs tarball with the full history is about 575MB
> in comparison, though I see it has grown by about 50MB in the last six
> months.  Bottom line is that non-shallow checkouts will need a decent
> amount of space.  Then again, my tmpfs /usr/portage uses 735M just by
> itself.
>
> --
> Rich
>


-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

Anthony G. Basile posted on Mon, 15 Sep 2014 17:43:09 -0400 as excerpted:

> We could just push out the word that ChangeLogs are going away and they
> have to read the git repo.  That might be the easiest solution.  I do
> have users that quote my ChangeLogs though.

As such a user... 

Given the proposed three-level system, dev-git, power-user-git, general-
user-rsync, dumping changelogs in rsync is my preferred solution as 
well.  Given power-user-git access I expect anyone who actually reads 
changelogs will be switching to the power-user-git level in a heartbeat, 
and I can't see the folks remaining on rsync actually caring about 
changelogs, either.  In the one-off case they find themselves needing a 
changelog, they can read the git commit log online.

When the discussion came up previously I was strongly in favor of keeping 
changelogs, because users /don't/ currently have a reasonable 
alternative.  With the proposed three-level system, the power-user git 
repo will be that alternative, and the changelogs can simply go away in 
favor of people actually having access to a full git repo and git log.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] gentoo git workflow

[hasufell]

Rich Freeman:
> 
> This is one of the blockers.  We haven't actually decided how we want
> to use git.
> 

There are IMO 3 main things to consider for a git workflow:
* commit policy
* branching model
* remote model
(and history format somewhere implicitly)

Deciding on a _commit policy_ should be fairly straightforward and we
already have one point
* gpg sign every commit (unless it's a merged branch, then we only care
about the merge commit)

More things to consider for commit policy are:
* commit message format (line length, maybe prepend category/PN?)
* do we expect repoman to run successfully for every commit (I'd say no)?
* additional information that must be provided
* when to force/avoid merge commits

Deciding on _branching model_ should be pretty easy here too. We are
mainly working on master and there may be developer-specific branches etc.
History does not need to be linear.
Creating additional branches is up to developers and there are no
specific rules about that.

The _remote model_ is: use a central repository with every developer
having push access. I think this is pretty reasonable for our use case,
although I'd love to see a linux-like workflow with enforced reviews
that propagate through project members/leads. But I'm not sure we need
that much overhead, except for non-trivial stuff like eclasses where we
already require reviews (well, more or less).


The only hard part is that people have to know the differences between
merging/rebasing, fast-forward merges, non-fast-forward merges etc. and
when and when not to do them.

'git rebase' is a powerful thing, but also pretty good to mess up your
local history if used wrong.

I think we can write up a gentoo-specific guide in 2-3 weeks.

Re: [gentoo-dev] gentoo git workflow

[Rich Freeman]

On Sun, Sep 14, 2014 at 11:11 AM, hasufell <hasufell@gentoo.org> wrote:
>
> The only hard part is that people have to know the differences between
> merging/rebasing, fast-forward merges, non-fast-forward merges etc. and
> when and when not to do them.
>
> 'git rebase' is a powerful thing, but also pretty good to mess up your
> local history if used wrong.
>
> I think we can write up a gentoo-specific guide in 2-3 weeks.
>

Sounds good.  I think one thing we need to get over with the whole git
migration is the fact that it isn't going to be perfect.  We probably
will find minor errors in the migration itself, little glitches in the
back-end stuff, problems in the proposed workflow, and so on.  We're
just going to have to adapt.  We've been using cvs for eons and have
learned to ignore its shortcomings and have well-polished workflows.

It isn't like there are 500 devs doing commits every day.  We're a
reasonably tight community and we're just going to have to work
together to get over the inevitable bumps.

It may make sense to just start out with guidelines in the beginning,
and then we can turn them into rules when problems actually come up.
Once upon a time there wasn't a hard rule about changelog entries for
removals/etc, and the world didn't end, but we decided that having the
rule made more sense than not having it.  With git we should expect
more of the same - we won't get it 100% right out of the gate.

--
Rich

Re: [gentoo-dev] gentoo git workflow

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 830 bytes --]


> Deciding on a _commit policy_ should be fairly straightforward and we
> already have one point
> * gpg sign every commit (unless it's a merged branch, then we only care
> about the merge commit)

+1

> More things to consider for commit policy are:
> * commit message format (line length, maybe prepend category/PN?)

this could be done in part by repoman... having a meaningful shortlog would be 
nice.

> * do we expect repoman to run successfully for every commit (I'd say no)?

commit no, push yes?

> * additional information that must be provided
> * when to force/avoid merge commits

my take- disallow (by policy) nontrivial rebases by third parties, encourage 
trivial rebases



-- 
Andreas K. Huettel
Gentoo Linux developer (council, kde)
dilfridge@gentoo.org
http://www.akhuettel.de/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] gentoo git workflow

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

On Sun, Sep 14, 2014 at 08:04:12PM +0200, Andreas K. Huettel wrote:
> 
> > Deciding on a _commit policy_ should be fairly straightforward and we
> > already have one point
> > * gpg sign every commit (unless it's a merged branch, then we only care
> > about the merge commit)
> 
> +1

Merge commits only happen if we allow non-fast-forward merges. I would
personally be against allowing merge commits on the master branch.

> 
> > More things to consider for commit policy are:
> > * commit message format (line length, maybe prepend category/PN?)
> 
> this could be done in part by repoman... having a meaningful shortlog would be 
> nice.
 
 I don't see how repoman could do anything about this, but here is a
 good description of how to write git commit messages [1].

> > * do we expect repoman to run successfully for every commit (I'd say no)?
> 
> commit no, push yes?

+1, every time we push that should indicate a successful repoman run.

> > * additional information that must be provided

I'm not sure what additional information is being referred to.

> > * when to force/avoid merge commits
 
 I would be against merge commits on the master branch; everything
 should be a fast-forward merge.

> my take- disallow (by policy) nontrivial rebases by third parties, encourage 
> trivial rebases

Rebases involving commits that are already pushed to master probably
shouldn't be allowed.

However, rebasing changes *on* master, before they are pushed, is a good
thing, because that kills non-fast-forward merges.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] gentoo git workflow

[hasufell]

William Hubbs:
> On Sun, Sep 14, 2014 at 08:04:12PM +0200, Andreas K. Huettel wrote:
>>
>>> Deciding on a _commit policy_ should be fairly straightforward and we
>>> already have one point
>>> * gpg sign every commit (unless it's a merged branch, then we only care
>>> about the merge commit)
>>
>> +1
> 
> Merge commits only happen if we allow non-fast-forward merges. I would
> personally be against allowing merge commits on the master branch.
> 

Allowing fast-forward merges will break signature verification if you
fetched from a user repo.
If we don't allow merge commits, then _every_ commit hast to be signed
by a gentoo dev (e.g. by using git-am). I don't see much sense in this.
It will rather complicate workflow.

The currently proposed verification script skips branch 'B', so what
matters is the signature of the merge commit which say "yes, I have
reviewed the users branch(es) and it's fine".

Merging from branches holds useful information. A linear history isn't
necessarily easier to understand, so from me linear history gets a

-1

It just isn't really "git" to me. But it also requires people to know
when to avoid merge commits.

> 
> Rebases involving commits that are already pushed to master probably
> shouldn't be allowed.
> 

Of course, yes. That has to be documented in a gentoo developer git guide.

Re: [gentoo-dev] gentoo git workflow

["C. Bergström"]

On 09/15/14 02:34 AM, hasufell wrote:
> William Hubbs:
>> On Sun, Sep 14, 2014 at 08:04:12PM +0200, Andreas K. Huettel wrote:
>>>> Deciding on a _commit policy_ should be fairly straightforward and we
>>>> already have one point
>>>> * gpg sign every commit (unless it's a merged branch, then we only care
>>>> about the merge commit)
>>> +1
>> Merge commits only happen if we allow non-fast-forward merges. I would
>> personally be against allowing merge commits on the master branch.
>>
> Allowing fast-forward merges will break signature verification if you
> fetched from a user repo.
> If we don't allow merge commits, then _every_ commit hast to be signed
> by a gentoo dev (e.g. by using git-am). I don't see much sense in this.
> It will rather complicate workflow.
>
> The currently proposed verification script skips branch 'B', so what
> matters is the signature of the merge commit which say "yes, I have
> reviewed the users branch(es) and it's fine".
>
> Merging from branches holds useful information. A linear history isn't
> necessarily easier to understand, so from me linear history gets a
>
> -1
>
> It just isn't really "git" to me. But it also requires people to know
> when to avoid merge commits.
>
>> Rebases involving commits that are already pushed to master probably
>> shouldn't be allowed.
>>
> Of course, yes. That has to be documented in a gentoo developer git guide.
Pretty please do NOT allow "merge" commits.. they are the bane of evil 
for the long term ability to have any sane work-flow. Trying browsing a 
commit history after a big merge commit.. or following the parent..

lastly - the "merge" commit itself could be very confusing to some 
people when viewed in github. (At least personally I find them 
frequently unreadable)

After 5 years of git where I work - they are now banned (policy) and I 
wish github would allow them to be banned (non-fast forward) to avoid 
mistakes

There's a big debate between merge vs rebase.. I'm not trying to go down 
the benefits of one workflow vs the other.. However, if rebase fails.. 
you can allow merge commits in the future.. The opposite isn't easily 
accomplished without squashing history and losing stuff..

Re: [gentoo-dev] gentoo git workflow

[hasufell]

"C. Bergström":
> Pretty please do NOT allow "merge" commits.. they are the bane of evil
> for the long term ability to have any sane work-flow.

It works pretty well for the linux kernel.

Ofc it's a matter of actually handling it. If people are unable to
properly handle tools/methods, everything could become the bane of evil,
no matter what tool you use.

> There's a big debate between merge vs rebase

I think most of those debates are nonsense. Both methods have their use
cases. But it matters when to use them. A lot of people don't even know
how to actually rebase, so they end up causing merge commits for
everything which will lead to a _very_ confusing history. Simply banning
that method is not a solution in my opinion.

The solution is to make a clear policy or recommendations when to use
one of them.

[gentoo-dev] Re: gentoo git workflow

[Duncan]

C. Bergström posted on Mon, 15 Sep 2014 02:49:48 +0700 as excerpted:

> Pretty please do NOT allow "merge" commits.. they are the bane of evil
> for the long term ability to have any sane work-flow. Trying browsing a
> commit history after a big merge commit.. or following the parent..

You just called the inventor of git and the workflow it was designed to 
manage insane.  If that's the case arguably quite a bit more insanity 
would be a good thing! =:^)

Try git log --color --stat --graph on the mainline kernel in a terminal 
and read only the main merge-commit logs unless that merge is something 
of special interest you want more info on.  It actually makes following a 
full kernel cycle, including the commit window, drilling down to sub-
merges and individual commits only on 2-3 areas of interest while keeping 
a general awareness of developments in the rest of the kernel not only 
practical once again, but relatively easy.  Without seeing merge-commits 
it was a LOT harder.  I know as I've done it both ways, and while I can 
get around in git to some extent, my git skills are definitely nothing 
special!  =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] gentoo git workflow

[Andreas K. Huettel]

> 
> However, rebasing changes *on* master, before they are pushed, is a good
> thing, because that kills non-fast-forward merges.
> 

Nontrivial rebases *on* master can be problematic because you're changing 
history. 

Imagine you pull some nice commits from a user. Then at some point you will 
have to rebase them before you push them. If this fails and requires manual 
interaction, the original version of the commits is lost (including 
signatures) and errors are not traceable. 

With a merge instead any manual intervention is clearly located in the merge 
commit and the authors of each change are uniquely identifiable.

-- 
Andreas K. Huettel
Gentoo Linux developer (council, kde)
dilfridge@gentoo.org
http://www.akhuettel.de/

Re: [gentoo-dev] gentoo git workflow

[hasufell]

Andreas K. Huettel:
>>
>> However, rebasing changes *on* master, before they are pushed, is a good
>> thing, because that kills non-fast-forward merges.
>>
> 
> Nontrivial rebases *on* master can be problematic because you're changing 
> history. 
> 
> Imagine you pull some nice commits from a user. Then at some point you will 
> have to rebase them before you push them. If this fails and requires manual 
> interaction, the original version of the commits is lost (including 
> signatures) and errors are not traceable. 
> 
> With a merge instead any manual intervention is clearly located in the merge 
> commit and the authors of each change are uniquely identifiable.
> 

So, I've tried to come up with a more detailed approach. Some people
mentioned that we maybe shouldn't do rebases at all, because git already
does trivial rebases automatically, but I'd try with them first.


== commit policy ==

* atomic commits (one logical change)
* commits may span across multiple ebuilds/directories if it's one
  logical change
* every commit on the first parent of the history must be gpg signed by
  a gentoo dev
* repoman must be run from all related directories (or the top-level
  directory) on the latest commit that is being pushed

=== commit message format ===
* all lines max 70-75 chars
* first line brief explanation
* second line always empty
* optional detailed multiline explanation must start at the third line
* for commits that affect only a single package, prepend
  "CATEGORY/PN: " to the first line
* for commits that affect only the profile directory, prepend
  "profiles: " to the first line
* for commits that affect only the eclass directory, prepend
  "ECLASSNAME.eclass: " to the first line
* for commits that affect only licenses directory, prepend
  "licenses: " to the first line
* for commits that affect only metadata directory, prepend
  "metadata: " to the first line

== branching model ==

* the primary production-ready branch is master (users will pull from
  here), there are no non-fast-forward pushes allowed
* there may be developer-specific, task-specific, project-specific
  branches etc (there are currently no specific rules about them)

=== when/how to rebase ===

* in case of a non-fast-forward push conflict to remote master, try
  'git pull --rebase=preserve' first, if that yields complicated
  conflicts, abort the rebase and continue with a regular merge
* use --preserve to preserve merges during a rebase (if appropriate,
  e.g. for user branches)
* don't use --preserve if you do an interactive rebase (see BUGS in
  git-rebase manpage)
* commits that are not on the remote master branch yet may be
  rewritten/squashed/splitted etc via interactive rebase, however the
  rebase must never span beyond those commits
* never rebase on already pushed commits
* there are no particular rules for rebasing on non-master remote
  branches, but be aware that others might base their work on them
* there are no particular rules for rebasing non-remote branches, as
  long as they don't clutter the history when merged back into master
* don't do complicated rebases to avoid a merge commit at all cost (it
  may even cause losing information, e.g. user signatures)

=== when/how to merge ===

* if a rebase fails or is too complicated, do a regular merge instead
* if the information is useful (e.g. pulled from a foreign remote user
  branch or merged a non-trivial eclass conversion back into master)
  force a merge commit (non-fast-forward merge via '--no-ff')
* to avoid a merge commit (e.g. information is not useful), you may try
  to force a fast-forward merge by first rebasing the feature branch
  (or similar) against master and then merging it into master
* extend merge commit messages with useful information, e.g. how
  conflicts were solved
* keep in mind that all commits of the first parent of the history must
  be gpg signed by a gentoo dev, so you may want to force merge commits
  especially for user branches

== remote model ==

We have a main developer repo where developers work & commit (every
developer has direct push access). For every push into developer repo,
automated magic thingie merges stuff into user sync repo and updates the
metadata cache there.

User sync repo is for power users than want to fetch via git. It's quite
fast and efficient for frequent updates, and also saves space by being
free of ChangeLogs.

On top of user sync repo rsync is propagated. The rsync tree is
populated with all old ChangeLogs copied from CVS (stored in 30M git
repo), new ChangeLogs are generated from git logs and Manifests are
expanded.

Re: [gentoo-dev] gentoo git workflow

[Rich Freeman]

On Mon, Sep 15, 2014 at 7:37 AM, hasufell <hasufell@gentoo.org> wrote:
> * repoman must be run from all related directories (or the top-level
>   directory) on the latest commit that is being pushed

This should be clarified.  Does repoman need to be run on the exact
commit that is being pushed, or perhaps on a "parent" commit prior to
rebasing/merging into the master branch?  (I use parent liberally
here, since that commit wouldn't be an actual parent if it were
rebased.)

The reason I ask is that repoman takes a while to run (especially on
the full tree), and if we don't want non-fast-forward pushes then
you'll have to keep repeating the pull ; merge ; repoman ; push cycle
until you finally complete it in less time than it takes somebody else
to do a push.  Today people do not run on the tree as it exists the
instant before commit because cvs makes no guarantees about tree
consistency at all (basically cvs is like git configured to accept
non-fast-forward commits).

>
> == branching model ==
>
> * the primary production-ready branch is master (users will pull from
>   here), there are no non-fast-forward pushes allowed
> * there may be developer-specific, task-specific, project-specific
>   branches etc (there are currently no specific rules about them)

I suggest we at least toss out some kind of naming convention to prevent chaos.
How about dev/<name> as the namespace for devs acting as individuals
(devs can do whatever they want below this), and project/<name> as the
namespace for projects (which can also do whatever they want below
this).  If we missed anything devs should discuss on-list before just
creating random branch names.  I don't really want to contrain what
people do here - just try to organize it at least a tiny bit.

--
Rich

Re: [gentoo-dev] gentoo git workflow

[hasufell]

Rich Freeman:
> On Mon, Sep 15, 2014 at 7:37 AM, hasufell <hasufell@gentoo.org> wrote:
>> * repoman must be run from all related directories (or the top-level
>>   directory) on the latest commit that is being pushed
> 
> This should be clarified.  Does repoman need to be run on the exact
> commit that is being pushed, or perhaps on a "parent" commit prior to
> rebasing/merging into the master branch?  (I use parent liberally
> here, since that commit wouldn't be an actual parent if it were
> rebased.)
> 

Yes, you have to rerun repoman after a rebase or merge. On the tip of
the local master branch (as in: right before you try to push).

Sure, this may lead to problems if repoman takes long... but that's on
purpose. If your changes are that big, then they should be communicated
and coordinated properly without people randomly pushing changes in
between that may break yours.

That's no different from what we are doing right now, except that we
have now enforced consistency instead of "maybe repoman is correct,
maybe not".


>>
>> == branching model ==
>>
>> * the primary production-ready branch is master (users will pull from
>>   here), there are no non-fast-forward pushes allowed
>> * there may be developer-specific, task-specific, project-specific
>>   branches etc (there are currently no specific rules about them)
> 
> I suggest we at least toss out some kind of naming convention to prevent chaos.
> How about dev/<name> as the namespace for devs acting as individuals
> (devs can do whatever they want below this), and project/<name> as the
> namespace for projects (which can also do whatever they want below
> this).  If we missed anything devs should discuss on-list before just
> creating random branch names.  I don't really want to contrain what
> people do here - just try to organize it at least a tiny bit.
> 

Makes sense. If there are no major disagreements, I'll open a wiki page
for gentoo git workflow in a few days... as a draft ofc that is open to
changes.

Re: [gentoo-dev] gentoo git workflow

[Rich Freeman]

On Mon, Sep 15, 2014 at 9:13 AM, hasufell <hasufell@gentoo.org> wrote:
> Yes, you have to rerun repoman after a rebase or merge. On the tip of
> the local master branch (as in: right before you try to push).
>
> Sure, this may lead to problems if repoman takes long... but that's on
> purpose. If your changes are that big, then they should be communicated
> and coordinated properly without people randomly pushing changes in
> between that may break yours.

What do you mean by proper coordination?  Are you suggesting that we
have scheduled downtime anytime we do a kde stablereq or something
like that, once per arch?

Simply announcing the change isn't going to make repoman run any
faster.  If you try to stablize something like kde and the packages
are in more than one category, then you will need something like
10-20min of tree downtime to do a commit if you have to do a repoman
scan in the middle on spinning disks.  If somebody so much as does a
whitespace fix on a completely unrelated package you won't get a
fast-forward commit.

I'm all for consistency, but repoman against the entire tree is not
fast, and if you ban non-fast-forward commits then effectively all git
commits collide.  Combining the two makes the collision window
unsuitable for commits even in the middle of the night.

So, unless we want to schedule all multi-category commits and build
some kind of mechanism to even enforce this, I think we'll have to
live with trusting devs to just run repoman before their final
merge/rebase, and then they can use their judgment about whether it
will cause problems.  I'm not saying that many big changes shouldn't
be coordinated, but that shouldn't apply to every kde stablereq/etc.
I think that this will still be a big improvement on what we have
today.

--
Rich

Re: [gentoo-dev] gentoo git workflow

[hasufell]

Rich Freeman:
> On Mon, Sep 15, 2014 at 9:13 AM, hasufell <hasufell@gentoo.org> wrote:
>> Yes, you have to rerun repoman after a rebase or merge. On the tip of
>> the local master branch (as in: right before you try to push).
>>
>> Sure, this may lead to problems if repoman takes long... but that's on
>> purpose. If your changes are that big, then they should be communicated
>> and coordinated properly without people randomly pushing changes in
>> between that may break yours.
> 
> What do you mean by proper coordination?  Are you suggesting that we
> have scheduled downtime anytime we do a kde stablereq or something
> like that, once per arch?
> 

I don't know how kde stablereqs work.

> Simply announcing the change isn't going to make repoman run any
> faster.  If you try to stablize something like kde and the packages
> are in more than one category, then you will need something like
> 10-20min of tree downtime to do a commit if you have to do a repoman
> scan in the middle on spinning disks.  If somebody so much as does a
> whitespace fix on a completely unrelated package you won't get a
> fast-forward commit.
> 

Whether the package is related or not has to be verified by someone. I
doubt that a lot of people will review the changeset after a rebase or
merge and I don't have a lot of faith in people doing things right
including myself.

> 
> So, unless we want to schedule all multi-category commits and build
> some kind of mechanism to even enforce this
> 

Why all? If the multi-category commits are so big that you actually have
to run repoman on top-level (you could as well run it on all related
ebuild directories or on category-level), then yes... those monster
pushes should definitely be scheduled.

I think top-level repoman checks will be the exception, even for
multi-category commits.

Re: [gentoo-dev] gentoo git workflow

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 15/09/14 09:13 AM, hasufell wrote:
> Rich Freeman:
>> On Mon, Sep 15, 2014 at 7:37 AM, hasufell <hasufell@gentoo.org>
>> wrote:
>>> * repoman must be run from all related directories (or the
>>> top-level directory) on the latest commit that is being pushed
>> 
>> This should be clarified.  Does repoman need to be run on the
>> exact commit that is being pushed, or perhaps on a "parent"
>> commit prior to rebasing/merging into the master branch?  (I use
>> parent liberally here, since that commit wouldn't be an actual
>> parent if it were rebased.)
>> 
> 
> Yes, you have to rerun repoman after a rebase or merge. On the tip
> of the local master branch (as in: right before you try to push).
> 
> Sure, this may lead to problems if repoman takes long... but that's
> on purpose. If your changes are that big, then they should be
> communicated and coordinated properly without people randomly
> pushing changes in between that may break yours.
> 
> That's no different from what we are doing right now, except that
> we have now enforced consistency instead of "maybe repoman is
> correct, maybe not".
> 

Thread TL;DR -- apologies if this has already been brought up and
answered.

I'm not that worried about the big (multi-package) commits, as it does
make sense we're going to have difficulty and lots of potential
conflicts there, but aren't we going to run into this issue just with
multiple people committing separate single-package commits at the same
time??


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlQW9wIACgkQ2ugaI38ACPBunQEAh5F1moJZnHhHEQDs8kiPuRBk
wwO2y9z6KEqKUPTzBAEA/A9Cv5Y444xC9tjmbUJ27qcRnziJgQnzu2K3YGKSCyL6
=Jc80
-----END PGP SIGNATURE-----

Re: [gentoo-dev] gentoo git workflow

[Rich Freeman]

On Mon, Sep 15, 2014 at 10:26 AM, Ian Stakenvicius <axs@gentoo.org> wrote:
> I'm not that worried about the big (multi-package) commits, as it does
> make sense we're going to have difficulty and lots of potential
> conflicts there, but aren't we going to run into this issue just with
> multiple people committing separate single-package commits at the same
> time??

So, if you require fast-forward commits (which I think is sensible),
then all commits collide.

The other factor here is the time to run repoman.  Running a repoman
scan on a single package only takes tens of seconds, which means we're
ok as long as we don't do commits every few seconds.

But yes, you're right that commits of unrelated packages can
potentially collide.

The issue with something like kde is that the stablereqs can involve
100+ packages in several categories.  Either you run repoman scan
against all of them individually, or you run it at the tree level, and
either way it is going to take a while to run (and if it is
disk-bound, running in parallel probably won't help much).

The reality is that a kde stabilization is not a huge risk.  The
packages are fairly contained, and they aren't system packages so if
something does go wrong recovery is generally easy.  I've yet to hear
of something like this going badly.  I think the worst I've seen is
the odd package getting missed on the first pass.

In any case, I don't have any hard numbers to prove that requiring
repoman scans on every commit is a bad idea.  However, I think that it
has a large potential to make larger-scale changes difficult.  It
would make a lot more sense if we had a release-oriented strategy,
even if releases were hourly/daily/etc.

--
Rich

Re: [gentoo-dev] gentoo git workflow

[hasufell]

Rich Freeman:
> It would make a lot more sense if we had a release-oriented strategy,
> even if releases were hourly/daily/etc.
> 

If we are going that way, then we should think over the whole branching
model. I have a few things in mind, but I think we are already
fine-tuning stuff here that can still be fine-tuned later.

The workflow will have to be adjusted after git deployment anyway. We
cannot foresee everything (e.g. maybe this will be a non-issue if
projects accumulate changes and push them in intervals, as pointed out
in my previous post).

If it doesn't work out with rerunning repoman, then we just tell people
it's fine without it and think about alternative approaches, like
creating additional branches, doing releases or whatnot.

Re: [gentoo-dev] gentoo git workflow

[Rich Freeman]

hasufell <hasufell@gentoo.org> wrote:
> * allow inconsistency and broken states as we do now with CVS (and rely
>   on QA to run a repoman tinderbox and reverse-fixing broken crap)
...
> Rich Freeman:
>> It would make a lot more sense if we had a release-oriented strategy,
>> even if releases were hourly/daily/etc.
>>
>
> If we are going that way, then we should think over the whole branching
> model. I have a few things in mind, but I think we are already
> fine-tuning stuff here that can still be fine-tuned later.

Apologies for replying to two different emails in one here, but these
are related for the purpose of this reply.

I'm all for proposals to improve the way Gentoo works.  However, part
of the reason that the git migration keeps on never happening is that
there is this general desire out there to tie it to some kind of
complete transformation in how we operate.

Today, we don't have the same kind of tree-consistency you're
advocating.  So, moving to git without achieving this kind of
consistency is not a regression.

Today, we don't have some kind of completely airtight
everything-is-gpg-signed secure code flow.  So, moving to git without
achieving that is also not a regression.

If we want to have a discussion around whether Gentoo would be better
off if we were more release-based (even if those releases were
frequent/automated/etc), or about how to improve the security of our
code base, I think those would be very healthy discussions to have.
However, I don't think we should tie them to the git migration.

Simply moving to git while keeping just about everything else the same
will be a fairly disruptive change, and as we've already seen in this
thread there are some who just prefer cvs (though I think they're in
the minority by far).  If we try to make several other big changes at
the same time I just think that it will never happen.

I suggest we just get git working in a fashion that is "good enough."
I think that will already bring an increased level of consistency
compared to our current cvs workflow, where people run repoman from
cvs checkouts where you can't even ensure that any two devs have quite
the same trees at the same time.

There is nothing that keeps us from moving from that model to one that
is more revolutionary at another time.  However, trying to do it all
at once probably means that none of us get anything we're looking for,
because we're not going to do it with cvs and we'll never get off of
cvs unless simply doing that by any means necessary is the imperative.

--
Rich

Re: [gentoo-dev] gentoo git workflow

[hasufell]

Rich Freeman:
> 
> I suggest we just get git working in a fashion that is "good enough."

Sure, that's what I've been saying. Otherwise I'd propose to remove
access for everyone and only grant project leads/reviewers direct push
access. But as said... we are not ready for something like that.

However, gpg signature verification is trivial to implement (and is
already) and starting with stricter policies is worth a try. Nothing of
that is really a big deal.

But... starting release-oriented strategies IS a big deal.

Re: [gentoo-dev] gentoo git workflow

[hasufell]

Ian Stakenvicius:
> 
> I'm not that worried about the big (multi-package) commits, as it does
> make sense we're going to have difficulty and lots of potential
> conflicts there, but aren't we going to run into this issue just with
> multiple people committing separate single-package commits at the same
> time??
> 

Yes. If that becomes a major problem, then we have to say goodbye to the
central repository workflow, because we either
* allow inconsistency and broken states as we do now with CVS (and rely
  on QA to run a repoman tinderbox and reverse-fixing broken crap)
* require devs to rebase+merge and rerun repoman 10+ times in order to
  get a single commit pushed

It is already pretty gross and broken to have ~200 people have direct
push access to a single repository.

However, having ~150 commits per day I don't think this commit rate is
high enough to deadlock push access.

git pull --rebase=preserve && repoman full && git push origin master

or something similar. This can even be implemented a bit more sanely
within repoman or another script, but I don't think we need that yet.

It's also possible that projects start accumulating changes in branches
or even their own repositories to minimize the amount of pushes to
master. The merge-rate will be something that has to be dynamically
figured out and will improve tree quality.

We shouldn't try to map our CVS workflow 1:1 to git, so there have to be
changes in workflow. The optimum would be to have a real review workflow
enforced with only a handful of people having direct push access, but I
doubt this will work in our current community and mindset.
However, such a review workflow can dynamically evolve, because it's git
after all! If that happens, we can just make it a policy at some point
or even implement it on infra-level.

[gentoo-dev] Re: [gentoo-scm] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Mon, Sep 15, 2014 at 11:50 PM, Brian Harring <ferringb@gmail.com> wrote:
>
> On Sun, Sep 14, 2014 at 10:33 AM, Rich Freeman <rich0@gentoo.org> wrote:
>>
>> On Sun, Sep 14, 2014 at 8:03 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> >
>> > Of course, that assumes infra is
>> > going to cooperate quickly or someone else is willing to provide the
>> > infra for it.
>>
>> The infra components to a git infrastructure are one of the main
>> blockers at this point.  I don't really see cooperation as the issue -
>> just lack of manpower or interest.
>
>
> I can't speak for current gentoo infra, but it'd help to think through
> exactly who will have RO access to the git repo.
>
> Trying to have the entire gentoo userbase accessing the repo via git is
> going to be resource intensive- if you're intending that, well, start
> donating frankly.
>
> If however you're angling for just devs having git access, and utilizing the
> existing rsync mirror, that should be a helluva lot more doable (even with
> shitty hardware).

I believe that is the proposal here.  There would be a dev-only main
repository which is where all commits would go.  Then there would be a
power-user sync repo derived from that, and then an rsync tree derived
from that.  This isn't all that different from what we do with cvs.

Git is pretty easy to mirror, since it is after all a distributed
system at its heart.  It would probably make sense to do one push/pull
from the main repository to the power-user repo, do whatever magic
needs to be done with it, and then push out from there to a mirror
network, github, wherever.  We'd probably want mirrors of both the
original dev tree and the one with metadata and such.


>> People love to point out linux and its insane commit rate.  The thing
>> is, the mainline git repo with all those commits has exactly one
>> committer - Linus himself.  They don't have one big repo with one
>> master branch that everybody pushes to.  At least, that is my
>> understanding (and there are certainly others here who are more
>> involved with kernel development).
>
>
> To be frank, I think you're seriously overthinking it and worrying about a
> nonissue.  Worrying about this while ignoring security enhancements like
> requiring signed commits on push is a bit weird to me.

So far the discussion in the thread has included the commit signing issue.

I tend to agree that commit rate probably won't be a problem after
some discussion, as long as we don't require a repoman check 100% of
the time in-between pull and push.  For a single package that won't be
a problem, but doing it at the category/tree level is just going to
make collisions inevitable.

> One additional point no one has mentioned; when cutting over to git,
> afterwards someone will need to go through and rip out all of the cvs $Id
> style tags that exist in the tree- depending on how folks are doing the
> conversion, it might be worth just jamming that in right off the bat rather
> than trying to clean the tree afterwards.

I'd suggest that we not go too much further with editing history -
consolidating Manifest/package commits did make sense, and the other
fixes do as well.  I'd prefer not to go trying to remove keywords from
the tree during conversion.  We've already had all kinds of headaches
from keywords as it is (especially with patches/etc).  I suggest that
we initially convert the tree and leave the keywords in, and we can
always clean them up later, either by script or manually.  I'd suggest
keeping scripting to a minimum guaranteed-safe level (such as just our
ebuild headers), and leave it up to maintainers to get any stray ones.

In-band signaling is lousy design in this day and age.  Oh, did
somebody bring up Changelogs?  :)

> ~ferringb, who was enjoying his retirement very much thank you

I appreciate your email - there is a lot of history we can stand to
benefit from.  I think I have a container set up now that can
efficiently migrate cvs->git using your scripts, and I'm just doing
some more testing and work to make it fully transportable.  From there
we just need to figure out where to run it when the time comes, which
should be the easy part.

I do believe that mgorny was offering to step up to help out with the
scripting though.  If we have the scripts/hooks needed to manipulate
gentoo-x86 to produce the various trees, then I think that
transplanting them to infra will not be a difficult next step.  Really
that part is the part which is most lacking right now, so if a few are
willing to step up on this I think we can get somewhere.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[James Cloos]

>>>>> "MG" == Michał Górny <mgorny@gentoo.org> writes:

MG> This means we don't have to wait till someone figures out the perfect
MG> way of converting the old CVS repository. You don't need that history
MG> most of the time, and you can play with CVS to get it if you really do.
MG> In any case, we would likely strip the history anyway to get a small
MG> repo to work with.

+1 on that.  The cvs repo can be converted to an historical git repo on
a slower timeframe, and remain available as cvs until then.

That old-vs-fresh concept worked fine for other projects (including Linux).

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[vivo75]

Il 14/09/2014 14:03, Michał Górny ha scritto:
> The rsync tree
> --------------
>
> We'd also propagate things to rsync. We'd have to populate it with old
> ChangeLogs, new ChangeLog entries (autogenerated from git) and thick
> Manifests. So users won't notice much of a change.
>
If this will change all Changelog the first rsync from the users will
generate a lot of traffic, rsync network need to be prepared

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Ivan Viso Altamirano]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

I think the better option Is to block rsync and force emerge-webrsync
.sended from a phone
Il 14/09/2014 14:03, Michał Górny ha scritto:
> The rsync tree
> --------------
>
> We'd also propagate things to rsync. We'd have to populate it with old
> ChangeLogs, new ChangeLog entries (autogenerated from git) and thick
> Manifests. So users won't notice much of a change.
>
If this will change all Changelog the first rsync from the users will
generate a lot of traffic, rsync network need to be prepared

[-- Attachment #2: Type: text/html, Size: 741 bytes --]

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

Michał Górny wrote:
> What I need others to do is provide the hosting for git repos.

I'm happy to set up repos on my git server with custom hooks and
accounts as needed.

It's probably not what we want long-term, but it might be useful as
proof of concept, so that infra only needs to do setup one time.

I even have some virtual hosting working, point an A to the right IP
and it looks like only desired repos are hosted there.

Gitweb, git-daemon and git over http and CAcert https with pretty URLs.


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]