[gentoo-project] Council / Git Migration Agenda

[gentoo-project] Council / Git Migration Agenda

[Rich Freeman]

Starting a new thread for discussion:

On Thu, Oct 2, 2014 at 7:06 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Dnia 2014-10-01, o godz. 13:30:55
> Rich Freeman <rich0@gentoo.org> napisał(a):
>
>> On Tue, Sep 30, 2014 at 10:08 PM, Rich Freeman <rich0@gentoo.org> wrote:
>> > If you'd like to contribute another agenda item, please reply to this email.
>>
>> I'll offer up a further topic for the git migration.
>
> I think that there are a few issues that the Council may actually want
> to discuss.

I was thinking of an additional item also worth consideration.

So far the general plan has tended to be that we would do a full
historical git migration, and the last commit would just be the active
tree, which would then be further cleaned up (remove cvs headers,
changelogs, switch to thin manifests, etc).

I was thinking that it might make more sense to just make things
really simple and ONLY migrate the active tree into the starting git
repository.  That is, basically take the rsync tree, remove metadata,
and do a git init.  (Then follow that up with removing changelogs,
cleaning up cvs headers, and so on.)

A historical migration could be done in parallel and released a few
hours later.  However, it would not be a contiguous repository.  That
is, the converted active tree commit would not have any parents.  If
you wanted to have a contiguous tree you would need to splice in the
historical migration with git replace.

Rationale:
1.  The historical git migration right now is not perfect.  It
probably will never be perfect, and there is debatable value in even
trying to make it perfect.

2.  Somebody may very well want to improve on the historical git
migration.  If the converted repository doesn't favor any particular
historical migration, then anybody can swap in the historical
migration of their choosing.  We aren't locked into a point-in-time
migration that isn't the best that it could be.  Anybody who wants a
better migrated history can make their own.

3.  I fear that some are going to argue that we shouldn't do the
migration until the historical migration is better than it is now.
Just how much better it needs to be could be a matter of considerable
debate.  Not making a permanent connection between the historical and
active migration sidesteps this debate.

4.  Decoupling the historical and active migration makes the active
migration brain-dead simple.  Downtime will be minimized, and we don't
have the possibility of running into some issue hours into a migration
where we need to try to figure out how to handle it.  The active tree
will migrate and cut over as quickly as things can be swapped out.
The historical migrated tree will be posted whenever it is ready -
likely within hours but if it takes longer there really is no big
loss.  We can fix it as many times as we like.

5.  In general I think that we're well past the point of diminishing
returns.  I personally don't see any return on improving the
historical git tree beyond the general fun of tinkering with the code.
In the meantime many are eager to see the active tree migrate.  Let's
stop holding up moving forward by obsessing about looking backwards.

--
Rich

Re: [gentoo-project] Council / Git Migration Agenda

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1503 bytes --]

Dnia 2014-10-03, o godz. 20:00:24
Rich Freeman <rich0@gentoo.org> napisał(a):

> Starting a new thread for discussion:
> 
> On Thu, Oct 2, 2014 at 7:06 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > Dnia 2014-10-01, o godz. 13:30:55
> > Rich Freeman <rich0@gentoo.org> napisał(a):
> >
> >> On Tue, Sep 30, 2014 at 10:08 PM, Rich Freeman <rich0@gentoo.org> wrote:
> >> > If you'd like to contribute another agenda item, please reply to this email.
> >>
> >> I'll offer up a further topic for the git migration.
> >
> > I think that there are a few issues that the Council may actually want
> > to discuss.
> 
> I was thinking of an additional item also worth consideration.
> 
> So far the general plan has tended to be that we would do a full
> historical git migration, and the last commit would just be the active
> tree, which would then be further cleaned up (remove cvs headers,
> changelogs, switch to thin manifests, etc).
> 
> I was thinking that it might make more sense to just make things
> really simple and ONLY migrate the active tree into the starting git
> repository.  That is, basically take the rsync tree, remove metadata,
> and do a git init.  (Then follow that up with removing changelogs,
> cleaning up cvs headers, and so on.)

Not rsync, 'cvs up -dP [-kk]'. rsync has a lot of cruft and out-of-repo
experiences. Using cvs directly gives us better guarantees about match
between full conversion & snapshot.


-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-project] Council / Git Migration Agenda

[Dirkjan Ochtman]

On Sat, Oct 4, 2014 at 2:00 AM, Rich Freeman <rich0@gentoo.org> wrote:
> I was thinking that it might make more sense to just make things
> really simple and ONLY migrate the active tree into the starting git
> repository.  That is, basically take the rsync tree, remove metadata,
> and do a git init.  (Then follow that up with removing changelogs,
> cleaning up cvs headers, and so on.)
>
> A historical migration could be done in parallel and released a few
> hours later.  However, it would not be a contiguous repository.  That
> is, the converted active tree commit would not have any parents.  If
> you wanted to have a contiguous tree you would need to splice in the
> historical migration with git replace.

I think that would be sad. IMO there should be full history to the
default tree (even if we advocate shallow clones by default). Yes, the
history might not be perfect; people can splice in an improved history
later with git replace. I would be disappointed if the git hash for
the default tree doesn't represent (some version of) the full history.

Cheers,

Dirkjan

Re: [gentoo-project] Council / Git Migration Agenda

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]

Dnia 2014-10-05, o godz. 10:18:21
Dirkjan Ochtman <djc@gentoo.org> napisał(a):

> On Sat, Oct 4, 2014 at 2:00 AM, Rich Freeman <rich0@gentoo.org> wrote:
> > I was thinking that it might make more sense to just make things
> > really simple and ONLY migrate the active tree into the starting git
> > repository.  That is, basically take the rsync tree, remove metadata,
> > and do a git init.  (Then follow that up with removing changelogs,
> > cleaning up cvs headers, and so on.)
> >
> > A historical migration could be done in parallel and released a few
> > hours later.  However, it would not be a contiguous repository.  That
> > is, the converted active tree commit would not have any parents.  If
> > you wanted to have a contiguous tree you would need to splice in the
> > historical migration with git replace.
> 
> I think that would be sad. IMO there should be full history to the
> default tree (even if we advocate shallow clones by default). Yes, the
> history might not be perfect; people can splice in an improved history
> later with git replace. I would be disappointed if the git hash for
> the default tree doesn't represent (some version of) the full history.

I see two issues with this:

1. this will make the initial repo huge (~1.5G) and therefore hard to
mirror. GitHub has a 'soft' limit of 1G. Other mirror providers may
also be unhappy given the perspective of 1.5G and growing, compared to
70M and growing with potential cutoff in future.

2. Replacing the history with a better one would still keep the old
commits. That is, after we fix the history properly, the repo grows
another 1.5G, while the now-unused old 1.5G still needs to be kept.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-project] Council / Git Migration Agenda

[hasufell]

On 10/05/2014 10:18 AM, Dirkjan Ochtman wrote:
> On Sat, Oct 4, 2014 at 2:00 AM, Rich Freeman <rich0@gentoo.org>
> wrote:
>> I was thinking that it might make more sense to just make things 
>> really simple and ONLY migrate the active tree into the starting
>> git repository.  That is, basically take the rsync tree, remove
>> metadata, and do a git init.  (Then follow that up with removing
>> changelogs, cleaning up cvs headers, and so on.)
>> 
>> A historical migration could be done in parallel and released a
>> few hours later.  However, it would not be a contiguous
>> repository.  That is, the converted active tree commit would not
>> have any parents.  If you wanted to have a contiguous tree you
>> would need to splice in the historical migration with git
>> replace.
> 
> I think that would be sad. IMO there should be full history to the 
> default tree (even if we advocate shallow clones by default). Yes,
> the history might not be perfect; people can splice in an improved
> history later with git replace. I would be disappointed if the git
> hash for the default tree doesn't represent (some version of) the
> full history.
> 

We've been chasing the dream of perfect git migration for a few years
now without a definite result. It's time to make compromises unless
you want to stick with a tool from the '80s for the rest of your
gentoo life.

Re: [gentoo-project] Council / Git Migration Agenda

[Seemant Kulleen]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

Libraries don't have to be sad.  If the history remains in a CVS repo isn't
that the perfect home for it in the museum of gentoo's history?

We can honor it by keeping it: we needn't  carry it everywhere to remember
it.  :)

Cheers,

seemantk empathic design
http://seemantk.com
On Oct 5, 2014 1:18 AM, "Dirkjan Ochtman" <djc@gentoo.org> wrote:

> On Sat, Oct 4, 2014 at 2:00 AM, Rich Freeman <rich0@gentoo.org> wrote:
> > I was thinking that it might make more sense to just make things
> > really simple and ONLY migrate the active tree into the starting git
> > repository.  That is, basically take the rsync tree, remove metadata,
> > and do a git init.  (Then follow that up with removing changelogs,
> > cleaning up cvs headers, and so on.)
> >
> > A historical migration could be done in parallel and released a few
> > hours later.  However, it would not be a contiguous repository.  That
> > is, the converted active tree commit would not have any parents.  If
> > you wanted to have a contiguous tree you would need to splice in the
> > historical migration with git replace.
>
> I think that would be sad. IMO there should be full history to the
> default tree (even if we advocate shallow clones by default). Yes, the
> history might not be perfect; people can splice in an improved history
> later with git replace. I would be disappointed if the git hash for
> the default tree doesn't represent (some version of) the full history.
>
> Cheers,
>
> Dirkjan
>
>

[-- Attachment #2: Type: text/html, Size: 1979 bytes --]

Re: [gentoo-project] Council / Git Migration Agenda

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

>>>>> On Sun, 5 Oct 2014, Seemant Kulleen wrote:

> Libraries don't have to be sad.  If the history remains in a CVS
> repo isn't that the perfect home for it in the museum of gentoo's
> history?

Having the complete history in a single repository would be much
preferable, even if that history is not perfect.

It's annoying if you search for the point when some change was
performed, only to find that the repo's history doesn't reach back
that far. I've had this issue e.g. with the Portage repository whose
history was cut off at some point. Having to change tools (from git to
cvs) in addition doesn't make it better.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-project] Council / Git Migration Agenda

[Seemant Kulleen]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

I guess it boils down to priority.  History becomes inconvenient to trace
at some point.  That's true for most human things...nothing is perfect.

seemantk empathic design
http://seemantk.com
On Oct 5, 2014 10:48 PM, "Ulrich Mueller" <ulm@gentoo.org> wrote:

> >>>>> On Sun, 5 Oct 2014, Seemant Kulleen wrote:
>
> > Libraries don't have to be sad.  If the history remains in a CVS
> > repo isn't that the perfect home for it in the museum of gentoo's
> > history?
>
> Having the complete history in a single repository would be much
> preferable, even if that history is not perfect.
>
> It's annoying if you search for the point when some change was
> performed, only to find that the repo's history doesn't reach back
> that far. I've had this issue e.g. with the Portage repository whose
> history was cut off at some point. Having to change tools (from git to
> cvs) in addition doesn't make it better.
>
> Ulrich
>

[-- Attachment #2: Type: text/html, Size: 1331 bytes --]

Re: [gentoo-project] Council / Git Migration Agenda

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]

Dnia 2014-10-06, o godz. 07:48:26
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Sun, 5 Oct 2014, Seemant Kulleen wrote:
> 
> > Libraries don't have to be sad.  If the history remains in a CVS
> > repo isn't that the perfect home for it in the museum of gentoo's
> > history?
> 
> Having the complete history in a single repository would be much
> preferable, even if that history is not perfect.
> 
> It's annoying if you search for the point when some change was
> performed, only to find that the repo's history doesn't reach back
> that far. I've had this issue e.g. with the Portage repository whose
> history was cut off at some point. Having to change tools (from git to
> cvs) in addition doesn't make it better.

I was thinking of prepending the old history via 'git replace' in
the gitweb repo to allow looking back. However, that idea has
the downside that users would be confused by having past commits in
gitweb yet not in clones.

Kent improved my idea suggesting that we use a separate repo for that
'complete history' view. That is, we would have three repos:

1. history.git -- with past CVS history up to conversion,

2. dev.git -- with history starting with conversion,

3. joined-history.git -- dev.git with 'git replace' for history.git,
that is the complete history including both pre- and post-conversion
commits.


-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-project] Council / Git Migration Agenda

[Dirkjan Ochtman]

On Mon, Oct 6, 2014 at 10:48 AM, Michał Górny <mgorny@gentoo.org> wrote:
> I was thinking of prepending the old history via 'git replace' in
> the gitweb repo to allow looking back. However, that idea has
> the downside that users would be confused by having past commits in
> gitweb yet not in clones.
>
> Kent improved my idea suggesting that we use a separate repo for that
> 'complete history' view. That is, we would have three repos:
>
> 1. history.git -- with past CVS history up to conversion,
>
> 2. dev.git -- with history starting with conversion,
>
> 3. joined-history.git -- dev.git with 'git replace' for history.git,
> that is the complete history including both pre- and post-conversion
> commits.

Can we do that without requiring the git replace stuff? E.g. have
dev.git be a shallow clone of a joined-history.git?

Cheers,

Dirkjan

Re: [gentoo-scm] Re: [gentoo-project] Council / Git Migration Agenda

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

Dnia 2014-10-07, o godz. 09:57:32
Dirkjan Ochtman <djc@gentoo.org> napisał(a):

> On Mon, Oct 6, 2014 at 10:48 AM, Michał Górny <mgorny@gentoo.org> wrote:
> > I was thinking of prepending the old history via 'git replace' in
> > the gitweb repo to allow looking back. However, that idea has
> > the downside that users would be confused by having past commits in
> > gitweb yet not in clones.
> >
> > Kent improved my idea suggesting that we use a separate repo for that
> > 'complete history' view. That is, we would have three repos:
> >
> > 1. history.git -- with past CVS history up to conversion,
> >
> > 2. dev.git -- with history starting with conversion,
> >
> > 3. joined-history.git -- dev.git with 'git replace' for history.git,
> > that is the complete history including both pre- and post-conversion
> > commits.
> 
> Can we do that without requiring the git replace stuff? E.g. have
> dev.git be a shallow clone of a joined-history.git?

No, I don't think we can push to a shallow repo. Additionally, this
brings back all the issues I mentioned in the first mail.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-scm] Re: [gentoo-project] Council / Git Migration Agenda

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

>>>>> On Tue, 7 Oct 2014, Michał Górny wrote:

> Dnia 2014-10-07, o godz. 09:57:32
> Dirkjan Ochtman <djc@gentoo.org> napisał(a):

>> On Mon, Oct 6, 2014 at 10:48 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> > I was thinking of prepending the old history via 'git replace' in
>> > the gitweb repo to allow looking back. However, that idea has the
>> > downside that users would be confused by having past commits in
>> > gitweb yet not in clones.
>> >
>> > Kent improved my idea suggesting that we use a separate repo for
>> > that 'complete history' view. That is, we would have three repos:
>> >
>> > 1. history.git -- with past CVS history up to conversion,
>> >
>> > 2. dev.git -- with history starting with conversion,
>> >
>> > 3. joined-history.git -- dev.git with 'git replace' for
>> > history.git, that is the complete history including both pre- and
>> > post-conversion commits.
>>
>> Can we do that without requiring the git replace stuff? E.g. have
>> dev.git be a shallow clone of a joined-history.git?

> No, I don't think we can push to a shallow repo. Additionally, this
> brings back all the issues I mentioned in the first mail.

But dev (= master) and history could be two branches in the same
repository? So we wouldn't need three repos?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-scm] Re: [gentoo-project] Council / Git Migration Agenda

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

Dnia 2014-10-07, o godz. 10:20:27
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Tue, 7 Oct 2014, Michał Górny wrote:
> 
> > Dnia 2014-10-07, o godz. 09:57:32
> > Dirkjan Ochtman <djc@gentoo.org> napisał(a):
> 
> >> On Mon, Oct 6, 2014 at 10:48 AM, Michał Górny <mgorny@gentoo.org> wrote:
> >> > I was thinking of prepending the old history via 'git replace' in
> >> > the gitweb repo to allow looking back. However, that idea has the
> >> > downside that users would be confused by having past commits in
> >> > gitweb yet not in clones.
> >> >
> >> > Kent improved my idea suggesting that we use a separate repo for
> >> > that 'complete history' view. That is, we would have three repos:
> >> >
> >> > 1. history.git -- with past CVS history up to conversion,
> >> >
> >> > 2. dev.git -- with history starting with conversion,
> >> >
> >> > 3. joined-history.git -- dev.git with 'git replace' for
> >> > history.git, that is the complete history including both pre- and
> >> > post-conversion commits.
> >>
> >> Can we do that without requiring the git replace stuff? E.g. have
> >> dev.git be a shallow clone of a joined-history.git?
> 
> > No, I don't think we can push to a shallow repo. Additionally, this
> > brings back all the issues I mentioned in the first mail.
> 
> But dev (= master) and history could be two branches in the same
> repository? So we wouldn't need three repos?

You mean 'history' branch enabling 'git replace' or something more
ugly?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-project] Council / Git Migration Agenda

[Rich Freeman]

On Tue, Oct 7, 2014 at 3:57 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
>
> Can we do that without requiring the git replace stuff? E.g. have
> dev.git be a shallow clone of a joined-history.git?
>

That was the original plan I suggested abandoning in my email, for the
reasons stated in my email.  None of them had to do with repository
size, actually, though having a repository with an obsolete 1.5GB
history embedded certainly isn't ideal.

--
Rich

Re: [gentoo-project] Council / Git Migration Agenda

[Anthony G. Basile]

On 10/07/14 06:58, Rich Freeman wrote:
> On Tue, Oct 7, 2014 at 3:57 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
>> Can we do that without requiring the git replace stuff? E.g. have
>> dev.git be a shallow clone of a joined-history.git?
>>
> That was the original plan I suggested abandoning in my email, for the
> reasons stated in my email.  None of them had to do with repository
> size, actually, though having a repository with an obsolete 1.5GB
> history embedded certainly isn't ideal.
>
> --
> Rich
>

What does a shallow clone mean exactly?

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-project] Council / Git Migration Agenda

[Rich Freeman]

On Tue, Oct 7, 2014 at 7:00 AM, Anthony G. Basile <blueness@gentoo.org> wrote:
> On 10/07/14 06:58, Rich Freeman wrote:
>>
>> On Tue, Oct 7, 2014 at 3:57 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
>>>
>>> Can we do that without requiring the git replace stuff? E.g. have
>>> dev.git be a shallow clone of a joined-history.git?
>>>
>> That was the original plan I suggested abandoning in my email, for the
>> reasons stated in my email.  None of them had to do with repository
>> size, actually, though having a repository with an obsolete 1.5GB
>> history embedded certainly isn't ideal.
>>
>
> What does a shallow clone mean exactly?
>

It means a git repository that does not contain the full history.
That is, if you take one of the heads, such as master, and follow the
parent commits back, you eventually reach a commit that isn't actually
in the repository.  Git now has a --depth option to allow making them
easily in a clone operation.

Not all operations may work on them, but I've found conflicting
stories online about what is/isn't possible.  Generally you would
probably want full clones in all the official repositories to avoid
issues.

--
Rich

Re: [gentoo-project] Council / Git Migration Agenda

[Tom Wijsman]

On Mon, 6 Oct 2014 07:48:26 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:

> Having the complete history in a single repository would be much
> preferable, even if that history is not perfect.

Everything can be preferable, but the future cannot be perfect; ...