[gentoo-dev] stabilization commits and atomicity

[gentoo-dev] stabilization commits and atomicity

[hasufell]

I'd like to discuss whether we should allow/encourage stabilization
commits to be less atomic. They often bloat the history, make it hard to
skim through the summaries list and people who are looking for
stabilization probably do 'git log -- <ebuild-dir>' anyway, no? In
addition, I'm not sure the bug information where people post "stable"
comments is very useful.

At least, the previous commits on
  app-leechcraft/

could have been category-commits, since they all refer to the same thing
and the same bug.

I'd go so far to say allow people to do commits like:
"""
amd64 stabilizations

<optional list of bugs>
"""
possibly pre-pending the rough domain like "kde", if any. I think kde
herd already does that, no?

Unless someone thinks the mass-100+ commits are really useful in the
visible history.

Re: [gentoo-dev] stabilization commits and atomicity

[Dirkjan Ochtman]

On Mon, Oct 19, 2015 at 1:21 PM, hasufell <hasufell@gentoo.org> wrote:
> I'd go so far to say allow people to do commits like:
> """
> amd64 stabilizations
>
> <optional list of bugs>
> """
> possibly pre-pending the rough domain like "kde", if any. I think kde
> herd already does that, no?

Sounds sane to me.

Cheers,

Dirkjan

Re: [gentoo-dev] stabilization commits and atomicity

[Rich Freeman]

On Mon, Oct 19, 2015 at 7:55 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
> On Mon, Oct 19, 2015 at 1:21 PM, hasufell <hasufell@gentoo.org> wrote:
>> I'd go so far to say allow people to do commits like:
>> """
>> amd64 stabilizations
>>
>> <optional list of bugs>
>> """
>> possibly pre-pending the rough domain like "kde", if any. I think kde
>> herd already does that, no?
>
> Sounds sane to me.

I think that standardizing how we comment on bulk-stabilization
commits makes more sense than making them less atomic.  Not getting
half a KDE update is actually one of the nice selling features of git.
Plus, in the event of a disaster it also makes rollback easier.

But, by all means we should update the wiki to recommend the standard
way to document these changes.

-- 
Rich

Re: [gentoo-dev] stabilization commits and atomicity

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 19/10/15 08:21 AM, Rich Freeman wrote:
> On Mon, Oct 19, 2015 at 7:55 AM, Dirkjan Ochtman <djc@gentoo.org>
> wrote:
>> On Mon, Oct 19, 2015 at 1:21 PM, hasufell <hasufell@gentoo.org>
>> wrote:
>>> I'd go so far to say allow people to do commits like: """ 
>>> amd64 stabilizations
>>> 
>>> <optional list of bugs> """ possibly pre-pending the rough
>>> domain like "kde", if any. I think kde herd already does
>>> that, no?
>> 
>> Sounds sane to me.
> 
> I think that standardizing how we comment on bulk-stabilization 
> commits makes more sense than making them less atomic.  Not
> getting half a KDE update is actually one of the nice selling
> features of git. Plus, in the event of a disaster it also makes
> rollback easier.
> 
> But, by all means we should update the wiki to recommend the
> standard way to document these changes.
> 


It may be my lack of coffee this morning, but I think you and
hasufell are saying the same thing but using "making commits less
atomic" conversely.

Just so i make sure i'm understanding this right, hasufell's
suggestion is to, instead of rolling a single "atomic" commit for
every package being stabilized under a tracker bug, that the whole
set of packages gets stabilized via one commit.  Thus ensuring one
doesn't get half a kde update, and rollbacks can be done at a single
commit level, etc.

Do I have this right?

(note, since all of these package changes are for a particular
single purpose imo it would still be an atomic commit)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlYlACsACgkQAJxUfCtlWe1yuQD+KaeYsBnQdxL/jCA7AywJwRW4
Iv6LSjNSgMAgYJRCtU8BANz5MrAh8uzqdA03oWetvISXz50nSDa0LuS3XebBZCfi
=UBQF
-----END PGP SIGNATURE-----

Re: [gentoo-dev] stabilization commits and atomicity

[hasufell]

On 10/19/2015 04:37 PM, Ian Stakenvicius wrote:
> 
> 
> 
> It may be my lack of coffee this morning, but I think you and
> hasufell are saying the same thing but using "making commits less
> atomic" conversely.
> 
> Just so i make sure i'm understanding this right, hasufell's
> suggestion is to, instead of rolling a single "atomic" commit for
> every package being stabilized under a tracker bug, that the whole
> set of packages gets stabilized via one commit.  Thus ensuring one
> doesn't get half a kde update, and rollbacks can be done at a single
> commit level, etc.
> 
> Do I have this right?
> 
> (note, since all of these package changes are for a particular
> single purpose imo it would still be an atomic commit)
> 
> 

Well yes. But you could go one step further and argue that we can allow
the same thing when ago's scripts make 300 commits for 300 stabilization
bugs at once (same category or not).

The question is if stabilization needs to be atomic history-wise. It is
nothing you revert or cherry-pick anyway and you could consider it a
global commit too with the subsystem "stable arch".

Re: [gentoo-dev] stabilization commits and atomicity

[Matthew Thode]

On 10/19/2015 10:04 AM, hasufell wrote:
> On 10/19/2015 04:37 PM, Ian Stakenvicius wrote:
>>
>>
>>
>> It may be my lack of coffee this morning, but I think you and
>> hasufell are saying the same thing but using "making commits less
>> atomic" conversely.
>>
>> Just so i make sure i'm understanding this right, hasufell's
>> suggestion is to, instead of rolling a single "atomic" commit for
>> every package being stabilized under a tracker bug, that the whole
>> set of packages gets stabilized via one commit.  Thus ensuring one
>> doesn't get half a kde update, and rollbacks can be done at a single
>> commit level, etc.
>>
>> Do I have this right?
>>
>> (note, since all of these package changes are for a particular
>> single purpose imo it would still be an atomic commit)
>>
>>
> 
> Well yes. But you could go one step further and argue that we can allow
> the same thing when ago's scripts make 300 commits for 300 stabilization
> bugs at once (same category or not).
> 
> The question is if stabilization needs to be atomic history-wise. It is
> nothing you revert or cherry-pick anyway and you could consider it a
> global commit too with the subsystem "stable arch".
> 
while I think that one commit per bug is preferred, having multiple bugs
in one commit is ok as well.  Some of us already do this sometimes when
updating packages (multiple birds with one stone and all that).

-- 
-- Matthew Thode (prometheanfire)

Re: [gentoo-dev] stabilization commits and atomicity

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 19/10/15 11:04 AM, hasufell wrote:
> On 10/19/2015 04:37 PM, Ian Stakenvicius wrote:
>> 
>> 
>> 
>> It may be my lack of coffee this morning, but I think you and 
>> hasufell are saying the same thing but using "making commits 
>> less atomic" conversely.
>> 
>> Just so i make sure i'm understanding this right, hasufell's 
>> suggestion is to, instead of rolling a single "atomic" commit 
>> for every package being stabilized under a tracker bug, that 
>> the whole set of packages gets stabilized via one commit.
>> Thus ensuring one doesn't get half a kde update, and rollbacks
>> can be done at a single commit level, etc.
>> 
>> Do I have this right?
>> 
>> (note, since all of these package changes are for a particular
>>  single purpose imo it would still be an atomic commit)
>> 
>> 
> 
> Well yes. But you could go one step further and argue that we
> can allow the same thing when ago's scripts make 300 commits for
> 300 stabilization bugs at once (same category or not).
> 
> The question is if stabilization needs to be atomic
> history-wise. It is nothing you revert or cherry-pick anyway and
> you could consider it a global commit too with the subsystem
> "stable arch".
> 


Ahh, so what you're referring to here is stabilization of multiple
unrelated packages in a single commit..  ok..  i'm not so
comfortable with that idea.. BUT, nothing stopped us from doing this
with CVS (although the mapping of commit between CVS and GIT aren't
exactly 1:1), so i guess it's fine..?

What about simply keeping things as they are but not strictly
enforcing them when they are used in a manner like this for special
cases, such as ago's stabilizations or other security@ or arch team
keyword-related commits?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlYlC/wACgkQAJxUfCtlWe1XSgD/fa4M2E7k4asOeUGgLEt2um6m
9NovN22eVUeLbSvtnLoBALT4+vhXqYhi3K3ytFv6dcfcKFpiYMbuWuMNu2YrVRj9
=Ef9v
-----END PGP SIGNATURE-----

Re: [gentoo-dev] stabilization commits and atomicity

[Rich Freeman]

On Mon, Oct 19, 2015 at 11:27 AM, Ian Stakenvicius <axs@gentoo.org> wrote:
>
> Ahh, so what you're referring to here is stabilization of multiple
> unrelated packages in a single commit..  ok..  i'm not so
> comfortable with that idea..

Nor am I.  A commit should be a set of related changes.  Stabilizing
all of KDE-n in one commit makes a lot of sense.  Stabilizing 5 random
packages in one commit doesn't make sense.  By all means push them all
at once, but don't commit them all at once.  It isn't like we have to
pay for each commit.

I also don't have a problem with fixing multiple bugs in one commit.
In an ideal world you'd do that on a branch and then do a merge, and I
don't have a problem with that either, but I get that we tend to not
work that way.  If you want every commit to stand on its own and
you're porting to a new EAPI and fixing 3 bugs at the same time, I
don't expect maintainers to rewrite their bugs into the new code to
port it to the new EAPI, then fix each bug in turn.

> BUT, nothing stopped us from doing this
> with CVS (although the mapping of commit between CVS and GIT aren't
> exactly 1:1), so i guess it's fine..?

In cvs all commits were at the file level, even if you happened to
create more than one as a single operation.  If you did one commit
that touched 100 ebuilds, you were actually doing 100 commits, and
there is nothing that really ties those 100 commits together and by
the time it gets to rsync you might only get 50 of them if the timing
is right.

So, this actually is a new problem, or rather benefit.

>
> What about simply keeping things as they are but not strictly
> enforcing them when they are used in a manner like this for special
> cases, such as ago's stabilizations or other security@ or arch team
> keyword-related commits?
>

I don't think we're strictly enforcing anything now but I could be
wrong.  I think we should have guidelines that recommend best
practices and try to stick to them.  If there is a really good reason
to do things differently, that is why we call them "guidelines."

-- 
Rich

Re: [gentoo-dev] stabilization commits and atomicity

[hasufell]

On 10/19/2015 07:08 PM, Rich Freeman wrote:
> On Mon, Oct 19, 2015 at 11:27 AM, Ian Stakenvicius <axs@gentoo.org> wrote:
>>
>> Ahh, so what you're referring to here is stabilization of multiple
>> unrelated packages in a single commit..  ok..  i'm not so
>> comfortable with that idea..
> 
> Nor am I.  A commit should be a set of related changes.  Stabilizing
> all of KDE-n in one commit makes a lot of sense.  Stabilizing 5 random
> packages in one commit doesn't make sense.  By all means push them all
> at once, but don't commit them all at once.  It isn't like we have to
> pay for each commit.
> 

We already know that. But if e.g. ago runs his scripts at 00:00 with
~300 packages stabilized, the history (without git command line) on
github/gitweb will be fun to read (and people DO that).

The argument is that those are related changes to the subsystem "stable
arch" (and affect not random ebuild details, but stable arch only, as in
KEYWORDS). Ofc, people can still create atomic commits if the
stabilization is security related.

Re: [gentoo-dev] stabilization commits and atomicity

[Rich Freeman]

On Mon, Oct 19, 2015 at 1:13 PM, hasufell <hasufell@gentoo.org> wrote:
>
> We already know that. But if e.g. ago runs his scripts at 00:00 with
> ~300 packages stabilized, the history (without git command line) on
> github/gitweb will be fun to read (and people DO that).
>

It doesn't seem like it would have been any better in the cvs days,
but I guess that isn't a reason to reject this on its own.

If this was about changing the copyright headers in all the ebuilds in
the tree I'd say that this is a million related trivial changes that
can be merged.  Nobody needs to see that in the history broken out.

However, stabilizing a single package really is an impactful change.
The fact that you're doing 100 of them at one time doesn't really
diminish the impact of each one.  Any of them could break a system or
need to be reverted.

If they're being done at once because they're all part of some library
stabilization then I'd combine them into a commit, because they are
actually related.

Maybe what is needed is better tools for tagging/filtering history?

-- 
Rich

Re: [gentoo-dev] stabilization commits and atomicity

[hasufell]

On 10/19/2015 07:37 PM, Rich Freeman wrote:
> 
> However, stabilizing a single package really is an impactful change.
> The fact that you're doing 100 of them at one time doesn't really
> diminish the impact of each one.  Any of them could break a system or
> need to be reverted.
> 

Since when do we allow reverting stabilization? The package needs to be
fixed and possibly revbumped instead.

Re: [gentoo-dev] stabilization commits and atomicity

[Rich Freeman]

On Mon, Oct 19, 2015 at 1:40 PM, hasufell <hasufell@gentoo.org> wrote:
> On 10/19/2015 07:37 PM, Rich Freeman wrote:
>>
>> However, stabilizing a single package really is an impactful change.
>> The fact that you're doing 100 of them at one time doesn't really
>> diminish the impact of each one.  Any of them could break a system or
>> need to be reverted.
>>
>
> Since when do we allow reverting stabilization? The package needs to be
> fixed and possibly revbumped instead.
>

It would really depend on the nature of the break.  If it is a serious
upstream problem and no fix is available, then reverting might be the
only practical solution.  It is of course not a preferred solution.

-- 
Rich

Re: [gentoo-dev] stabilization commits and atomicity

[hasufell]

On 10/19/2015 07:52 PM, Rich Freeman wrote:
> On Mon, Oct 19, 2015 at 1:40 PM, hasufell <hasufell@gentoo.org> wrote:
>> On 10/19/2015 07:37 PM, Rich Freeman wrote:
>>>
>>> However, stabilizing a single package really is an impactful change.
>>> The fact that you're doing 100 of them at one time doesn't really
>>> diminish the impact of each one.  Any of them could break a system or
>>> need to be reverted.
>>>
>>
>> Since when do we allow reverting stabilization? The package needs to be
>> fixed and possibly revbumped instead.
>>
> 
> It would really depend on the nature of the break.  If it is a serious
> upstream problem and no fix is available, then reverting might be the
> only practical solution.  It is of course not a preferred solution.
> 

I don't think we depend on 'git revert' in that case. KEYWORDS are
trivial changes (in terms of file diffs).

Re: [gentoo-dev] stabilization commits and atomicity

[Rich Freeman]

On Mon, Oct 19, 2015 at 1:55 PM, hasufell <hasufell@gentoo.org> wrote:
> On 10/19/2015 07:52 PM, Rich Freeman wrote:
>> On Mon, Oct 19, 2015 at 1:40 PM, hasufell <hasufell@gentoo.org> wrote:
>>> On 10/19/2015 07:37 PM, Rich Freeman wrote:
>>>>
>>>> However, stabilizing a single package really is an impactful change.
>>>> The fact that you're doing 100 of them at one time doesn't really
>>>> diminish the impact of each one.  Any of them could break a system or
>>>> need to be reverted.
>>>>
>>>
>>> Since when do we allow reverting stabilization? The package needs to be
>>> fixed and possibly revbumped instead.
>>>
>>
>> It would really depend on the nature of the break.  If it is a serious
>> upstream problem and no fix is available, then reverting might be the
>> only practical solution.  It is of course not a preferred solution.
>>
>
> I don't think we depend on 'git revert' in that case. KEYWORDS are
> trivial changes (in terms of file diffs).
>

True, as long as they're truly standalone.

Maybe the compromise is:
1.  Groups of related stabilizations should be grouped.
2.  Groups of only unrelated stabilizations can also be grouped.
3.  You must not try to mix #1 and #2 in the same commit.

As you say individual packages are easy to revert anyway.  However, we
wouldn't want 100 of those to be mixed in with 50 packages that all
needed to be coordinated, because those 50 aren't easy to revert.

-- 
Rich

[gentoo-dev] Re: stabilization commits and atomicity

[Duncan]

Rich Freeman posted on Mon, 19 Oct 2015 13:52:58 -0400 as excerpted:

> On Mon, Oct 19, 2015 at 1:40 PM, hasufell <hasufell@gentoo.org> wrote:
>> On 10/19/2015 07:37 PM, Rich Freeman wrote:
>>>
>>> However, stabilizing a single package really is an impactful change.
>>> The fact that you're doing 100 of them at one time doesn't really
>>> diminish the impact of each one.  Any of them could break a system or
>>> need to be reverted.
>>>
>>>
>> Since when do we allow reverting stabilization? The package needs to be
>> fixed and possibly revbumped instead.
>>
>>
> It would really depend on the nature of the break.  If it is a serious
> upstream problem and no fix is available, then reverting might be the
> only practical solution.  It is of course not a preferred solution.

Didn't a semi-minor arch (arm, AFAIK) recently revert a major lib 
stabilizing, as it was clearly broken on at least some arm variants, the 
maintainer that did it either didn't consult with the arm-arch folks or 
signals got crossed and he stabilized without approval, and it was caught 
within an hour or less, such that the quickest and most effective way to 
fix the breakage was to do an immediate revert, before even figuring out 
a more long term fix?

The commit log said something like, "Not trying to be rude, but this is 
the quickest way to limit the damage", and within just a few days (two?), 
the package and several deps were stabilized by the arch team in 
question, properly this time, with fixes as appropriate to keep whole sub-
archs from breaking as they were doing with the previous stabilization 
attempt.

[So yes, this demonstrates the point someone made above about people 
actually reading these things, too. =:^)  And I too have been frustrated 
trying to do so, but IMO this is fixing the bit that's /not/ broken, not 
what is.  More about that in a response I'll be posting elsewhere on-
thread.]

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: stabilization commits and atomicity

[Duncan]

hasufell posted on Mon, 19 Oct 2015 19:13:50 +0200 as excerpted:

> On 10/19/2015 07:08 PM, Rich Freeman wrote:
>> On Mon, Oct 19, 2015 at 11:27 AM, Ian Stakenvicius <axs@gentoo.org>
>> wrote:
>>>
>>> Ahh, so what you're referring to here is stabilization of multiple
>>> unrelated packages in a single commit..  ok..  i'm not so comfortable
>>> with that idea..
>> 
>> Nor am I.  A commit should be a set of related changes.  Stabilizing
>> all of KDE-n in one commit makes a lot of sense.  Stabilizing 5 random
>> packages in one commit doesn't make sense.  By all means push them all
>> at once, but don't commit them all at once.  It isn't like we have to
>> pay for each commit.
>> 
> We already know that. But if e.g. ago runs his scripts at 00:00 with
> ~300 packages stabilized, the history (without git command line) on
> github/gitweb will be fun to read (and people DO that).

I'm one of those people (see my reply a few posts down-thread], tho I 
don't read /everything/, but would definitely read rather more if this 
were handled correctly.

But IMO, the proposal here is trying to fix the bit that's /not/ broken, 
not what is.

On the kernel, I can very easily do a ...

git log --color --stat --graph ORIG_HEAD..

... get the nice color graphical listing piped to most, hit end to read 
the whole range of commit logs into buffer, and then starting from the 
end, pageup one page at a time to get the merge chronology correct, and 
read all of Linus's very nicely summarized merge commits, telling me what 
trees were merged and listing the one-line title/summary for each one.

If I want to drill down, as I generally do for specific subsystems
(fs/btrfs, drm/radeon, various subsystems like power I've had bugs with 
in the past), it's a multi-level hierarchy and thus easy enough to go 
from say the drm merge to the radeon sub-merge and check individual 
commits at the comment and affected file levels (--stat).

If I want to drill down further, the commit ID is right there and I can 
git show <ID> to read the specific code.

Meanwhile, if the top-level merge commit says it's for example arm, I'll 
immediately move on to the next top-level merge commit, effectively 
skipping all those possibly hundreds of "boring and not interesting to 
me" individual arm commits, by very simply skipping to the next very 
nicely commented merge commit, with its one-liner list of individual 
commits.

The merge hierarchy, informative merge commit comments, and color-coded
--graph makes finding the merges so easy! That's the way git was 
/designed/ to work. =:^)


Unfortunately, gentoo is by policy trying to flatten all that multi-level-
hierarchy-for-a-reason into a single flat list of CVS/SVN-like strictly 
ordered parent-child relationship commits, destroying the whole rich 
hierarchy and all the information that git includes with it, using 
horrible rebase-pushes as an incredibly poor substitute for the very 
natural git-native merge hierarchy along with all its richness.


Now we're complaining about how flat/boring/uninformative all those 
atomic commits are, but it's not the atomic commits that are the problem, 
it's the fact that we're deliberately stripping out the merge commits and 
all the richness that comes with them, including the natural hierarchy 
and the ability to make those merge-commits as informative as they 
normally are in the kernel, with a short mention of the highlights and a 
quick list of the included one-liners.


If gentoo were doing git the way it was designed to work instead of 
trying to force it into the one-dimensional CVS mold, ago's 300-commit 
bore for anyone not interested in that subsystem, on gentoo commit after 
boring one-dimensional commit that's of less interest to me than the 
price of tea in China, would be in a single merge tree with the arch team 
in the one-liner, so I could immediately skip on to the next top-level 
merge.  Pageup to that merge, spotted quickly by the asterisk in the left 
column, the two lines merging to it, and the change in color in the left-
hand-line, see it's of no interested, and on to the next one, instead of 
having to go thru 300 boring one dimensional commits in case something 
I'm interested in squeezed in there somewhere.

So as I said, the problem isn't the atomic commits, it's gentoo's 
strongly recommended lack of merge hierarchy.  Now we're complaining 
about that and trying to kill the atomic commits, but they're not the 
problem, trying to use that new git nailgun as if it's the manual 
screwdriver of yesteryear is the problem.  If we don't fix that, we'll 
simply be trying to replace one small bit of the information we so 
mercilessly stripped out with those forced-to-one-dimension rebases, 
instead of deciding that stripping all that information out in the first 
place wasn't such a good idea after all and that the real solution is to 
simply stop trying to use that power nailgun as a manual screwdriver!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman