[gentoo-dev] rfc: [QA] Ban policy introduction

[gentoo-dev] rfc: [QA] Ban policy introduction

[Mikle Kolyada]

[-- Attachment #1.1: Type: text/plain, Size: 863 bytes --]

Hello,

The Gentoo QA team would like to introduce the following policy that
would be applied to individuals breaking the state and quality of the
main gentoo.git tree

( as we do not have this strictly documented yet):

<policy>

If recommended Gentoo workflow policies are not followed by an
individual developer
(e.g make major changes to the widely used eclasses without prior
discussion on the mailing list or
commit changes that lead to multiple CI checks failure), the standard QA
procedure is:

1.) Two warnings granted by QA team, after two independent breakages
2.) Revoking the commit access for 14 days

These violations will be evaluated individually by all QA team members.
Warnings can be revoked, if during 6 months period a developer makes at
least 20 non trivial changes not producing more breakages.

</policy>





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] rfc: [QA] Ban policy introduction

[Sergei Trofimovich]

On Sun, 29 Jul 2018 00:40:18 +0300
Mikle Kolyada <zlogene@gentoo.org> wrote:

> Hello,
> 
> The Gentoo QA team would like to introduce the following policy that
> would be applied to individuals breaking the state and quality of the
> main gentoo.git tree
> 
> ( as we do not have this strictly documented yet):
> 
> <policy>
> 
> If recommended

It's not called "recommended" but "enforced".

> Gentoo workflow policies are not followed by an
> individual developer
> (e.g make major changes to the widely used eclasses without prior
> discussion on the mailing list or
> commit changes that lead to multiple CI checks failure)

Here should go exhaustive list of links to the policies to be enforced.

> the standard QA
> procedure is:
> 
> 1.) Two warnings granted by QA team, after two independent breakages
> 2.) Revoking the commit access for 14 days
> 
> These violations will be evaluated individually by all QA team members.
> Warnings can be revoked, if during 6 months period a developer makes at
> least 20 non trivial changes not producing more breakages.
> 
> </policy>

-- 

  Sergei

Re: [gentoo-dev] rfc: [QA] Ban policy introduction

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 1802 bytes --]

Completely agreeing with Sergei, with some additional suggestions:

On 28-07-2018 23:14:12 +0100, Sergei Trofimovich wrote:
> On Sun, 29 Jul 2018 00:40:18 +0300
> Mikle Kolyada <zlogene@gentoo.org> wrote:
> 
> > Hello,
> > 
> > The Gentoo QA team would like to introduce the following policy that
> > would be applied to individuals breaking the state and quality of the
> > main gentoo.git tree
> > 
> > ( as we do not have this strictly documented yet):
> > 
> > <policy>
> > 
> > If recommended
> 
> It's not called "recommended" but "enforced".

I agree.  If you put penalties on these, they become hard rules.  I
think that change should be discussed by the council perhaps?

> > Gentoo workflow policies are not followed by an
> > individual developer
> > (e.g make major changes to the widely used eclasses without prior
> > discussion on the mailing list or
> > commit changes that lead to multiple CI checks failure)
> 
> Here should go exhaustive list of links to the policies to be enforced.

At least.  And they should be clear and concise.  No "common sense" or
anything involved for exceptions and the like.  In addition, new checks
should be introduced to the community and possibly approved by council
as to whether being enforced or not.

Fabian

> 
> > the standard QA
> > procedure is:
> > 
> > 1.) Two warnings granted by QA team, after two independent breakages
> > 2.) Revoking the commit access for 14 days
> > 
> > These violations will be evaluated individually by all QA team members.
> > Warnings can be revoked, if during 6 months period a developer makes at
> > least 20 non trivial changes not producing more breakages.
> > 
> > </policy>
> 
> -- 
> 
>   Sergei
> 

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] rfc: [QA] Ban policy introduction

[Alice Ferrazzi]

[-- Attachment #1: Type: text/plain, Size: 2205 bytes --]

Sent from my phone. Please excuse my brevity.

On Sun, 29 Jul 2018, 16:39 Fabian Groffen, <grobian@gentoo.org> wrote:

> Completely agreeing with Sergei, with some additional suggestions:
>
> On 28-07-2018 23:14:12 +0100, Sergei Trofimovich wrote:
> > On Sun, 29 Jul 2018 00:40:18 +0300
> > Mikle Kolyada <zlogene@gentoo.org> wrote:
> >
> > > Hello,
> > >
> > > The Gentoo QA team would like to introduce the following policy that
> > > would be applied to individuals breaking the state and quality of the
> > > main gentoo.git tree
> > >
> > > ( as we do not have this strictly documented yet):
> > >
> > > <policy>
> > >
> > > If recommended
> >
> > It's not called "recommended" but "enforced".
>
> I agree.  If you put penalties on these, they become hard rules.  I
> think that change should be discussed by the council perhaps?
>
> > > Gentoo workflow policies are not followed by an
> > > individual developer
> > > (e.g make major changes to the widely used eclasses without prior
> > > discussion on the mailing list or
> > > commit changes that lead to multiple CI checks failure)
> >
> > Here should go exhaustive list of links to the policies to be enforced.
>
> At least.  And they should be clear and concise.  No "common sense" or
> anything involved for exceptions and the like.  In addition, new checks
> should be introduced to the community and possibly approved by council
> as to whether being enforced or not.
>
> Fabian
>
> >
> > > the standard QA
> > > procedure is:
> > >
> > > 1.) Two warnings granted by QA team, after two independent breakages
> > > 2.) Revoking the commit access for 14 days
> > >
> > > These violations will be evaluated individually by all QA team members.
> > > Warnings can be revoked, if during 6 months period a developer makes at
> > > least 20 non trivial changes not producing more breakages.
> > >
> > > </policy>
> >
> > --
> >
>
if you want to enforce rules, would be productive to also have extensive
documentation on how to avoid to make such problems.
Better would be to invest more time in something like the breckage checker
script, similar at what mgorny is doing, than adding more ways to block
developers contributions.

thanks,
Alice

>

[-- Attachment #2: Type: text/html, Size: 3328 bytes --]

Re: [gentoo-dev] rfc: [QA] Ban policy introduction

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 2427 bytes --]

On Sun, 29 Jul 2018 16:47:47 +0900 Alice Ferrazzi wrote:
> Sent from my phone. Please excuse my brevity.
> 
> On Sun, 29 Jul 2018, 16:39 Fabian Groffen, <grobian@gentoo.org> wrote:
> 
> > Completely agreeing with Sergei, with some additional suggestions:
> >
> > On 28-07-2018 23:14:12 +0100, Sergei Trofimovich wrote:
> > > On Sun, 29 Jul 2018 00:40:18 +0300
> > > Mikle Kolyada <zlogene@gentoo.org> wrote:
> > >
> > > > Hello,
> > > >
> > > > The Gentoo QA team would like to introduce the following policy that
> > > > would be applied to individuals breaking the state and quality of the
> > > > main gentoo.git tree
> > > >
> > > > ( as we do not have this strictly documented yet):
> > > >
> > > > <policy>
> > > >
> > > > If recommended
> > >
> > > It's not called "recommended" but "enforced".
> >
> > I agree.  If you put penalties on these, they become hard rules.  I
> > think that change should be discussed by the council perhaps?

+1. Also please provide some tool for developers to check for
compliance to these rules, e.g. repoman full must perform all these
checks.

If developers have no way to verify correctness of the code, they
can't be held responsible for accidental violation of the rules.

> > > > the standard QA
> > > > procedure is:
> > > >
> > > > 1.) Two warnings granted by QA team, after two independent breakages
> > > > 2.) Revoking the commit access for 14 days
> > > >
> > > > These violations will be evaluated individually by all QA team members.
> > > > Warnings can be revoked, if during 6 months period a developer makes at
> > > > least 20 non trivial changes not producing more breakages.

Why 6 months period? Why time frame at all? 20 good commits sounds
OK. If you want time frame, then you should set autoexpire of
warning as well.

What is the definition of non-trivial change? There will be commits
which may be seen as trivial by one person (e.g. because it is
one-liner) and as non-trivial by another (e.g. because such commit
fixes serious bug).

> if you want to enforce rules, would be productive to also have extensive
> documentation on how to avoid to make such problems.
> Better would be to invest more time in something like the breckage checker
> script, similar at what mgorny is doing, than adding more ways to block
> developers contributions.
> 
> thanks,
> Alice

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] rfc: [QA] Ban policy introduction

[Gordon Pettey]

On Sun, Jul 29, 2018 at 4:17 AM, Andrew Savchenko <bircoph@gentoo.org> wrote:
> On Sun, 29 Jul 2018 16:47:47 +0900 Alice Ferrazzi wrote:
>> Sent from my phone. Please excuse my brevity.
>>
>> On Sun, 29 Jul 2018, 16:39 Fabian Groffen, <grobian@gentoo.org> wrote:
>>
>> > Completely agreeing with Sergei, with some additional suggestions:
>> >
>> > On 28-07-2018 23:14:12 +0100, Sergei Trofimovich wrote:
>> > > On Sun, 29 Jul 2018 00:40:18 +0300
>> > > Mikle Kolyada <zlogene@gentoo.org> wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > The Gentoo QA team would like to introduce the following policy that
>> > > > would be applied to individuals breaking the state and quality of the
>> > > > main gentoo.git tree
>> > > >
>> > > > ( as we do not have this strictly documented yet):
>> > > >
>> > > > <policy>
>> > > >
>> > > > If recommended
>> > >
>> > > It's not called "recommended" but "enforced".
>> >
>> > I agree.  If you put penalties on these, they become hard rules.  I
>> > think that change should be discussed by the council perhaps?
>
> +1. Also please provide some tool for developers to check for
> compliance to these rules, e.g. repoman full must perform all these
> checks.
>
> If developers have no way to verify correctness of the code, they
> can't be held responsible for accidental violation of the rules.
>
>> > > > the standard QA
>> > > > procedure is:
>> > > >
>> > > > 1.) Two warnings granted by QA team, after two independent breakages
>> > > > 2.) Revoking the commit access for 14 days
>> > > >
>> > > > These violations will be evaluated individually by all QA team members.
>> > > > Warnings can be revoked, if during 6 months period a developer makes at
>> > > > least 20 non trivial changes not producing more breakages.
>
> Why 6 months period? Why time frame at all? 20 good commits sounds
> OK. If you want time frame, then you should set autoexpire of
> warning as well.
> Best regards,
> Andrew Savchenko

That reads to me as "warnings become permanent after six months if not
remediated with 20 non-trivial good commits", not that they expire.

[gentoo-dev] Re: rfc: [QA] Ban policy introduction

[Mikle Kolyada]

[-- Attachment #1.1: Type: text/plain, Size: 1087 bytes --]



On 29.07.2018 00:40, Mikle Kolyada wrote:
> Hello,
>
> The Gentoo QA team would like to introduce the following policy that
> would be applied to individuals breaking the state and quality of the
> main gentoo.git tree
>
> ( as we do not have this strictly documented yet):
>
> <policy>
>
> If recommended Gentoo workflow policies are not followed by an
> individual developer
> (e.g make major changes to the widely used eclasses without prior
> discussion on the mailing list or
> commit changes that lead to multiple CI checks failure), the standard QA
> procedure is:
>
> 1.) Two warnings granted by QA team, after two independent breakages
> 2.) Revoking the commit access for 14 days
>
> These violations will be evaluated individually by all QA team members.
> Warnings can be revoked, if during 6 months period a developer makes at
> least 20 non trivial changes not producing more breakages.
>
> </policy>
>
>
>
>
Meh, I sent this one a bit prematurely, sorry for the spam, we will work
on it a bit more before it reaches an official point


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] rfc: [QA] Ban policy introduction

[Robin H. Johnson]

On Sun, Jul 29, 2018 at 12:40:18AM +0300, Mikle Kolyada wrote:
> (e.g make major changes to the widely used eclasses without prior
> discussion on the mailing list or
What's the metric to say if a given eclass is widely used?

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Re: [gentoo-dev] rfc: [QA] Ban policy introduction

[Guilherme Amadio]

Hello,

I like the idea of enforcing quality rules in the tree, but I have a few
reservations regarding the current plan.

On Sun, Jul 29, 2018 at 04:47:47PM +0900, Alice Ferrazzi wrote:
> Sent from my phone. Please excuse my brevity.
> 
> On Sun, 29 Jul 2018, 16:39 Fabian Groffen, <grobian@gentoo.org> wrote:
> 
> > Completely agreeing with Sergei, with some additional suggestions:
> >
> > On 28-07-2018 23:14:12 +0100, Sergei Trofimovich wrote:
> > > On Sun, 29 Jul 2018 00:40:18 +0300
> > > Mikle Kolyada <zlogene@gentoo.org> wrote:
> > >
> > > > Hello,
> > > >
> > > > The Gentoo QA team would like to introduce the following policy that
> > > > would be applied to individuals breaking the state and quality of the
> > > > main gentoo.git tree

If you introduce penalties for breaking prefix as well, I'm afraid many
people will be unnecessarily penalized. I think that such penalties are
counter productive in most cases. If someone is really being careless it
might make sense to get some warning and lose commit access temporarily.
If someone made a simple mistake that can be easily fixed, like version
bumping a package that starts to fail in some corner case, then
punishment doesn't make much sense. 

I'd rather prefer to see improvements to our automated checks as the
mean to improve the quality of the tree, and penalties for people that
do not use repoman. Rather than a temporary ban, it would also make more
sense to just require someone to go through the ebuild quiz again as
penalty for breaking the tree many times.

> > > > ( as we do not have this strictly documented yet):
> > > >
> > > > <policy>
> > > >
> > > > If recommended
> > >
> > > It's not called "recommended" but "enforced".
> >
> > I agree.  If you put penalties on these, they become hard rules.  I
> > think that change should be discussed by the council perhaps?
> >
> > > > Gentoo workflow policies are not followed by an
> > > > individual developer
> > > > (e.g make major changes to the widely used eclasses without prior
> > > > discussion on the mailing list or
> > > > commit changes that lead to multiple CI checks failure)
> > >
> > > Here should go exhaustive list of links to the policies to be enforced.
> >
> > At least.  And they should be clear and concise.  No "common sense" or
> > anything involved for exceptions and the like.  In addition, new checks
> > should be introduced to the community and possibly approved by council
> > as to whether being enforced or not.
> >
> > Fabian
> >
> > >
> > > > the standard QA
> > > > procedure is:
> > > >
> > > > 1.) Two warnings granted by QA team, after two independent breakages
> > > > 2.) Revoking the commit access for 14 days
> > > >
> > > > These violations will be evaluated individually by all QA team members.
> > > > Warnings can be revoked, if during 6 months period a developer makes at
> > > > least 20 non trivial changes not producing more breakages.
> > > >
> > > > </policy>
> > >
> > > --
> > >
> >
> if you want to enforce rules, would be productive to also have extensive
> documentation on how to avoid to make such problems.
> Better would be to invest more time in something like the breckage checker
> script, similar at what mgorny is doing, than adding more ways to block
> developers contributions.

I agree with Alice here. We already have the devmanual and repoman, as
well as the CI system, which are all very good. Extending them is the
best way to keep Gentoo in good shape. Also, if we have enough
resources, we can try to reject broken pushes, rather than punishing
devs after broken changes reach the main tree.

Cheers,
-Guilherme

Re: [gentoo-dev] rfc: [QA] Ban policy introduction

[Dirkjan Ochtman]

[-- Attachment #1: Type: text/plain, Size: 1752 bytes --]

On Mon, Jul 30, 2018 at 8:52 AM Guilherme Amadio <amadio@gentoo.org> wrote:

> If you introduce penalties for breaking prefix as well, I'm afraid many
> people will be unnecessarily penalized. I think that such penalties are
> counter productive in most cases. If someone is really being careless it
> might make sense to get some warning and lose commit access temporarily.
> If someone made a simple mistake that can be easily fixed, like version
> bumping a package that starts to fail in some corner case, then
> punishment doesn't make much sense.
>

The proposed policy already mentions that people will only be punished
after two warnings. This seems enough for me -- if people keep breaking
stuff despite warnings, a little penalty is probably a good thing.

The proposed policy already goes out of its way to require two warnings for
"independent" breakage, but it's not entirely clear what independent means
here. If you commit three breakages that are technically unrelated on the
same day, then you probably shouldn't be banned immediately. So I would
suggest also making clear that the warnings should be sent at least a few
days apart and that an initial penalty cannot happen until a few days apart
the second warning.

That said, I agree with those who are saying that breaking things should be
obvious, things like ignoring repoman and/or other CI messaging. If the
breakage is non-obvious and hard to spot locally, then we should instead
invest in tooling to make it more obvious. By "ignoring" here I do mean
that there needs to be a reasonable timeout -- sometimes if I commit a
change and get a CI alert after a few hours, it might be tricky due to
work/family/whatever concerns to fix it within, say, 24 hours.

Regards,

Dirkjan

[-- Attachment #2: Type: text/html, Size: 2151 bytes --]

Re: [gentoo-dev] rfc: [QA] Ban policy introduction SLIPUP

[Amy Liffey]

[-- Attachment #1.1: Type: text/plain, Size: 2416 bytes --]

Hello folks,

I apologize to everyone for sending this proposal before it was
finished. It was not voted on by the QA team hence it was not an
official proposal by the QA team. There was probably some
misunderstanding in communication.

After we finish the official draft and it is accepted by QA team
members, we will be very happy to accept comments on the mailing list in
the future.

Thank you for understanding

On behalf of QA team,

Amy Liffey

On 07/30/2018 10:03 AM, Dirkjan Ochtman wrote:
> On Mon, Jul 30, 2018 at 8:52 AM Guilherme Amadio <amadio@gentoo.org
> <mailto:amadio@gentoo.org>> wrote:
> 
>     If you introduce penalties for breaking prefix as well, I'm afraid many
>     people will be unnecessarily penalized. I think that such penalties are
>     counter productive in most cases. If someone is really being careless it
>     might make sense to get some warning and lose commit access temporarily.
>     If someone made a simple mistake that can be easily fixed, like version
>     bumping a package that starts to fail in some corner case, then
>     punishment doesn't make much sense.
> 
> 
> The proposed policy already mentions that people will only be punished
> after two warnings. This seems enough for me -- if people keep breaking
> stuff despite warnings, a little penalty is probably a good thing.
> 
> The proposed policy already goes out of its way to require two warnings
> for "independent" breakage, but it's not entirely clear what independent
> means here. If you commit three breakages that are technically unrelated
> on the same day, then you probably shouldn't be banned immediately. So I
> would suggest also making clear that the warnings should be sent at
> least a few days apart and that an initial penalty cannot happen until a
> few days apart the second warning.
> 
> That said, I agree with those who are saying that breaking things should
> be obvious, things like ignoring repoman and/or other CI messaging. If
> the breakage is non-obvious and hard to spot locally, then we should
> instead invest in tooling to make it more obvious. By "ignoring" here I
> do mean that there needs to be a reasonable timeout -- sometimes if I
> commit a change and get a CI alert after a few hours, it might be tricky
> due to work/family/whatever concerns to fix it within, say, 24 hours.
> 
> Regards,
> 
> Dirkjan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] rfc: [QA] Ban policy introduction SLIPUP

[Alexis Ballier]

On Tue, 31 Jul 2018 14:36:37 +0100
Amy Liffey <amynka@gentoo.org> wrote:

> Hello folks,
> 
> I apologize to everyone for sending this proposal before it was
> finished. It was not voted on by the QA team hence it was not an
> official proposal by the QA team. There was probably some
> misunderstanding in communication.
> 
> After we finish the official draft and it is accepted by QA team
> members, we will be very happy to accept comments on the mailing list
> in the future.

During that drafting process, please clarify how this changes and
differs from the current GLEP48 wording and if the glep needs to be
updated:

If a particular developer persistently causes breakage, the QA team may
request that Comrel re-evaluates that developer's commit rights.
Evidence of past breakages will be presented with this request to
Comrel.





Alexis.