[gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]

Hello,

As you have seen multiple times, I'm running a minimalistic CI service
for Gentoo that checks the repository for major issues using pkgcheck.
So far it's automation is limited to sending a mail to dedicated
gentoo-automated-testing@lists.gentoo.org mailing list on breakage
changes. From there, I compare the results to recent git log and mail
the developers at fault, pointing out the bad commit.

A few developers have already subscribed to the mailing list to check
if they haven't caused any new breakages and fix them quickly. For
others, it's pretty much just me caring to check, which also means that
when I'm not around things are left broken.

Automating the blaming process has been suggested multiple times
already but I so far considered it not worth the effort. Mostly because
many of the issues are indirect, and trying to automatically figure
them out from combination of the pkgcheck report and recent commits
would be hard, and could cause false positives. For example, some of
the depgraph breakages happen because of package.mask changes --
figuring that out automatically wouldn't be easy, and the script could
blame an irrelevant commit in the package.

However, it was suggested recently that I could make it mail
the maintainers of the affected packages. Even though most often it's
not them who are at fault, it was suggested that they'd prefer to know
that their packages are broken.

So what do you think? Would it be fine to mail the package maintainers
whenever their packages break? Would it be a problem if I just CC-ed
all the maintainers on the gentoo-automated-testing mails? Please note
that the breakages are catched per-package, and the script wouldn't be
able to respect restrict="" or hand-written maintainer descriptions ;-).

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Dirkjan Ochtman]

On Sun, Dec 6, 2015 at 3:36 PM, Michał Górny <mgorny@gentoo.org> wrote:
> So what do you think? Would it be fine to mail the package maintainers
> whenever their packages break? Would it be a problem if I just CC-ed
> all the maintainers on the gentoo-automated-testing mails? Please note
> that the breakages are catched per-package, and the script wouldn't be
> able to respect restrict="" or hand-written maintainer descriptions ;-).

Sounds great to me.

Cheers,

Dirkjan

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Michael Orlitzky]

On 12/06/2015 09:36 AM, Michał Górny wrote:
> 
> So what do you think? Would it be fine to mail the package maintainers
> whenever their packages break? Would it be a problem if I just CC-ed
> all the maintainers on the gentoo-automated-testing mails? 

Of course. Add the commit author, too: I want to know if I break someone
else's package.

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

On Sun, 6 Dec 2015 10:31:37 -0500
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 12/06/2015 09:36 AM, Michał Górny wrote:
> > 
> > So what do you think? Would it be fine to mail the package maintainers
> > whenever their packages break? Would it be a problem if I just CC-ed
> > all the maintainers on the gentoo-automated-testing mails?   
> 
> Of course. Add the commit author, too: I want to know if I break someone
> else's package.

So far, can't do that since we don't know which commit exactly broke. I
don't want to do any heuristics that could blame the wrong person.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Michael Orlitzky]

On 12/06/2015 11:00 AM, Michał Górny wrote:
>>
>> Of course. Add the commit author, too: I want to know if I break someone
>> else's package.
> 
> So far, can't do that since we don't know which commit exactly broke. I
> don't want to do any heuristics that could blame the wrong person.
> 

Is the testing performed per-push rather than per-commit? Either way, I
would like to get a notification that something broke, even if it wasn't
my commit at fault. Just change the word "blame" to "alert" so no one
feels slandered.

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

On Sun, 6 Dec 2015 11:09:48 -0500
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 12/06/2015 11:00 AM, Michał Górny wrote:
> >>
> >> Of course. Add the commit author, too: I want to know if I break someone
> >> else's package.  
> > 
> > So far, can't do that since we don't know which commit exactly broke. I
> > don't want to do any heuristics that could blame the wrong person.
> >   
> 
> Is the testing performed per-push rather than per-commit? Either way, I
> would like to get a notification that something broke, even if it wasn't
> my commit at fault. Just change the word "blame" to "alert" so no one
> feels slandered.

It is done every 20 minutes, and takes around 7-10 minutes.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Michael Orlitzky]

On 12/06/2015 11:49 AM, Michał Górny wrote:
>>
>> Is the testing performed per-push rather than per-commit? Either way, I
>> would like to get a notification that something broke, even if it wasn't
>> my commit at fault. Just change the word "blame" to "alert" so no one
>> feels slandered.
> 
> It is done every 20 minutes, and takes around 7-10 minutes.
> 

Looking through our repo history, this should only hit a few unique
authors. Keep it simple, email everyone.

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Andreas K. Huettel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Am Sonntag, 6. Dezember 2015, 17:49:00 schrieb Michał Górny:
> On Sun, 6 Dec 2015 11:09:48 -0500
> 
> Michael Orlitzky <mjo@gentoo.org> wrote:
> > On 12/06/2015 11:00 AM, Michał Górny wrote:
> > >> Of course. Add the commit author, too: I want to know if I break
> > >> someone else's package.
> > > 
> > > So far, can't do that since we don't know which commit exactly broke. I
> > > don't want to do any heuristics that could blame the wrong person.
> > 
> > Is the testing performed per-push rather than per-commit? Either way, I
> > would like to get a notification that something broke, even if it wasn't
> > my commit at fault. Just change the word "blame" to "alert" so no one
> > feels slandered.
> 
> It is done every 20 minutes, and takes around 7-10 minutes.

How about just adding everyone who pushed in that timeframe?

- -- 
Andreas K. Huettel
Gentoo Linux developer (council, perl, libreoffice)
dilfridge@gentoo.org
http://www.akhuettel.de/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJWZHUYAAoJEHRrah2soMK+WwEQAM3kIuB898xKOXhYRkCWxZKP
UeXyqHWfUOuSoFwmNKwfOiHV7bJfAOMgKI6dymqFRjOU32gOMe2hSeR4W5E7ytT3
ykTjDO1zf+YmZ4Pn4xeXUa4f/SwqeD9exNl3O8+9wbMSd1CfQj1RL1YRQPjuDfsY
zZFbk7WBrkHVlIyUpiwE/3/CF4fCzg2FIDQdQv78p8MvEAEUz2JFolyaCAJzroje
TaAweY2oFC9e2aMfg2dtW2m85q/B1ZtYZzNVvCM+5iDRwbVZyNzttvb31GbQwRLO
yz5SJQqTk0TpoFXomq7hK0xb/leRnbbwjOC02sebMxP6Dw3ya33zeXG64gzEkYUB
7FrLo3U7jTWx+vepACJPOfULqtBP0u0YtFOzPdEHJbtv1qzGRXeSMctMvaS3XnBl
NFceqTxt19qiRu3zN3cpjuf/C0FhthAplooQfRD8quy7KLgM8gO8fuX6g706qXUJ
1QS2UXAyVVrHUDpU0fPM73Uts4RivaYyjSyiy1+neGPOVQ6TpkaSPgh2bjlsb7d8
gmM5s8Y8RmXi8LOGEr1w8cuiCTaFVlRgNQB3WAYpLQ/m/kDNakIAFlX4KhVZdBYh
n15BMUGJRfz1r/XYUXd89iwesjye0YGtww2TVWSpLkeZDInua+JhRy/30HaPuJ6O
LHJon3GqAM8fmjcZOa7N
=0QHR
-----END PGP SIGNATURE-----

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Michał Górny]

Dnia 6 grudnia 2015 18:49:12 CET, "Andreas K. Huettel" <dilfridge@gentoo.org> napisał(a):
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA512
>
>Am Sonntag, 6. Dezember 2015, 17:49:00 schrieb Michał Górny:
>> On Sun, 6 Dec 2015 11:09:48 -0500
>> 
>> Michael Orlitzky <mjo@gentoo.org> wrote:
>> > On 12/06/2015 11:00 AM, Michał Górny wrote:
>> > >> Of course. Add the commit author, too: I want to know if I break
>> > >> someone else's package.
>> > > 
>> > > So far, can't do that since we don't know which commit exactly
>broke. I
>> > > don't want to do any heuristics that could blame the wrong
>person.
>> > 
>> > Is the testing performed per-push rather than per-commit? Either
>way, I
>> > would like to get a notification that something broke, even if it
>wasn't
>> > my commit at fault. Just change the word "blame" to "alert" so no
>one
>> > feels slandered.
>> 
>> It is done every 20 minutes, and takes around 7-10 minutes.
>
>How about just adding everyone who pushed in that timeframe?

It just occurred to me that I can run checks just for the one failing package, which should have much decreased run time. I'm going to see if bisecting would work there.

>
>- -- 
>Andreas K. Huettel
>Gentoo Linux developer (council, perl, libreoffice)
>dilfridge@gentoo.org
>http://www.akhuettel.de/
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v2
>
>iQIcBAEBCgAGBQJWZHUYAAoJEHRrah2soMK+WwEQAM3kIuB898xKOXhYRkCWxZKP
>UeXyqHWfUOuSoFwmNKwfOiHV7bJfAOMgKI6dymqFRjOU32gOMe2hSeR4W5E7ytT3
>ykTjDO1zf+YmZ4Pn4xeXUa4f/SwqeD9exNl3O8+9wbMSd1CfQj1RL1YRQPjuDfsY
>zZFbk7WBrkHVlIyUpiwE/3/CF4fCzg2FIDQdQv78p8MvEAEUz2JFolyaCAJzroje
>TaAweY2oFC9e2aMfg2dtW2m85q/B1ZtYZzNVvCM+5iDRwbVZyNzttvb31GbQwRLO
>yz5SJQqTk0TpoFXomq7hK0xb/leRnbbwjOC02sebMxP6Dw3ya33zeXG64gzEkYUB
>7FrLo3U7jTWx+vepACJPOfULqtBP0u0YtFOzPdEHJbtv1qzGRXeSMctMvaS3XnBl
>NFceqTxt19qiRu3zN3cpjuf/C0FhthAplooQfRD8quy7KLgM8gO8fuX6g706qXUJ
>1QS2UXAyVVrHUDpU0fPM73Uts4RivaYyjSyiy1+neGPOVQ6TpkaSPgh2bjlsb7d8
>gmM5s8Y8RmXi8LOGEr1w8cuiCTaFVlRgNQB3WAYpLQ/m/kDNakIAFlX4KhVZdBYh
>n15BMUGJRfz1r/XYUXd89iwesjye0YGtww2TVWSpLkeZDInua+JhRy/30HaPuJ6O
>LHJon3GqAM8fmjcZOa7N
>=0QHR
>-----END PGP SIGNATURE-----

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Rich Freeman]

On Sun, Dec 6, 2015 at 11:09 AM, Michael Orlitzky <mjo@gentoo.org> wrote:
> On 12/06/2015 11:00 AM, Michał Górny wrote:
>>>
>>> Of course. Add the commit author, too: I want to know if I break someone
>>> else's package.
>>
>> So far, can't do that since we don't know which commit exactly broke. I
>> don't want to do any heuristics that could blame the wrong person.
>>
>
> Is the testing performed per-push rather than per-commit? Either way, I
> would like to get a notification that something broke, even if it wasn't
> my commit at fault. Just change the word "blame" to "alert" so no one
> feels slandered.
>

++

This isn't about shaming people.  It is about alerting that the tree
is broken.  I think we can agree that when packages don't build it is
a problem, and it won't fix itself.

How many commits typically go by in-between checks?  Would it be
practical to just alert any commit author in that time range?  Sure,
it would generate a bit of spam, but:

1.  Better to get problems fixed sooner than later.
2.  The overall improved attention to QA will hopefully reduce the
error rate and thus make the number of emails regulate themselves.

One of the first steps towards reducing errors is to increase their visibility.

-- 
Rich

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Ian Stakenvicius]

> On Dec 6, 2015, at 11:52 AM, Rich Freeman <rich0@gentoo.org> wrote:
> 
>> On Sun, Dec 6, 2015 at 11:09 AM, Michael Orlitzky <mjo@gentoo.org> wrote:
>> On 12/06/2015 11:00 AM, Michał Górny wrote:
>>>> 
>>>> Of course. Add the commit author, too: I want to know if I break someone
>>>> else's package.
>>> 
>>> So far, can't do that since we don't know which commit exactly broke. I
>>> don't want to do any heuristics that could blame the wrong person.
>> 
>> Is the testing performed per-push rather than per-commit? Either way, I
>> would like to get a notification that something broke, even if it wasn't
>> my commit at fault. Just change the word "blame" to "alert" so no one
>> feels slandered.
> 
> ++
> 
> This isn't about shaming people.  It is about alerting that the tree
> is broken.  I think we can agree that when packages don't build it is
> a problem, and it won't fix itself.
> 
> How many commits typically go by in-between checks?  Would it be
> practical to just alert any commit author in that time range?  Sure,
> it would generate a bit of spam, but:
> 
> 1.  Better to get problems fixed sooner than later.
> 2.  The overall improved attention to QA will hopefully reduce the
> error rate and thus make the number of emails regulate themselves.
> 
> One of the first steps towards reducing errors is to increase their visibility.
> 

Couldn't we just alert the people listed in the metadata for the packages affected?  Even if it wasn't them that caused the breakage, aren't they ultimately responsible for making sure the package works?  They could ping the actual committer...

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Rich Freeman]

On Sun, Dec 6, 2015 at 12:26 PM, Ian Stakenvicius <axs@gentoo.org> wrote:
>
>> On Dec 6, 2015, at 11:52 AM, Rich Freeman <rich0@gentoo.org> wrote:
>>
>>> On Sun, Dec 6, 2015 at 11:09 AM, Michael Orlitzky <mjo@gentoo.org> wrote:
>>> On 12/06/2015 11:00 AM, Michał Górny wrote:
>>>>>
>>>>> Of course. Add the commit author, too: I want to know if I break someone
>>>>> else's package.
>>>>
>>>> So far, can't do that since we don't know which commit exactly broke. I
>>>> don't want to do any heuristics that could blame the wrong person.
>>>
>>> Is the testing performed per-push rather than per-commit? Either way, I
>>> would like to get a notification that something broke, even if it wasn't
>>> my commit at fault. Just change the word "blame" to "alert" so no one
>>> feels slandered.
>>
>> ++
>>
>> This isn't about shaming people.  It is about alerting that the tree
>> is broken.  I think we can agree that when packages don't build it is
>> a problem, and it won't fix itself.
>>
>> How many commits typically go by in-between checks?  Would it be
>> practical to just alert any commit author in that time range?  Sure,
>> it would generate a bit of spam, but:
>>
>> 1.  Better to get problems fixed sooner than later.
>> 2.  The overall improved attention to QA will hopefully reduce the
>> error rate and thus make the number of emails regulate themselves.
>>
>> One of the first steps towards reducing errors is to increase their visibility.
>>
>
> Couldn't we just alert the people listed in the metadata for the packages affected?  Even if it wasn't them that caused the breakage, aren't they ultimately responsible for making sure the package works?  They could ping the actual committer...
>

It is best to give feedback to those who know what they did, not to a
bunch of random people who have to puzzle it out.  Maybe the toolchain
guys change something and 200 packages break.  Now, we can all sit and
stare at bizarre gcc output for an hour each trying to figure out the
cause of some misleading error message, or we could let the toolchain
guys know and as soon as they see 200 C-based packages break after
committing a toolchain change they're going to know that they're the
cause and likely what is going on.  I'm not sure if the toolchain team
would be better served if they get hit CC'ed on 200 carefully-crafted
bug reports either, over the span of a few days as the maintainers
catch up on their likely-outdated emails.

By all means CC the package maintainers, since they likely do care,
but they may not be in the best place to debug a problem that
originated in a dependency.

If we're talking about sending an email to ~5 committers just do it.
If I committed a change to mythtv and perl breaks, I'll probably
ignore it.  If I committed a change to dar and some backup utility
breaks, I'll probably take a closer look.  If in a month we're sick
and tired of the emails we can always give up, or start beating
anybody over the head who doesn't run repoman.

-- 
Rich

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 3417 bytes --]

On Sun, Dec 6, 2015 at 6:36 AM, Michał Górny <mgorny@gentoo.org> wrote:

> Hello,
>

Hi!


>
> As you have seen multiple times, I'm running a minimalistic CI service
> for Gentoo that checks the repository for major issues using pkgcheck.
> So far it's automation is limited to sending a mail to dedicated
> gentoo-automated-testing@lists.gentoo.org mailing list on breakage
> changes. From there, I compare the results to recent git log and mail
> the developers at fault, pointing out the bad commit.
>
> A few developers have already subscribed to the mailing list to check
> if they haven't caused any new breakages and fix them quickly. For
> others, it's pretty much just me caring to check, which also means that
> when I'm not around things are left broken.
>
>
So this sort of brings up a point of responsibility.



> Automating the blaming process has been suggested multiple times
> already but I so far considered it not worth the effort. Mostly because
> many of the issues are indirect, and trying to automatically figure
> them out from combination of the pkgcheck report and recent commits
> would be hard, and could cause false positives. For example, some of
> the depgraph breakages happen because of package.mask changes --
> figuring that out automatically wouldn't be easy, and the script could
> blame an irrelevant commit in the package.
>
> However, it was suggested recently that I could make it mail
> the maintainers of the affected packages. Even though most often it's
> not them who are at fault, it was suggested that they'd prefer to know
> that their packages are broken.
>

I think there are a few issues:

1) Not everyone cares. I think you can either go for an opt-in approach
(hard..you need to keep state) or offer clear opt-out / filtering
instructions (link in the bottom of the email that points at the opt-out
instructions on wiki.) Either decision will piss people off; I wouldn't
fret it as long as you pick one.

2) Unclear ownership of the problem. One guy makes a commit, 100 packages
break. Who is responsible? Its really murky. This is really the toughest
problem to me.

3) Problems are not stateless (e.g. many are transient as they are fixed
later by developers.) Is the email I got 8 hours ago still relevant? What
we normally see in items like this is a framework to manage "incidents". So
what you might see is an incident App. The CI infrastructure detects a
problem and opens an incident. At incident open, you trigger a notification
(said email). Typically incidents can be claimed (a human takes ownership
and fixes the incident) or perhaps a future run of the automation detects
that the incident is fixed and closes the incident.

The problem of course with 3 is that you are very much re-inventing a bunch
of functionality that is already in bugzilla; which leads to the argument
of 'why not open bugs for breakages' ;)

-A


>
> So what do you think? Would it be fine to mail the package maintainers
> whenever their packages break? Would it be a problem if I just CC-ed
> all the maintainers on the gentoo-automated-testing mails? Please note
> that the breakages are catched per-package, and the script wouldn't be
> able to respect restrict="" or hand-written maintainer descriptions ;-).
>
> --
> Best regards,
> Michał Górny
> <http://dev.gentoo.org/~mgorny/>
>

[-- Attachment #2: Type: text/html, Size: 4595 bytes --]

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Rich Freeman]

On Mon, Dec 7, 2015 at 7:05 PM, Alec Warner <antarus@gentoo.org> wrote:
> I think there are a few issues:
>
> 1) Not everyone cares. I think you can either go for an opt-in approach
> (hard..you need to keep state) or offer clear opt-out / filtering
> instructions (link in the bottom of the email that points at the opt-out
> instructions on wiki.) Either decision will piss people off; I wouldn't fret
> it as long as you pick one.

I'd offer another option - just send them email and let them deal with
it.  That will also tick people off, but again I wouldn't fret it.  By
all means stick a header in the mail or something to make filtering
easy.  It is far more sensible for users to filter emails than to
build a fancy filtering system into every application that might send
mail.

>
> 2) Unclear ownership of the problem. One guy makes a commit, 100 packages
> break. Who is responsible? Its really murky. This is really the toughest
> problem to me.

It isn't murky at all.  Nobody should ever commit something that
simply breaks something else.  Sometimes it is unforseen, and that
might be ok if it is rare, but the committer can still go and revert
their commit and sort things out.

If you want to make a commit that will break 100 packages you have
many valid options available to you.

1.  You could go fix those 100 packages yourself so that they never
break, ideally posting notice somewhere that you're going to do it.
2.  You could create a blocker and bugs against those 100 packages
asking their maintainers to fix it ahead of time, with a deadline.
3.  After doing #2 if some packages still aren't fixed then mask them,
and make your commit.

If you do any of the above you'll get zero emails from the CI system,
as you aren't breaking the tree.  Of course #3 is undesirable since
you're making a smaller unbroken tree, but as long as you give fair
warning it is acceptable.

Breaking the tree is just never correct.  Sometimes it is hard to
catch and we need to bear with things since we're only human.  But, if
the CI system sends out an email, something HAS gone wrong (either
with the tree or the CI system).

>
> 3) Problems are not stateless (e.g. many are transient as they are fixed
> later by developers.) Is the email I got 8 hours ago still relevant? What we
> normally see in items like this is a framework to manage "incidents". So
> what you might see is an incident App. The CI infrastructure detects a
> problem and opens an incident. At incident open, you trigger a notification
> (said email). Typically incidents can be claimed (a human takes ownership
> and fixes the incident) or perhaps a future run of the automation detects
> that the incident is fixed and closes the incident.
>
> The problem of course with 3 is that you are very much re-inventing a bunch
> of functionality that is already in bugzilla; which leads to the argument of
> 'why not open bugs for breakages' ;)

I guess you can do that, but do you propose CCing the bug to everybody
who made a commit in the time range?  An email is definitely more
lightweight.  I am also concerned that those bugs will just tend to
stay open if people avoid ownership of the issue.  Who do you assign
the bug to (it isn't fair to make the package maintainer of the
affected package clean up somebody else's failures)?

I don't think this is a bad idea, but if email alerts are something
that will happen and bugs are something that won't happen, then I'd
prefer the email to nothing.

This is really a fairly lighthanded solution to the problem.

-- 
Rich

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Anthony G. Basile]

On 12/7/15 7:27 PM, Rich Freeman wrote:
> On Mon, Dec 7, 2015 at 7:05 PM, Alec Warner <antarus@gentoo.org> wrote:
>> 2) Unclear ownership of the problem. One guy makes a commit, 100 packages
>> break. Who is responsible? Its really murky. This is really the toughest
>> problem to me.
> It isn't murky at all.  Nobody should ever commit something that
> simply breaks something else.  Sometimes it is unforseen, and that
> might be ok if it is rare, but the committer can still go and revert
> their commit and sort things out.

Does that include stuff that breaks on systems using musl instead of
glibc?  Or uclibc?  or eudev instead of udev.  What about openrc vs
systemd?  Shall I go on?  Of course its murky.  If you disagree, I'll be
more than happy to pull out dozens of emails where people object when
their stuff breaks other people's stuff with the infamous "why should we
support that shit?"

Anyhow, if the emails are easily filter or have an easy out, I see no
problem.  Murky or not.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Rich Freeman]

On Mon, Dec 7, 2015 at 9:19 PM, Anthony G. Basile <blueness@gentoo.org> wrote:
>
> Does that include stuff that breaks on systems using musl instead of
> glibc?  Or uclibc?  or eudev instead of udev.  What about openrc vs
> systemd?  Shall I go on?  Of course its murky.

Sure, but none of that stuff is going to get caught by the CI system
most likely.  The scenario was one commit breaks 100 packages, and I
guess I read into it that this is in a vanilla profile.

I suppose there always is a gray area, but if it is breaking on a
default profile I'd consider that something every committer should be
concerned with, and that is the sort of thing that the tinderbox is
going to catch.

In any case, it is just data.  As you say emails can be ignored if appropriate.

-- 
Rich

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Michał Górny]

Dnia 8 grudnia 2015 01:05:26 CET, Alec Warner <antarus@gentoo.org> napisał(a):
>On Sun, Dec 6, 2015 at 6:36 AM, Michał Górny <mgorny@gentoo.org> wrote:
>
>> Hello,
>>
>
>Hi!
>
>
>>
>> As you have seen multiple times, I'm running a minimalistic CI
>service
>> for Gentoo that checks the repository for major issues using
>pkgcheck.
>> So far it's automation is limited to sending a mail to dedicated
>> gentoo-automated-testing@lists.gentoo.org mailing list on breakage
>> changes. From there, I compare the results to recent git log and mail
>> the developers at fault, pointing out the bad commit.
>>
>> A few developers have already subscribed to the mailing list to check
>> if they haven't caused any new breakages and fix them quickly. For
>> others, it's pretty much just me caring to check, which also means
>that
>> when I'm not around things are left broken.
>>
>>
>So this sort of brings up a point of responsibility.
>
>
>
>> Automating the blaming process has been suggested multiple times
>> already but I so far considered it not worth the effort. Mostly
>because
>> many of the issues are indirect, and trying to automatically figure
>> them out from combination of the pkgcheck report and recent commits
>> would be hard, and could cause false positives. For example, some of
>> the depgraph breakages happen because of package.mask changes --
>> figuring that out automatically wouldn't be easy, and the script
>could
>> blame an irrelevant commit in the package.
>>
>> However, it was suggested recently that I could make it mail
>> the maintainers of the affected packages. Even though most often it's
>> not them who are at fault, it was suggested that they'd prefer to
>know
>> that their packages are broken.
>>
>
>I think there are a few issues:
>
>1) Not everyone cares. I think you can either go for an opt-in approach
>(hard..you need to keep state) or offer clear opt-out / filtering
>instructions (link in the bottom of the email that points at the
>opt-out
>instructions on wiki.) Either decision will piss people off; I wouldn't
>fret it as long as you pick one.

Opt-out: reopen your developer bug and ask to be retired ;-).

>
>2) Unclear ownership of the problem. One guy makes a commit, 100
>packages
>break. Who is responsible? Its really murky. This is really the
>toughest
>problem to me.

So far I kept the commit author the only one responsible. However, this doesn't really work without CC-ing the maintainers of broken packages.

Long story short, developer removes old version of X, breaking a dozen packages. I report this to him, he reverts the commit and... usually everything is left as-is waiting for another developer to do the same mistake. CC-ing other package maintainers could at least be a helpful reminder 'you require old version of X'.

>
>3) Problems are not stateless (e.g. many are transient as they are
>fixed
>later by developers.) Is the email I got 8 hours ago still relevant?
>What
>we normally see in items like this is a framework to manage
>"incidents". So
>what you might see is an incident App. The CI infrastructure detects a
>problem and opens an incident. At incident open, you trigger a
>notification
>(said email). Typically incidents can be claimed (a human takes
>ownership
>and fixes the incident) or perhaps a future run of the automation
>detects
>that the incident is fixed and closes the incident.

That's a good point I missed. Though otoh I already keep previous list of failures and commit id. It wouldn't be hard to add list of mails there.

>
>The problem of course with 3 is that you are very much re-inventing a
>bunch
>of functionality that is already in bugzilla; which leads to the
>argument
>of 'why not open bugs for breakages' ;)

The infra was already against automated bug filling. That's why repository QA failures are reported with manual confirmation. But they're not the highest priority, so the delay caused by that is less problematic than for gentoo-ci.

>
>-A
>
>
>>
>> So what do you think? Would it be fine to mail the package
>maintainers
>> whenever their packages break? Would it be a problem if I just CC-ed
>> all the maintainers on the gentoo-automated-testing mails? Please
>note
>> that the breakages are catched per-package, and the script wouldn't
>be
>> able to respect restrict="" or hand-written maintainer descriptions
>;-).
>>
>> --
>> Best regards,
>> Michał Górny
>> <http://dev.gentoo.org/~mgorny/>
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-dev] RFC: automatically mailing people on pkgcheck problems with their packages

[Daniel Campbell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 12/06/2015 06:36 AM, Michał Górny wrote:
> Hello,
> 
> As you have seen multiple times, I'm running a minimalistic CI
> service for Gentoo that checks the repository for major issues
> using pkgcheck. So far it's automation is limited to sending a mail
> to dedicated gentoo-automated-testing@lists.gentoo.org mailing list
> on breakage changes. From there, I compare the results to recent
> git log and mail the developers at fault, pointing out the bad
> commit.
> 
> A few developers have already subscribed to the mailing list to
> check if they haven't caused any new breakages and fix them
> quickly. For others, it's pretty much just me caring to check,
> which also means that when I'm not around things are left broken.
> 
> Automating the blaming process has been suggested multiple times 
> already but I so far considered it not worth the effort. Mostly
> because many of the issues are indirect, and trying to
> automatically figure them out from combination of the pkgcheck
> report and recent commits would be hard, and could cause false
> positives. For example, some of the depgraph breakages happen
> because of package.mask changes -- figuring that out automatically
> wouldn't be easy, and the script could blame an irrelevant commit
> in the package.
> 
> However, it was suggested recently that I could make it mail the
> maintainers of the affected packages. Even though most often it's 
> not them who are at fault, it was suggested that they'd prefer to
> know that their packages are broken.
> 
> So what do you think? Would it be fine to mail the package
> maintainers whenever their packages break? Would it be a problem if
> I just CC-ed all the maintainers on the gentoo-automated-testing
> mails? Please note that the breakages are catched per-package, and
> the script wouldn't be able to respect restrict="" or hand-written
> maintainer descriptions ;-).
> 

Sounds fine to me. It's annoying when I come across something that
breaks my deptree, and I don't want my packages to break things,
either. No complaints here, as long as it's clear what the screw-up is.

- -- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWZytdAAoJEAEkDpRQOeFwx10P/1/AaiYBvw67kiFCnQSQ/K89
aHdkwI9KCmVSriBv3lRUMYhR9u48NdwwNPj1X+qlYP9hkFLgE2YEnnDeeegr4EtS
YgqTDuxwrWFzPRX/s8n199drlt5Y71S7B3LBDnOWRZcVOQlqjoLqdPN/FLmfi/Gh
57jCBcCn1nUx9SchidDXLa3xW9Yy0D3UPavIYKknmakVtMTnSx0qfsq2pIc15dp8
k2/m40a2UEitdn8sJKVJpqILs5l/1hGPJhtDkcRtYaHnVq7hVb9ibV7jKC2F/sZh
TgdWhc4VmghpCCZ4ZCXESQa8C3ISCIHp6m1OU9OHYPYfLabQRdHmi9GIgdt0Jyex
UJQzIhEeEatjkYIUwdFJsPYVJ93dOI/Ekymt4UjZR9Ww+LX/HilrH4AXZTTJsX9e
C5bvTnFy0OmSP1/t0IRZ63DWgrphxTZuviP3l8g9fLBZOY9bebeqixtGFpnw9ua+
WYHavsm3ExknYEcSYJi6wLKqYnkM5mK3eK0z85mZ6ONcRsydvBo2lbhzrRpNG8xk
uGrTFHirRVMTmJNWCd0e9pJ26xD7OvKTWMIxp+R+J4xHNe9j20keVFtHevkDJBWG
Qj/lzmFANczg7gW+X0CLKdrlzSI+KB0HMOeJO4+FIn/vgJtUOucrndhEM+jtqL1O
iBRPvpMGZDKFtIjSUii2
=PbDc
-----END PGP SIGNATURE-----