[gentoo-dev] A feedback about the CI bug reporting system

[gentoo-dev] A feedback about the CI bug reporting system

[Agostino Sarubbo]

Hello all,

6 months have been passed after the CI system started to file bug reports.
~ 4700 bugs have been submitted

We _know_ that atm is not possible to set a specific summary, instead a 
generic summary is used in case of compile failures and test failures.
There are also some documented limitations.

If there aren't much commits, usually you get the bug after 30 minutes after 
the commit and this looks to be nice.

Since there are conflicting opinions I would like to know if you find it 
useful or not.

More info about the project here:
https://blogs.gentoo.org/ago/2020/07/04/gentoo-tinderbox/

Please keep me CC'ed

Thank you
Agostino

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 1520 bytes --]



On 11/6/20 9:21 AM, Agostino Sarubbo wrote:
> Hello all,
>
> 6 months have been passed after the CI system started to file bug reports.
> ~ 4700 bugs have been submitted
>
> We _know_ that atm is not possible to set a specific summary, instead a 
> generic summary is used in case of compile failures and test failures.
> There are also some documented limitations.
Would it be possible for "someone" to figure it out, if you made your
tinderbox scripts/code public? ;) Hate to say it, but toralf does pretty
good job here, so it could be better.

> Since there are conflicting opinions I would like to know if you find it 
> useful or not.
Yes, I do find your tinderbox work useful most of the time. Thanks!
However the latest, ehh, show with DISTUTILS_USE_SETUPTOOLS has made
people do *hundreds* of commits that now apparently need a fixing
anyway, or reverting back, and that doesn't feel nice. Sure maybe the
eclass could do some fixing too, but maybe this notice wasn't meant to
be full-tree scanned (at least _yet_).

From my point of view your work is, and has been, appreciated, but you
could coordinate better with other people. Hate to say it again, but
toralf does seems to do a better job here too in that regard. Unless
you're fine with comparing tinderboxers.

With toralf's logs it's easy to reproduce the whole environment leading
to a build failure, while with yours it's just build.log, thay may or
may not be enough to find the build-breaking issue.

-- juippis


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Agostino Sarubbo]

On venerdì 6 novembre 2020 09:18:50 CET Joonas Niilola wrote:
> Would it be possible for "someone" to figure it out, if you made your
> tinderbox scripts/code public? ;) Hate to say it, but toralf does pretty
> good job here, so it could be better.

As stated multiple times, toralf said that the way he's filing bugs expect a 
copy-paste action for the summary, this is not possible if you want to do that 
in a more automatic way.


> Yes, I do find your tinderbox work useful most of the time. Thanks!
> However the latest, ehh, show with DISTUTILS_USE_SETUPTOOLS has made
> people do *hundreds* of commits that now apparently need a fixing
> anyway, or reverting back, and that doesn't feel nice. Sure maybe the
> eclass could do some fixing too, but maybe this notice wasn't meant to
> be full-tree scanned (at least _yet_).

This class of bugs comes from a request. Months ago I also asked if I had 
report these bugs during the stabilization and I got a positive feedback.


> From my point of view your work is, and has been, appreciated, but you
> could coordinate better with other people. Hate to say it again, but
> toralf does seems to do a better job here too in that regard. Unless
> you're fine with comparing tinderboxers.
> With toralf's logs it's easy to reproduce the whole environment leading
> to a build failure, while with yours it's just build.log, thay may or
> may not be enough to find the build-breaking issue.

This is something already discussed (maybe privately?) and clearly state by 
me.
If you say that with my logs you are unable to reproduce the issue, since I'm 
providing emerge --info and build log, you are saying that those info are not 
enough.
So, I'd suggest to fix this issue upstream by clearly defining what is needed 
in a bug report, because AFAIK atm for a bug report is needed a build log and 
emerge --info

If you want more info, why do not extend the emerge --info into for example 
emerge --info --extended instead of ask one by one to attach more info??

So the point is that there is no rule but just everyone ask without fixing the 
problem upstream.

While you have a new option for detailed info, every people that run tinderbox 
or just people that report bugs can use it.

If you see an example of toralf bug, I can see:

gcc-config -l:
 [1] x86_64-pc-linux-gnu-7.3.1
 [2] x86_64-pc-linux-gnu-10.2.0 *
clang version 11.0.0
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm/11/bin
/usr/lib/llvm/11
11.0.0
Available Python interpreters, in order of preference:
  [1]   python3.7
  [2]   python3.9 (fallback)
  [3]   python3.8 (fallback)
  [4]   python2.7 (fallback)
Available Ruby profiles:
  [1]   ruby25 (with Rubygems)
  [2]   ruby26 (with Rubygems)
  [3]   ruby27 (with Rubygems) *
Available Rust versions:
  [1]   rust-bin-1.47.0 *
The following VMs are available for generation-2:
1)	IcedTea JDK 3.17.0 [icedtea-8]
*)	AdoptOpenJDK 8.272_p10 [openjdk-bin-8]
Available Java Virtual Machines:
  [1]   icedtea-8 
  [2]   openjdk-bin-8  system-vm

The Glorious Glasgow Haskell Compilation System, version 8.8.4

Again: why these info are not in emerge --info if are useful for a bugreport?


I'm sure you understand the concept

Agostino

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Aaron Bauman]

[-- Attachment #1: Type: text/plain, Size: 3172 bytes --]

On Fri, Nov 06, 2020 at 09:39:35AM +0100, Agostino Sarubbo wrote:
> On venerdì 6 novembre 2020 09:18:50 CET Joonas Niilola wrote:
> > Would it be possible for "someone" to figure it out, if you made your
> > tinderbox scripts/code public? ;) Hate to say it, but toralf does pretty
> > good job here, so it could be better.
> 
> As stated multiple times, toralf said that the way he's filing bugs expect a 
> copy-paste action for the summary, this is not possible if you want to do that 
> in a more automatic way.
>

Is this coming from the same individual who would complain when security
bugs were not filled out properly in the summary? So, take a dose of
your own medicine here. People prefer usable reports that allow them to
solve problems.

> 
> > Yes, I do find your tinderbox work useful most of the time. Thanks!
> > However the latest, ehh, show with DISTUTILS_USE_SETUPTOOLS has made
> > people do *hundreds* of commits that now apparently need a fixing
> > anyway, or reverting back, and that doesn't feel nice. Sure maybe the
> > eclass could do some fixing too, but maybe this notice wasn't meant to
> > be full-tree scanned (at least _yet_).
> 
> This class of bugs comes from a request. Months ago I also asked if I had 
> report these bugs during the stabilization and I got a positive feedback.
> 

Where was this positive feedback? As you stated on #gentoo-dev today you don't
really participate in the ML... so, I presume the positive feedback came
on IRC. Most of us don't scan those logs to "prove" such things. Keep it
on the ML and people will have record.

> 
> > From my point of view your work is, and has been, appreciated, but you
> > could coordinate better with other people. Hate to say it again, but
> > toralf does seems to do a better job here too in that regard. Unless
> > you're fine with comparing tinderboxers.
> > With toralf's logs it's easy to reproduce the whole environment leading
> > to a build failure, while with yours it's just build.log, thay may or
> > may not be enough to find the build-breaking issue.
> 
> This is something already discussed (maybe privately?) and clearly state by 
> me.
> If you say that with my logs you are unable to reproduce the issue, since I'm 
> providing emerge --info and build log, you are saying that those info are not 
> enough.
> So, I'd suggest to fix this issue upstream by clearly defining what is needed 
> in a bug report, because AFAIK atm for a bug report is needed a build log and 
> emerge --info
> 

+1 to coordinate with others. Keep it here on the ML.

This shouldn't be "ago v toralf"... it should be about the impact of
your automation efforts on the rest of the community. Right now, it
looks like that is mostly negative given the ML feedback.

Frankly, if this is anything like your security efforts (re: fuzzing)
then I can understand the concerns people have expressed.

Please, stop with the "automate everything, open many bugs, and move on"
philosophy. It didn't work well in security and it won't work here.
Build a quality solution that makes an impact for the distro.

-- 
Cheers,
Aaron

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Agostino Sarubbo]

IN REPLY TO Michał Górny that didn't keep me CC'ed as requested:

>I do disagree with your presumption that this needs to be automated.
>The whole point behind providing a service is that you should be ready
>to dedicate *your* time into the service. However, we keep feeling that
>you assume that your time is too precious, and it is better to waste
>a little bit of everybody else's time. This is why Toralf's effort is
>much more appreciated.

I don't guess that my time is precious and your time is not.

If you take a look at the amount of bugs you will see that it is not possible 
for an human to file those bugs manually.
So if is not clear, the concept would be: I invest my time to find bugs that 
have not been discovered by others, but you give me an hand to find the exact 
issue by analyzing the build log helping yourself with ctrl-f function of the 
browser






>I am concerned about new test scenarios being run and QA warnings being
>reported without proper research and documentation. It's not exactly
>helpful to spam people with hundreds of bugs without a single proper
>document explaining how to resolve issues. This just provokes people to
>do apply bad workarounds, not to mention wasting everyone's time.

Usually, qa notices are not something invented by me. So in other words you 
are saying that while I'm reporting a QA notice that comes from portage I 
should explain how to solve the problem?
I guess you should resolve the problem upstream. Each QA notice must have 
something that explain how to solve that.
While considering the above, usually to help I add something useful at the end 
of the bug.




>The compiler-rt bugs were one example of both of my points being valid.
>You've started running a specific scenario without researching
>the underlying problems first, and you have missed entirely that you're
>reporting a lots of bugs for the same root issue. People have gotten
>hundreds of bugs with no clear explanation how to fix them. The only
>reason we didn't get hundreds of horrible 'fixes' is that the problem
>was probably too hard to workaround.

Did you forget how the compiler-rt class of bugs started?
Let me remind that:
- I had the tinderbox with no load and nothing to check
- I thought about a round with compiler-rt
- Since _YOU_ are the most active behind the llvm stuff I asked to you if you 
know how to use the compiler-rt stuff ( I have the IRC logs if you really have 
a memory lapse )
- You and Afrever answered me about the C/LD flags to use.
- After the first report (it was an error never seen by me) I asked: Hey is 
this a report related to compiler-rt? You answered yes
- I asked if it was worth to create a bug tracker
- you confirmed
- I started to file the other bugs

So would be better to say to the community that:
- I have asked to you "how-to"
- You shared the necessary info
- I started to file bugs (~50 and not hundreds as you said)
- You (or someone else) discovered that it is broken without libcxx




>DISTUTILS_USE_SETUPTOOLS is yet another example of a minor catastrophe.
>Sure, I could've added more information to the check message. However,
>there is a major difference between people slowly getting QA warnings
>from a new check and people getting hundreds of bugs telling them to fix
>these warnings. This is a difference between involving a thinking
>person, and a semi-automated process that doesn't account for obvious
>mistakes.


DISTUTILS_USE_SETUPTOOLS is yet another example of memory lapse.

Did you forget how the DISTUTILS_USE_SETUPTOOLS class of bugs started?
Let me remind that:
- During the stabilization process I saw these DISTUTILS_USE_SETUPTOOLS 
warnings
- I asked if was fine to report them
- You answered that was fine, but avoid the cases where the warning was in 
stable but was fixed in ~arch
- after few weeks or month ( I don't remember exactly) I was asked by aballier 
to file this class of bugs.

Since tinderbox runs always the latest version of the software, I started to 
file bugs

When you started to see these bugs you also requested to show the exact value 
of DISTUTILS_USE_SETUPTOOLS that I immediately added.

So what I would to point out is: If someone reads this thread think that I 
autonomously started posting nonsense bugs, but in reality these bugs were 
requested and initially also resolved.

So if at some point you discovered that these bugs are nosense because of 
upstream changes, I have no fault.




>AR bugs is another class of controversy. There is one thing to ensure
>that build systems respect AR for toolchain work, there is another to
>assume that it's fine not to have 'ar' archiver on the system. And yes,
>I do realize there are other people involved in this bad idea that
>apparently now you need an arch-specific toolchain to unpack
>the archives inside Debian packages.

So this is something that points out that the situation is not clear, it's not 
me, the tinderbox or the CI.





>I am also concerned about the sheer number of bugs caused by missing
>dependencies. Sure, I do find it valuable to get reports of missing
>dependencies and fix them in my free time. But I disagree that
>stabilizations and keywordings should keep being blocked on problems
>that are unlikely to ever hit real user systems.

I really agree with you, so can you share all cases where you seen this 
problem?
I'm pretty sure you are confusing what comes from the CI and what comes from 
the arch test machines.


Agostino

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Agostino Sarubbo]

IN REPLY to Aaron Bauman that didn't keep me CC'ed as requested:

>Is this coming from the same individual who would complain when security
>bugs were not filled out properly in the summary? So, take a dose of
>your own medicine here. People prefer usable reports that allow them to
>solve problems.

First: we are talking about a different topic, so what happened in security 
context doesn't matter here.

Second: I never complained about summary of security bugs, so since you said: 
"Keep it on the ML and people will have record." can you tell me where your 
statement is recorded?



>Where was this positive feedback? As you stated on #gentoo-dev today you 
>don't really participate in the ML... so, I presume the positive feedback 
>came on IRC. Most of us don't scan those logs to "prove" such things. Keep it
>on the ML and people will have record.

By positive feedback I mean that the system worked and discovered bugs.



>This shouldn't be "ago v toralf"

This isn't ago v toralf and it never was unless you misunderstood.


> Right now, it looks like that is mostly negative given the ML feedback.

I really guess you have a distorted view of reality.


>Frankly, if this is anything like your security efforts (re: fuzzing)
>then I can understand the concerns people have expressed.
>Please, stop with the "automate everything, open many bugs, and move on"
>philosophy. It didn't work well in security and it won't work here.
>Build a quality solution that makes an impact for the distro.

Again, this is something not related of what we are talking about. Fuzzing 
research have been stopped over 3 years ago so what you're talking about?





>ACK. This is the same level of coordination the security team received
>when a multitude of bugs were filed once ago discovered fuzzing. 

Sorry, but I real do not have tracks of what you are talking about.

> It was lots of bugs little information, inabilities to reproduce various
>crashes, invalid ratings/severity levels, and often a blog that
>simply regurgitated the same inaccuracies.

Usually I don't partecipate in mailing list because it is a place where other 
can throw mud on others like this.
Little Information? I do not guess so because the provided information were:
1) command to reproduce
2) stacktrace
3) affected version
4) fixed version
5) commit fix 
6) reproducer
7) timeline

> inabilities to reproduce various crashes
If you can't reproduce a crash it is not my fault

> Any attempt to ask/coordinate was met with lack of information or simply 
"see my blog" responses.

Do you have a track of this?

> The only time interaction occured was when bugs were closed due to
invalidity, lack of information, or severity/ratings downgraded.

Do you have track of this?



In short, please remain on topic, if you have anything to say about other 
projects, feel free to open a thread where we can do a separate discussion ;)

Thanks

P.S.
I don't know why but instead of seeing a constructive discussion I notice that 
there is always a bit of contempt about what others do, and this is really bad 
for an opensource community

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3335 bytes --]

On Fri, 2020-11-06 at 08:21 +0100, Agostino Sarubbo wrote:
> Hello all,
> 
> 6 months have been passed after the CI system started to file bug reports.
> ~ 4700 bugs have been submitted
> 
> We _know_ that atm is not possible to set a specific summary, instead a 
> generic summary is used in case of compile failures and test failures.
> There are also some documented limitations.

I do disagree with your presumption that this needs to be automated.
The whole point behind providing a service is that you should be ready
to dedicate *your* time into the service.  However, we keep feeling that
you assume that your time is too precious, and it is better to waste
a little bit of everybody else's time.  This is why Toralf's effort is
much more appreciated.

> If there aren't much commits, usually you get the bug after 30 minutes after 
> the commit and this looks to be nice.
> 
> Since there are conflicting opinions I would like to know if you find it 
> useful or not.
> 

I am concerned about new test scenarios being run and QA warnings being
reported without proper research and documentation.  It's not exactly
helpful to spam people with hundreds of bugs without a single proper
document explaining how to resolve issues.  This just provokes people to
do apply bad workarounds, not to mention wasting everyone's time.

The compiler-rt bugs were one example of both of my points being valid.
 You've started running a specific scenario without researching
the underlying problems first, and you have missed entirely that you're
reporting a lots of bugs for the same root issue.  People have gotten
hundreds of bugs with no clear explanation how to fix them.  The only
reason we didn't get hundreds of horrible 'fixes' is that the problem
was probably too hard to workaround.

DISTUTILS_USE_SETUPTOOLS is yet another example of a minor catastrophe.
 Sure, I could've added more information to the check message.  However,
there is a major difference between people slowly getting QA warnings
from a new check and people getting hundreds of bugs telling them to fix
these warnings.  This is a difference between involving a thinking
person, and a semi-automated process that doesn't account for obvious
mistakes.

AR bugs is another class of controversy.  There is one thing to ensure
that build systems respect AR for toolchain work, there is another to
assume that it's fine not to have 'ar' archiver on the system.  And yes,
I do realize there are other people involved in this bad idea that
apparently now you need an arch-specific toolchain to unpack
the archives inside Debian packages.

I am also concerned about the sheer number of bugs caused by missing
dependencies.  Sure, I do find it valuable to get reports of missing
dependencies and fix them in my free time.  But I disagree that
stabilizations and keywordings should keep being blocked on problems
that are unlikely to ever hit real user systems.

To summarize, what your tinderboxing effort lacks is really a human
touch.  You seem to have set the goal to file as many bugs as possible
automatically.  I disagree with that, as I would like this effort to
focus on helping developers, not pursuing them.  This requires a human
touch, not a machine lord.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Aaron Bauman]

[-- Attachment #1: Type: text/plain, Size: 1982 bytes --]

On Fri, Nov 06, 2020 at 10:14:30AM +0100, Michał Górny wrote:
> On Fri, 2020-11-06 at 08:21 +0100, Agostino Sarubbo wrote:
> > Hello all,
> > 
> > 6 months have been passed after the CI system started to file bug reports.
> > ~ 4700 bugs have been submitted
> > 
> > We _know_ that atm is not possible to set a specific summary, instead a 
> > generic summary is used in case of compile failures and test failures.
> > There are also some documented limitations.
> 
> I do disagree with your presumption that this needs to be automated.
> The whole point behind providing a service is that you should be ready
> to dedicate *your* time into the service.  However, we keep feeling that
> you assume that your time is too precious, and it is better to waste
> a little bit of everybody else's time.  This is why Toralf's effort is
> much more appreciated.
>

ACK. This is the same level of coordination the security team received
when a multitude of bugs were filed once ago discovered fuzzing. It was
lots of bugs, little information, inabilities to reproduce various
crashes, invalid ratings/severity levels, and often a blog that
simply regurgitated the same inaccuracies. Any attempt to ask/coordinate
was met with lack of information or simply "see my blog" responses.

The only time interaction occured was when bugs were closed due to
invalidity, lack of information, or severity/ratings downgraded.

All this to say, I concur his actions seem to show that he believes his
time is more precious than others and that "numbers matter" when it
comes to opening bugs and CVE's.

> To summarize, what your tinderboxing effort lacks is really a human
> touch.  You seem to have set the goal to file as many bugs as possible
> automatically.  I disagree with that, as I would like this effort to
> focus on helping developers, not pursuing them.  This requires a human
> touch, not a machine lord.
>

This right here.

-- 
Cheers,
Aaron

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 2039 bytes --]

On Fri, Nov 06, 2020 at 08:21:39AM +0100, Agostino Sarubbo wrote:
> Hello all,
> 
> 6 months have been passed after the CI system started to file bug reports.
> ~ 4700 bugs have been submitted
> 
> We _know_ that atm is not possible to set a specific summary, instead a 
> generic summary is used in case of compile failures and test failures.
> There are also some documented limitations.
> 
> If there aren't much commits, usually you get the bug after 30 minutes after 
> the commit and this looks to be nice.
I value the quantitative knowledge of "there is some problem", but I
absolutely want process improvement to tell me "there is XYZ specific
problem" (quantitative).

Can you please tell us what you need to let others contribute to
improving the quality of the reports from your CI system?
I will happily submit patches to help improve things that matter to me
(recently, I've been repeatedly bitten by packages where DEPEND needs to
specify a newer min version; the current min version in the tree is
fine, but since I have an older version installed, and the DEPEND
statement wasn't updated, I get a failure until I re-order the updates).

Examples:
- It needs $ABC added
- If it could use $SOMETHING, it could run on a large build farm results
  in 2 minutes instead of 30 minutes.
- Compile fails on XYZ arch

No single CI system will ever continue to be successful with a growing
ecosystem if it's limited to a single build system. Sooner or later they
all need to be able to build on a farm.

> 
> Since there are conflicting opinions I would like to know if you find it 
> useful or not.
> 
> More info about the project here:
> https://blogs.gentoo.org/ago/2020/07/04/gentoo-tinderbox/
> 
> Please keep me CC'ed
> 
> Thank you
> Agostino
> 
> 
> 

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Agostino Sarubbo]

On sabato 7 novembre 2020 07:15:31 CET Robin H. Johnson wrote:
> Can you please tell us what you need to let others contribute to
> improving the quality of the reports from your CI system?

Hello Robin,

I don't understand why in general people focus on what is missing into a 
simple script that merges packages instead of ask himself where this feature 
should go.

In my opinion, we can try to do whatever to the script or we can write a new 
one but the main problem is in portage that only states in which phase emerge 
is dying.

If emerge is able to tell the exact error, then it can be used also from users 
that want to do bug reports.

Agostino

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Andreas K. Hüttel]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

Am Samstag, 7. November 2020, 10:26:27 EET schrieb Agostino Sarubbo:
> On sabato 7 novembre 2020 07:15:31 CET Robin H. Johnson wrote:
> > Can you please tell us what you need to let others contribute to
> > improving the quality of the reports from your CI system?
> 
> Hello Robin,
> 
> I don't understand why in general people focus on what is missing into a
> simple script that merges packages instead of ask himself where this feature
> should go.

Ago, 

this whole thread started out about the quality of the bug reports. However... 
Seriously, with the insistence to keep your setup closed-source, you are not 
helping your case. 

If you want this to be an in any way official Gentoo project, you'll have to 
stick to the Gentoo social contract just like everyone else.

Cheers,
Andreas

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, qa, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Agostino Sarubbo]

On venerdì 6 novembre 2020 08:21:39 CET Agostino Sarubbo wrote:
> Hello all,
> 
> 6 months have been passed after the CI system started to file bug reports.
> ~ 4700 bugs have been submitted
> 
> We _know_ that atm is not possible to set a specific summary, instead a
> generic summary is used in case of compile failures and test failures.
> There are also some documented limitations.
> 
> If there aren't much commits, usually you get the bug after 30 minutes after
> the commit and this looks to be nice.
> 
> Since there are conflicting opinions I would like to know if you find it
> useful or not.
> 
> More info about the project here:
> https://blogs.gentoo.org/ago/2020/07/04/gentoo-tinderbox/
> 
> Please keep me CC'ed
> 
> Thank you
> Agostino


Thanks all for the feedback.

What looks strange is that ci/tinderbox opened ~4700 bugs, sure, there are 
invalid/duplicate bugs but the majority are fixed so in my opinion they were 
useful, but looking at the mailing list feedback looks to be a completely 
crap.

I want to play the game of destructive people but in a constructive way.

Thanks to @gyakovlev for the idea I have opened a new github project where we 
can collect the requests and the bugs:

https://github.com/asarubbo/ci/issues


An important note:

I consider this 'project' as something related to QA ( if you have a different 
opinion feel free to say ), so since I received rant from developers for 
something requested by other developers, I will touch the code ONLY when there 
is the QA lead approval.


When you want a new feature or you want to modify something that already 
exists, please open a thread (gentoo-dev is fine, and thanks to bman for the 
tracking idea) and open a github issue only when there is a written track of 
the QA lead approval mentioned above.


There is a README into the repository that explains all class of bugs 
discovered by the CI system.

Thank you
Agostino

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 1332 bytes --]

Hi,

On 2020-11-07 12:30, Agostino Sarubbo wrote:
> On venerdì 6 novembre 2020 08:21:39 CET Agostino Sarubbo wrote:
>> Hello all,
>> 
>> 6 months have been passed after the CI system started to file bug
>> reports. ~ 4700 bugs have been submitted
>> 
>> We _know_ that atm is not possible to set a specific summary,
>> instead a generic summary is used in case of compile failures and
>> test failures. There are also some documented limitations.

I have to second what other already said.

Dropping bugs and forcing maintainer to review and spend time to check
if there is a problem and what was the reported problem at all creates
more work. And I consider anything creating more load for others which 
was not requested not as helpful.

That said, I don't have these problems with toralf's reports. They are
more complete and will show the problem in the report for most bugs.


> but the majority are fixed so in my opinion they were useful

I do not agree with this conclusion. Just because developers didn't 
ignore you and spent additional time to understand and try to help like 
we normally do when we get reports from inexperienced users, doesn't 
mean it was a pleasure...


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Agostino Sarubbo]

Hello Thomas,

On sabato 7 novembre 2020 17:06:56 CET Thomas Deutschmann wrote:
> I have to second what other already said.
> 
> Dropping bugs and forcing maintainer to review and spend time to check
> if there is a problem and what was the reported problem at all creates
> more work. And I consider anything creating more load for others which
> was not requested not as helpful.

Let me repeat what I have already written.

Does portage print the exact error when dying? No, so we have a package 
manager that ends without saying why and while everyone is refusing to talk 
about that and understand this, in my opinion this issue denies everyone to 
make an automated full/complete report.

Toralf did and is doing a great job, but you are confusing CI with a man that 
reviews and files bug half-manually.

> That said, I don't have these problems with toralf's reports. They are
> more complete and will show the problem in the report for most bugs.

Let me repeat what I have already written somewhere.
The rule in case of bug report is to attach the build log and provide emerge 
--info. If you think that those info are not enough that's fine, but please 
document that.

> I do not agree with this conclusion. Just because developers didn't
> ignore you and spent additional time to understand and try to help like
> we normally do when we get reports from inexperienced users, doesn't
> mean it was a pleasure...

If find the exact error in a build log requires too much time and you do not 
want to spend time, instead of be forced to not ignore me you can request to 
be excluded from the reports.

In general the CI reports are there to help. People that does not see those 
reports as help can request to not receive reports anymore.

Agostino

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Marek Szuba]

[-- Attachment #1.1.1: Type: text/plain, Size: 1034 bytes --]

On 2020-11-07 12:30, Agostino Sarubbo wrote:

> What looks strange is that ci/tinderbox opened ~4700 bugs, sure, there are
> invalid/duplicate bugs but the majority are fixed so in my opinion they were
> useful, but looking at the mailing list feedback looks to be a completely
> crap.

I think this might be the case of a "silent majority, vocal minority" 
scenario. For the record: I for one have found your tinderbox bugs very 
useful, not in the least because you test unusual configurations my own 
build testing does not cover. There have been glitches - but IMHO very 
few of them, and human operators make mistakes too.

I still very much wish your tickets were tagged in a way which would 
make it possible to separate failures from QA notices (especially those 
that show up during normal emerge runs, such as that suspected 
DISTUTILS_USE_SETUPTOOLS mismatches), though. I really do not need 
e-mail notifications about the latter, looking them up via my p.g.o page 
is quite sufficient.

-- 
Marecki

[-- Attachment #1.1.2: OpenPGP_0xD4945FFDD7FE7E37_and_old_rev.asc --]
[-- Type: application/pgp-keys, Size: 122125 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] A feedback about the CI bug reporting system

[Agostino Sarubbo]

On lunedì 9 novembre 2020 13:13:19 CET Marek Szuba wrote:
> I think this might be the case of a "silent majority, vocal minority"
> scenario. For the record: I for one have found your tinderbox bugs very
> useful, not in the least because you test unusual configurations my own
> build testing does not cover. There have been glitches - but IMHO very
> few of them, and human operators make mistakes too.
> 
> I still very much wish your tickets were tagged in a way which would
> make it possible to separate failures from QA notices (especially those
> that show up during normal emerge runs, such as that suspected
> DISTUTILS_USE_SETUPTOOLS mismatches), though. I really do not need
> e-mail notifications about the latter, looking them up via my p.g.o page
> is quite sufficient.

Hello Marek and thank you for the feedback.

Ftr DISTUTILS_USE_SETUPTOOLS has been disabled for a while but I think we can 
use a tag in WHITEBOARD to meet your needs.
I think it has been used in the past and it doesn't hurt have it.

Agostino