On Tue, 9 Aug 2016 01:59:35 -0400 Rich Freeman wrote: > While I think your proposal is a great one, I think this is actually > the biggest limitation. A lot of our packages (most?) don't actually > have tests that can be run on every build (most don't have tests, some > have tests that take forever to run or can't be used on a clean > install). IMHO, That's not "ideal", but we don't need idealism to be useful here. Tests passing give one kind of useful kind of quality test. But "hey, it compiles" gives useful data in itself. By easy counter example, "it doesn't compile" is in itself useful information ( and the predominant supply of bugs filed are compilation failures ). Hell, sometimes I hit a compile failure and I just go "eeh, I'll look into it next week". How many people are doing the same? The beauty of the automated datapoint is it doesn't have to be "awesome quality" to be useful, its just guidance for further investigation. > While runtime testing doesn't HAVE to be extensive, we do want > somebody to at least take a glance at it. Indeed, I'm not hugely in favour of abolishing manual stabilization entirely, but sometimes it just gets to a point where its a bit beyond a joke with requests languishing untouched for months. If there was even data saying "hey, look, its obvious this isn't ready for stabilization", we could *remove* or otherwise mark for postponement stabilization requests that were failing due to crowd-source metrics. This means it can also be used to focus existing stabilization efforts to reduce the number of things being thrown in the face of manual stabilizers. > > If everything you're proposing is just on top of what we're already > doing, then we have the issue that people aren't keeping up with the > current workload, and even if that report is ultra-nice it is actually > one more step than we have today. The workload would only go down if > a machine could look at the report and stabilize things without input > at least some of the time. Indeed, it would require the crowd service to be automated, and the relevant usage of the data as automated as possible, and humans would only go looking at the data when interested. For instance, when somebody manually files a stable request, some watcher could run off and scour the reports in a given window and comment "Warning: Above threshold failure rates for target in last n-days, proceed with caution", and it would only enhance the existing stabilization workflow.