On Tue, May 17, 2016 at 5:15 AM, Kent Fredric <kentfredric@gmail.com> wrote:
> On 17 May 2016 at 20:46, Tobias Klausmann <klausman@gentoo.org> wrote:
>> And as for my pet peeve, tests that are known to fail, can we
>> also annotate that somehow so I don't waste hours running a test
>> suite that gives zero signal on whether I should add the stable
>> keyword? Even a one-line hin in the stabilization request would
>> be nice. As it is, I keep a list of known-to-fail packages and my
>> testing machinery tells me to not bother with FEATURES=test in
>> those case.
>
> IMO: Tests that are "expected to fail" should be killed.
>

That makes sense, though ironically the only specific hypothetical use
case to come up so far was an example of just this situation.  A
package is broken in stable, and a test was proposed to detect if
future stable candidates fix the flaw.  There would be no point in
delaying stabilization of a package that contains the same error as
the current stable version.

I don't see any harm in adding support for automated Gentoo-specific
tests, but I am skeptical of how much use they'll actually get.  After
all, we started off with the statement that this is for situations
where upstream doesn't provide test suites, and if upstream can't be
bothered, why would we expect a distro maintainer to care more?

--
Rich