Re: [gentoo-dev] Stabilizations and src_test

["Michał Górny" <mgorny@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 1949 bytes --]

On Sun, 2020-04-12 at 10:43 +0200, Agostino Sarubbo wrote:
> Hello all,
> 
> If you work on the stabilization workflow you may have noticed that:
> 
> - There are people that rant if you don't run src_test against their packages;
> - There are people that rant if you open a test failure bug against their 
> packages and you block the stabilization.
> 
> So, unless there will be a clear policy about that, I'd suggest to introduce a 
> way to establish if a package is expected to pass src_test or not.
> 
> A simple way to achieve this goal would be:
> introduce a new bugzilla custom flag (like "Runtime testing required" we 
> already have) maybe called "src_test pass" or similar, that, by default is yes 
> and the maintainer should set it to no when needed.
> 
> In case of 'yes', the arch team member must compile with FEATURE="test" and he 
> is allowed to block the stabilization in case of test-failure.
> 
> In case there will be a test-failure there are two choices:
> 1) The maintainer fixes the test failure;
> 2) The maintainer does not want to take care, so he can simply remove the 
> blocker and set "src_test pass" to no.
> 
> Both of the above are fine for me.
> 
> Obviously, if there are already test-failure bugs open, the flag "src_test 
> pass" should be set to 'no' when the stabilization bugs is filled.
> 

This is not a problem that can be solved by a binary flag.

If package's test suite is entirely broken and unmaintained, the package
should use RESTRICT=test and not silently ask arch teams to ignore it.

If package's test suite is only slightly broken, then I'd prefer saying
'please report but ignore *these* test failures' because I can't fix
them right now but they don't seem major.  Skipping the test suite
entirely is not a solution because it doesn't disambiguate between 'few
tests fail' and 'every single test explodes'.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]