[gentoo-dev] Stabilizations and src_test

[gentoo-dev] Stabilizations and src_test

[Agostino Sarubbo]

Hello all,

If you work on the stabilization workflow you may have noticed that:

- There are people that rant if you don't run src_test against their packages;
- There are people that rant if you open a test failure bug against their 
packages and you block the stabilization.

So, unless there will be a clear policy about that, I'd suggest to introduce a 
way to establish if a package is expected to pass src_test or not.

A simple way to achieve this goal would be:
introduce a new bugzilla custom flag (like "Runtime testing required" we 
already have) maybe called "src_test pass" or similar, that, by default is yes 
and the maintainer should set it to no when needed.

In case of 'yes', the arch team member must compile with FEATURE="test" and he 
is allowed to block the stabilization in case of test-failure.

In case there will be a test-failure there are two choices:
1) The maintainer fixes the test failure;
2) The maintainer does not want to take care, so he can simply remove the 
blocker and set "src_test pass" to no.

Both of the above are fine for me.

Obviously, if there are already test-failure bugs open, the flag "src_test 
pass" should be set to 'no' when the stabilization bugs is filled.

I'm sure that this way would help to avoid wasting time on both sides.

What do you think about?

--
Agostino

Re: [gentoo-dev] Stabilizations and src_test

[Marek Szuba]

[-- Attachment #1.1: Type: text/plain, Size: 1339 bytes --]

On 2020-04-12 09:43, Agostino Sarubbo wrote:

> If you work on the stabilization workflow you may have noticed that:
> 
> - There are people that rant if you don't run src_test against their packages;
> - There are people that rant if you open a test failure bug against their 
> packages and you block the stabilization.
> 
> So, unless there will be a clear policy about that, I'd suggest to introduce a 
> way to establish if a package is expected to pass src_test or not.

My first reaction was "not against this, even though I personally very
much try (as in, that I occasionally fail or forget) to make sure even
unstable ebuilds I push into the tree pass all tests on amd64", having
thought about it more however it feels like lowering quality standards
for stable arch trees. Test failures in ~arch are one thing but I would
be somewhat suspicious of stabilising anything which fails tests unless
it is a known (as in, known to the upstream) issue with no easy solution.

There is also the slight image problem which could arise from a user
running tests themselves and reporting the failure back to us, with the
"I thought this was supposed to be *stable*, guys?" undertone.

In summary, I am not entirely decided but leaning towards continuing to
always run tests during stabilisation.

-- 
Marecki


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Stabilizations and src_test

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1949 bytes --]

On Sun, 2020-04-12 at 10:43 +0200, Agostino Sarubbo wrote:
> Hello all,
> 
> If you work on the stabilization workflow you may have noticed that:
> 
> - There are people that rant if you don't run src_test against their packages;
> - There are people that rant if you open a test failure bug against their 
> packages and you block the stabilization.
> 
> So, unless there will be a clear policy about that, I'd suggest to introduce a 
> way to establish if a package is expected to pass src_test or not.
> 
> A simple way to achieve this goal would be:
> introduce a new bugzilla custom flag (like "Runtime testing required" we 
> already have) maybe called "src_test pass" or similar, that, by default is yes 
> and the maintainer should set it to no when needed.
> 
> In case of 'yes', the arch team member must compile with FEATURE="test" and he 
> is allowed to block the stabilization in case of test-failure.
> 
> In case there will be a test-failure there are two choices:
> 1) The maintainer fixes the test failure;
> 2) The maintainer does not want to take care, so he can simply remove the 
> blocker and set "src_test pass" to no.
> 
> Both of the above are fine for me.
> 
> Obviously, if there are already test-failure bugs open, the flag "src_test 
> pass" should be set to 'no' when the stabilization bugs is filled.
> 

This is not a problem that can be solved by a binary flag.

If package's test suite is entirely broken and unmaintained, the package
should use RESTRICT=test and not silently ask arch teams to ignore it.

If package's test suite is only slightly broken, then I'd prefer saying
'please report but ignore *these* test failures' because I can't fix
them right now but they don't seem major.  Skipping the test suite
entirely is not a solution because it doesn't disambiguate between 'few
tests fail' and 'every single test explodes'.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Stabilizations and src_test

[Thomas Deutschmann]

[-- Attachment #1.1: Type: text/plain, Size: 1295 bytes --]

On 2020-04-12 11:21, Michał Górny wrote:
> This is not a problem that can be solved by a binary flag.
> 
> If package's test suite is entirely broken and unmaintained, the package
> should use RESTRICT=test and not silently ask arch teams to ignore it.
> 
> If package's test suite is only slightly broken, then I'd prefer saying
> 'please report but ignore *these* test failures' because I can't fix
> them right now but they don't seem major.  Skipping the test suite
> entirely is not a solution because it doesn't disambiguate between 'few
> tests fail' and 'every single test explodes'.

ACK. I also see no need for any new mechanism.


> - There are people that rant if you open a test failure bug against their 
> packages and you block the stabilization.

Maybe start ignoring those packages until people learn that
stabilization is a lot of effort/work. Really, if you call for
stabilization and haven't tested your own package you are offloading
work to others which is not nice. I also dislike maintainers who simply
restrict tests on first failure. But in the end it's at least a strong
signal about package quality and state in Gentoo. :)


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Stabilizations and src_test

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 2157 bytes --]

On Sun, 12 Apr 2020 10:43:07 +0200
Agostino Sarubbo <ago@gentoo.org> wrote:

> If you work on the stabilization workflow you may have noticed that:
> 
> - There are people that rant if you don't run src_test against their packages;
> - There are people that rant if you open a test failure bug against their 
> packages and you block the stabilization.
> 
> So, unless there will be a clear policy about that, I'd suggest to introduce a 
> way to establish if a package is expected to pass src_test or not.
> 
> A simple way to achieve this goal would be:
> introduce a new bugzilla custom flag (like "Runtime testing required" we 
> already have) maybe called "src_test pass" or similar, that, by default is yes 
> and the maintainer should set it to no when needed.
> 
> In case of 'yes', the arch team member must compile with FEATURE="test" and he 
> is allowed to block the stabilization in case of test-failure.
> 
> In case there will be a test-failure there are two choices:
> 1) The maintainer fixes the test failure;
> 2) The maintainer does not want to take care, so he can simply remove the 
> blocker and set "src_test pass" to no.
> 
> Both of the above are fine for me.
> 
> Obviously, if there are already test-failure bugs open, the flag "src_test 
> pass" should be set to 'no' when the stabilization bugs is filled.
> 
> I'm sure that this way would help to avoid wasting time on both sides.

Expecting the package to be marked stable while the tests fail seems
like a very odd stance to take. I would politely request that you flag
this up and if you get pushback but understandably don't want to fight
it then just add RESTRICT="test" yourselves, maybe filtered on a
particular USE flag if it only happens in specific situations. I think
the QA team would back you up here?

I understand the comments about only a few minor tests failing but if
the maintainer doesn't want to skip the entire suite then it is their
responsibility to limit the set of tests run by whatever means.
Unfortunately there is no universal method of doing this.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Stabilizations and src_test

[Andreas Sturmlechner]

On Sunday, 12 April 2020 11:33:38 CEST Thomas Deutschmann wrote:
> On Sunday, 12 April 2020 10:43:07 CEST Agostino Sarubbo wrote:
> > - There are people that rant if you open a test failure bug against their
> > packages and you block the stabilization.
> 
> Maybe start ignoring those packages until people learn that
> stabilization is a lot of effort/work. Really, if you call for
> stabilization and haven't tested your own package you are offloading
> work to others which is not nice. I also dislike maintainers who simply
> restrict tests on first failure. But in the end it's at least a strong
> signal about package quality and state in Gentoo. :)

Not so fast. Let's not forget that so many tests are fragile and not even easy 
to reproduce on a different system. And then there are those that randomly 
break by a dependency bump after the fact, we don't have continuous build CI 
so no one is to blame. It sends more of a signal about upstream package 
quality than Gentoo's, or that they just do not care to make tests work for 
packaging environments, and it is not a good use of our time to make it so. I 
agree then that it is better to simply restrict if we know the tests fail, 
rather than introduce a new mechanism.

And when I look at the amount of foreign (to me) packages I need to call 
stabilisation for because their maintainer didn't, then that work has already 
been offloaded to me, so we're in the same boat.

People may rant when their stabilisation is blocked by failing tests, but let  
me recommend to not take that personally when it most likely is just a big  
sigh at the circumstances/package in question.

Regards,
Andreas

Re: [gentoo-dev] Stabilizations and src_test

[Patrick McLean]

On Sun, 12 Apr 2020 11:21:29 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> On Sun, 2020-04-12 at 10:43 +0200, Agostino Sarubbo wrote:
> > Hello all,
> > 
> > If you work on the stabilization workflow you may have noticed that:
> > 
> > - There are people that rant if you don't run src_test against their packages;
> > - There are people that rant if you open a test failure bug against their 
> > packages and you block the stabilization.
> > 
> > So, unless there will be a clear policy about that, I'd suggest to introduce a 
> > way to establish if a package is expected to pass src_test or not.
> > 
> > A simple way to achieve this goal would be:
> > introduce a new bugzilla custom flag (like "Runtime testing required" we 
> > already have) maybe called "src_test pass" or similar, that, by default is yes 
> > and the maintainer should set it to no when needed.
> > 
> > In case of 'yes', the arch team member must compile with FEATURE="test" and he 
> > is allowed to block the stabilization in case of test-failure.
> > 
> > In case there will be a test-failure there are two choices:
> > 1) The maintainer fixes the test failure;
> > 2) The maintainer does not want to take care, so he can simply remove the 
> > blocker and set "src_test pass" to no.
> > 
> > Both of the above are fine for me.
> > 
> > Obviously, if there are already test-failure bugs open, the flag "src_test 
> > pass" should be set to 'no' when the stabilization bugs is filled.
> >   
> 
> This is not a problem that can be solved by a binary flag.
> 
> If package's test suite is entirely broken and unmaintained, the package
> should use RESTRICT=test and not silently ask arch teams to ignore it.
> 
> If package's test suite is only slightly broken, then I'd prefer saying
> 'please report but ignore *these* test failures' because I can't fix
> them right now but they don't seem major.  Skipping the test suite
> entirely is not a solution because it doesn't disambiguate between 'few
> tests fail' and 'every single test explodes'.
> 

I agree with this, the metric for marking an ebuild as "tests
don't work, please ignore them is RESTRICT=test". Usually for packages
that have a *few* tests that fail and don't seem major, I would suggest
trying to patch out the tests, or ask the test harness to skip the
known bad tests. This way the tinderbox will (hopefully) still succeed.

Re: [gentoo-dev] Stabilizations and src_test

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 3362 bytes --]

On Sun, 12 Apr 2020 10:43:07 +0200
Agostino Sarubbo <ago@gentoo.org> wrote:

> In case of 'yes', the arch team member must compile with FEATURE="test" and he 
> is allowed to block the stabilization in case of test-failure.
> 
> In case there will be a test-failure there are two choices:
> 1) The maintainer fixes the test failure;
> 2) The maintainer does not want to take care, so he can simply remove the 
> blocker and set "src_test pass" to no.
> 
> Both of the above are fine for me.
> 
> Obviously, if there are already test-failure bugs open, the flag "src_test 
> pass" should be set to 'no' when the stabilization bugs is filled.
> 
> I'm sure that this way would help to avoid wasting time on both sides.
> 
> What do you think about?

Agree mostly, but this problem is in two places.

Ebuilds themselves need more mechanisms to communicate this, as if an
ebuild has problematic tests, then they're likely to be problematic
every time, and remembering to set the appropriate bugzilla flags
becomes a weakness.

In this vein, there's lots of similar inadequacies you find if you
follow this thread.

Like for instance, I maintain a lot of packages where the ability for
the tests to work is *environment specific*, and there's ultimately
different *kinds* of testing, and the audiences who run each varies.

Take perl packages for instance. There's a few where the test suite
*cannot* work, because they need either network IO or access to
physical devices to function.

But simply disabling tests because of this produces a gross weakness,
because Perl code cannot even be considered to be compatible with the
target perl until you actually execute it, because those
incompatibilities are ultimately "compiler errors", and you only see
them when you spin up a runtime, unlike code with a dedicated compile
stage.

Subsequently, I've taken to bolting on ebuild-internal hacks that
simply enumerate the modules and forces a perl instance to try loading
them, to at least provide *some* degree of assurance that the code
*probably* works.

But ultimately this means the way I see it, every ebuild could have
*multiple* test targets, and could have a control variable indicating
which targets to execute by default, and consumer knobs could expand,
or reduce, the testing based on preference.

We could even have something like say:

TEST_TARGETS="+build integration"
TEST_RESTRICT="network-sandbox? ( !root? ( integration ))"


Things like say, mysql, which has a standard test suite that takes days
to run, might be well served by having controls like this, so consumers
who want to have *some* basic assurance code works, without paying the
price for a comprehensive smoke, can do so, instead of opting to choose
"tests are hard, lets not do them".

Just I've been cautious about talking about this, because this is such
a can of worms, and leads into the debate about TDEPEND, and how we
stuff everything into USE flags...

And I really would *hate* for testing control variables to be USE
flags, where --changed-use rampantly introduces unnecessary havoc

( And the eventual fact will be there will be code that tries to depend
on specific values of these test control flags, and code that wants to
omit dependencies based on them, and portage complexity goes cray cray )

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]