Re: [gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Jeroen Roovers <jer@gentoo.org>]

On Mon, 7 Jun 2010 12:10:04 +0200
Thilo Bangert <bangert@gentoo.org> wrote:

> i do agree, that all packages should build successfully including the
> test phase. RESTRICTing the test and an open bug when this is not the
> case.

I see more and more calls for either 1) "fixing the test suite", as if
that is suddenly not an UPSTREAM issue but the ebuilds' maintainers' or
2) setting RESTRICT=test, which would have everyone ignore the
useful aspects of a package's test suite.

The main problem with RESTRICT=test is that it's too restrictive - it
prevents a normal emerge(1) or ebuild(1) run from entering the test
phase at all, with no way to work around it, except by editing the
ebuild to remove the restriction.

======
marga /keeps/gentoo/local/app-portage/foobar # ebuild foobar-1.ebuild
test Forcing test.
 * checking ebuild
checksums ;-) ...                                            [ ok ]
 * checking auxfile
checksums ;-) ...                                           [ ok ]
 * checking miscfile
checksums ;-) ...                                          [ ok ]
 * CPV:  app-portage/foobar-1
 * REPO: JeR
 * USE:  elibc_glibc kernel_linux ppc test userland_GNU
>>> Unpacking source...
>>> Source unpacked in /var/tmp/portage/app-portage/foobar-1/work
>>> Compiling source in /var/tmp/portage/app-portage/foobar-1/work ...
>>> Source compiled.
 * Skipping make test/check due to ebuild restriction.
>>> Test phase [explicitly disabled]: app-portage/foobar-1
marga /keeps/gentoo/local/app-portage/foobar # cat foobar-1.ebuild 
# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

SLOT="0"
RESTRICT="test"
======

If you believe that test suites are a Good Thing, then you should view
RESTRICT=test as the ultimate solution to fix the problem of a broken
test suite that will never be fixed. Now, in case a test suite could
actually be destructive, dangerous to run on an unprepared system, then
there RESTRICT=test is a fine solution.

When instead a test suite should do a SKIP but erroneously does a FAIL,
then RESTRICT=test is not the solution. Fixing the test suite is, but
then that's not the maintainer's problem, but upstream's. Oddly enough
we have QA checks in place (for ICEs, for instance) that direct users
directly to upstream (through the HOMEPAGE variable), when it's
suddenly the maintainer's problem if a package fails its test suite
(because of FEATURES=userpriv or a missing kernel feature, perhaps -
nothing the maintainer can prepare the user's system for). There's
currently no way to describe the test suite's requirements to the user
except by building in many checks or perhaps by simply listing them in
an ewarn.

Since the variable will be passed on to every next ebuild you're stuck
with it, unless you are prepared to edit out RESTRICT=test, see if the
new version still fails, then edit it in again when you find nothing
has been fixed. This in turn doesn't make for easy ebuild maintenance.

There's something wrong with the way we do test phases, the way some
of us rely on them to foretell the stability or quality of a package
version, and the way we stop test suites from failing (by
only having a binary RESTRICT=test).

We could easily extend metadata.xml to describe the test phase to the
developer and user right now, for one thing, and aim for a more
automated approach to fixing the binary problem in the future.

Another solution is to change how emerge(1) and ebuild(1) deal with
RESTRICT=test. On the one hand FEATURES=test should be enabled
by testers and users with caution, as many test suites simply don't do
what you might think they do - there is no guarantee that a program
will run well in the Real World by merely satisfying its (limited?) test
conditions. With that in mind, RESTRICT=test should perhaps not block
testing the way that it does now.


     jer