[gentoo-dev] Re: tests

[Ryan Hill <dirtyepic@gentoo.org>]

Piotr Jaroszyński wrote:
> Hello,
> 
> There was some discussion about forcing/not forcing tests in EAPI-1, but there 
> was clearly no compromise. Imho, tests are very important and thus I want to 
> discuss them a little more, but in more sensible fashion.
> 
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing tests
> Is that list comprehensive?

I've been running with FEATURES=test for a long time now.  Here's some
of the more interesting cases:

- fail only on little/big-endian archs
- fail only with/without root privs
- fail only if dependencies are / are not compiled with certain optional
support
- fail only with GCC >=4
- are expected to fail and are only meant as a regression test
- take 3 minutes on x86 and 3 hours on mips
- fail on hardened
- fail with/without new tar versions
- fail with/without new flex versions (etc.)
- fail if a kernel component is a module instead of built-in
- fail if certain environment variables are set
- fail if compiled with certain (safe) CFLAGS/CXXFLAGS

Can we qualify each of these into one of your categories?  (NB: I
realize there are solutions for each of these examples.  I'm pointing
out that not only is the situation not black and white, it often ranges
in the ultra-violet.)

> Secondly we must answer the question how precisely we want to distinguish 
> them, so users/dev can choose which categories of tests they want to run. 
> What comes to mind is:
> - run all tests
> - run only necessary tests
> - run only reasonable tests
> - don't run tests at all
> Again, is that list comprehensive?

- run only tests that don't require extra deps
- run only tests that work on hardened
- run only tests that work on my arch

> Please don't post solutions unless we figure out which options we really want 
> to deliver.

Sorry. (neener neener) ;)  Would people accept running src_test() by
default only on packages in the system set?  There are some that we
might want to turn off - glibc, gcc, binutils, autoconf, and automake
are on my current short list.  coreutils is also a lot of fun.  db takes
six hours.

Anyone, however, who is of the opinion that tests for their packages are
so important that they should never be skipped, and who is willing to
deal with the bug reports that will undeniably be generated as a result,
should IMO be allowed to shoot themselves in the foot, while those
uninterested can go about their business without further interruption.

In no way should this be tied to EAPI.


-- 
                                where to now? if i had to guess
dirtyepic gentoo org        i'm afraid to say antarctica's next
9B81 6C9F E791 83BB 3AB3  5B2D E625 A073 8379 37E8 (0x837937E8)

-- 
gentoo-dev@gentoo.org mailing list