Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Michael Orlitzky <mjo@gentoo.org>]

On 05/17/2016 06:01 AM, Pallav Agarwal wrote:
> Hi,
> You are right, of course.
> The target is to standardize something that would encourage maintainers
> to actually provide non-upstream scripts to test packages. At the same
> time, it should be possible to use those scripts for automated testing
> without
> human interference.
> 

We already have "emerge --config" which is expected to be run after the
install process has completed, so I don't think that this is too much of
a stretch. Maybe call the phase "pkg_test" analogous to "pkg_config" and
in contrast to "src_test" which runs within the working directory. Then
"emerge --test" could run it.

I stole the Emacs "test plan" idea for a few Haskell packages:

  https://wiki.gentoo.org/wiki/Project:Haskell/Test_plans

These do go a little beyond the scope of unit tests: they check that we
installed the package in the right place, that it's available to your
ghci interpreter, and that we haven't installed some other library with
a name collision.

Some of them have predictable output (like when we hash a string), but
others might cause some trouble. The expected output for hscolour is
"you should see a syntax-highlighted version of Example.hs echoed to
your terminal." I don't care what colors show up, as long as that happens.

Still, even if it doesn't work in 100% of cases, I think it could be
useful. Some people may have inflated ideas of what the stabilization
process currently involves. In many cases, it's a compile/execute test
with no expected results and that could easily be automated even if it's
not the world's best stabilization process.