Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Tobias Klausmann <klausman@gentoo.org>]

Hi! 

On Tue, 17 May 2016, Kent Fredric wrote:

> On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com> wrote:
> > For normal users we wouldn't. But currently, arch-testers need to make a
> > judgement call on what to test when a stable-req bug is filed. Tests run in
> > src_test are provided by upstream, and does not guarantee that a package
> > that has been merged will actually run on the system.
> > If the maintainer could add a couple small scripts to check basic
> > functionality
> > of the merged package, it would make testing for arch testers much easier
> > and reliable.
> > Let me give an example. Let's say
> > app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in
> > the portage tree.
> > However, on running, it produces an error on the top, along with the proper
> > output.
> > If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple
> > script
> >
> >        if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then
> >              eerror "Still producing error"
> >        fi
> >
> > To make sure the build is properly updating the screenfetch version, and
> > that
> > the bug has in fact been fixed. This is the only way I can see to reliabily
> > and automatically test packages that have been merged successfully
> 
> 
> I don't think this needs to be an EAPI change. And if we can acheive
> the goal without one, the better.

Agreed. 

> [... lots of interesting stuff ...]

Or maintainers and teams could do what the Emacs team does:
provide test plans[0].

Naturally, I'd prefer something automated over the very hands-on
tests that are a bit of a necessity for an application like
Emacs, but they are still better than nothing.

Either way, I think it is preferable to have non-upstream tests
of whatever shape we pick to be separate from the ebuilds and the
source files, for the simple reason that most users won't care
about them. Those who do can retrieve them in the same manner as
the ATs.

And as for my pet peeve, tests that are known to fail, can we
also annotate that somehow so I don't waste hours running a test
suite that gives zero signal on whether I should add the stable
keyword? Even a one-line hin in the stabilization request would
be nice. As it is, I keep a list of known-to-fail packages and my
testing machinery tells me to not bother with FEATURES=test in
those case.

Not ideal, but less time-wasty.

Regards,
Tobias

[0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans


-- 
if (user_specified)
    /* Didn't work, but the user is convinced this is the
     * place. */
        linux-2.4.0-test2/drivers/parport/parport_pc.c