Hi,
You are right, of course.
The target is to standardize something that would encourage maintainers
to actually provide non-upstream scripts to test packages. At the same
time, it should be possible to use those scripts for automated testing
without
human interference.

Even if they are provided independently, not as part of ebuilds, we need a
to have them associated with a particular ebuild if need be as in case of a
bug fix, or many ebuilds in case of basic functionality for all versions of
a
package.
> And under this scheme, individual projects can define their custom
> hooks and tricks in the ebuild themselves as metadata ( like we
> currently to via eclass variables ), and be handled, not by portage,
> or PMS, not by EAPI, but by the QA tool in conjunction with eclass
> designers.
>
I'm not sure how to deal with eclasses but I guess I'll find out.

On Tue, May 17, 2016 at 2:16 PM, Tobias Klausmann <klausman@gentoo.org>
wrote:

> Hi!
>
> On Tue, 17 May 2016, Kent Fredric wrote:
>
> > On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com>
> wrote:
> > > For normal users we wouldn't. But currently, arch-testers need to make
> a
> > > judgement call on what to test when a stable-req bug is filed. Tests
> run in
> > > src_test are provided by upstream, and does not guarantee that a
> package
> > > that has been merged will actually run on the system.
> > > If the maintainer could add a couple small scripts to check basic
> > > functionality
> > > of the merged package, it would make testing for arch testers much
> easier
> > > and reliable.
> > > Let me give an example. Let's say
> > > app-misc/screenfetch-2.7.7 is the current stable package for
> screenfetch in
> > > the portage tree.
> > > However, on running, it produces an error on the top, along with the
> proper
> > > output.
> > > If screenfetch-3.0.0 happens to fix that error, maintainer can add a
> simple
> > > script
> > >
> > >        if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then
> > >              eerror "Still producing error"
> > >        fi
> > >
> > > To make sure the build is properly updating the screenfetch version,
> and
> > > that
> > > the bug has in fact been fixed. This is the only way I can see to
> reliabily
> > > and automatically test packages that have been merged successfully
> >
> >
> > I don't think this needs to be an EAPI change. And if we can acheive
> > the goal without one, the better.
>
> Agreed.
>
> > [... lots of interesting stuff ...]
>
> Or maintainers and teams could do what the Emacs team does:
> provide test plans[0].
>
> Naturally, I'd prefer something automated over the very hands-on
> tests that are a bit of a necessity for an application like
> Emacs, but they are still better than nothing.
>
> Either way, I think it is preferable to have non-upstream tests
> of whatever shape we pick to be separate from the ebuilds and the
> source files, for the simple reason that most users won't care
> about them. Those who do can retrieve them in the same manner as
> the ATs.
>
> And as for my pet peeve, tests that are known to fail, can we
> also annotate that somehow so I don't waste hours running a test
> suite that gives zero signal on whether I should add the stable
> keyword? Even a one-line hin in the stabilization request would
> be nice. As it is, I keep a list of known-to-fail packages and my
> testing machinery tells me to not bother with FEATURES=test in
> those case.
>
> Not ideal, but less time-wasty.
>
> Regards,
> Tobias
>
> [0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans
>
>
> --
> if (user_specified)
>     /* Didn't work, but the user is convinced this is the
>      * place. */
>         linux-2.4.0-test2/drivers/parport/parport_pc.c
>
>