[gentoo-dev] Proposal for changes for the next EAPI version

[gentoo-dev] Proposal for changes for the next EAPI version

[Pallav Agarwal]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

Hi,
I am a student selected for GSoC 2016. One of the things in my proposal
requires the ebuilds to carry a mechanism to test the built software by
running some script provided by the maintainer of the ebuild.
This would be basically similar to whatever tests an Arch Tester would do,
however made easier by the fact that it would be written by the maintainer.
This could be to test whatever bug the ebuild fixes, or the basic
functionality of the software. This would help to automate the
stabilization of the built packages (which my project on
continuous stabilization wishes to accomplish).

What I'm suggesting is to add a new function post_install_test. The
function will run only if the build is being run for stabilization (either
as a part of automated stabilization, or manual) which can be controlled by
a USE flag. The function would also require independent dependencies in
case it uses external applications to test the one being built.

I wanted to know other opinions on this matter.

Thanks,
Pallav Agarwal

[-- Attachment #2: Type: text/html, Size: 1165 bytes --]

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Luis Ressel]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On Mon, 16 May 2016 18:13:33 +0530
Pallav Agarwal <pallavagarwal07@gmail.com> wrote:

> What I'm suggesting is to add a new function post_install_test. The
> function will run only if the build is being run for stabilization
> (either as a part of automated stabilization, or manual) which can be
> controlled by a USE flag. The function would also require independent
> dependencies in case it uses external applications to test the one
> being built.

Could you please elaborate on this? We already have src_test() for
automated tests. Why would we want to run additional tests after the
package has been merged?

-- 
Regards,
Luis Ressel

Luis Ressel <aranea@aixah.de>
GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Pallav Agarwal]

[-- Attachment #1: Type: text/plain, Size: 1943 bytes --]

For normal users we wouldn't. But currently, arch-testers need to make a
judgement call on what to test when a stable-req bug is filed. Tests run in
src_test are provided by upstream, and does not guarantee that a package
that has been merged will actually run on the system.
If the maintainer could add a couple small scripts to check basic
functionality
of the merged package, it would make testing for arch testers much easier
and reliable.
Let me give an example. Let's say
app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in
the portage tree.
However, on running, it produces an error on the top, along with the proper
output.
If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple
script

       if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then
             eerror "Still producing error"
       fi

To make sure the build is properly updating the screenfetch version, and
that
the bug has in fact been fixed. This is the only way I can see to reliabily
and automatically test packages that have been merged successfully.

-------
Regards,
Pallav

On Mon, May 16, 2016 at 10:08 PM, Luis Ressel <aranea@aixah.de> wrote:

> On Mon, 16 May 2016 18:13:33 +0530
> Pallav Agarwal <pallavagarwal07@gmail.com> wrote:
>
> > What I'm suggesting is to add a new function post_install_test. The
> > function will run only if the build is being run for stabilization
> > (either as a part of automated stabilization, or manual) which can be
> > controlled by a USE flag. The function would also require independent
> > dependencies in case it uses external applications to test the one
> > being built.
>
> Could you please elaborate on this? We already have src_test() for
> automated tests. Why would we want to run additional tests after the
> package has been merged?
>
> --
> Regards,
> Luis Ressel
>
> Luis Ressel <aranea@aixah.de>
> GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD
>

[-- Attachment #2: Type: text/html, Size: 2811 bytes --]

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Kent Fredric]

On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com> wrote:
> For normal users we wouldn't. But currently, arch-testers need to make a
> judgement call on what to test when a stable-req bug is filed. Tests run in
> src_test are provided by upstream, and does not guarantee that a package
> that has been merged will actually run on the system.
> If the maintainer could add a couple small scripts to check basic
> functionality
> of the merged package, it would make testing for arch testers much easier
> and reliable.
> Let me give an example. Let's say
> app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in
> the portage tree.
> However, on running, it produces an error on the top, along with the proper
> output.
> If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple
> script
>
>        if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then
>              eerror "Still producing error"
>        fi
>
> To make sure the build is properly updating the screenfetch version, and
> that
> the bug has in fact been fixed. This is the only way I can see to reliabily
> and automatically test packages that have been merged successfully


I don't think this needs to be an EAPI change. And if we can acheive
the goal without one, the better.

For instance, repoman is progressing in a way that repoman is
becomming pluggable, and that will hopefully be more useful.

Mostly, because it will allow us to define checks based on "inherit"
lines, and based on project/maintainer values in metadata.xml, in
theory.

You *could* theoretically augment this on an ebuild-by-ebuild basis,
but I would encourage doing as much as we can in this sphere outside
that workflow, because it will burden ebuilds with unhelpful stuff,
and it will increase commit churn on gentoo simply for QA reasons.

But the reality  is, you have to think not on ebuild-by-ebuild basis,
you have to think how this scales in volume, both against large
repositories of ebuilds, and how it scales in regards to
1-maintainer-thousands-of-ebuilds and 1-project-thousands-of-ebuilds
and 1-eclass-thousands-of-ebuilds scales.

For instance, there's very obvious applications for sets of gentoo
defined quality tests in dev-perl/, but anything we devise is going to
apply to all of them. _And_ to achieve our goal, we will likely need
external dependencies satisfied.

And I'd hate to be burdening those hundreds of ebuilds with that
overhead, its much better implemented out-of-band first, and then only
supplemented on an ebuild-by-ebuild basis.

And under this scheme, individual projects can define their custom
hooks and tricks in the ebuild themselves as metadata ( like we
currently to via eclass variables ), and be handled, not by portage,
or PMS, not by EAPI, but by the QA tool in conjunction with eclass
designers.


Then, no need for formal EAPI is required.

The QA tool just sources the ebuild in bash, hands the metadata to the
plugin, and the plugin can then execute whatever functions it wanted.

And the other benefit of doing this outside EAPI: It doesn't have to
be part of PMS, and maintainers of other packaging tools are not
compelled to implement a feature that is not relevant to end users.

And this also means our velocity with regards to changing what we do
here is much quicker, as we're no longer bound by the tight QA
controls we have to make sure we don't break users systems, because we
are *only* targeting devs.

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Tobias Klausmann]

Hi! 

On Tue, 17 May 2016, Kent Fredric wrote:

> On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com> wrote:
> > For normal users we wouldn't. But currently, arch-testers need to make a
> > judgement call on what to test when a stable-req bug is filed. Tests run in
> > src_test are provided by upstream, and does not guarantee that a package
> > that has been merged will actually run on the system.
> > If the maintainer could add a couple small scripts to check basic
> > functionality
> > of the merged package, it would make testing for arch testers much easier
> > and reliable.
> > Let me give an example. Let's say
> > app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in
> > the portage tree.
> > However, on running, it produces an error on the top, along with the proper
> > output.
> > If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple
> > script
> >
> >        if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then
> >              eerror "Still producing error"
> >        fi
> >
> > To make sure the build is properly updating the screenfetch version, and
> > that
> > the bug has in fact been fixed. This is the only way I can see to reliabily
> > and automatically test packages that have been merged successfully
> 
> 
> I don't think this needs to be an EAPI change. And if we can acheive
> the goal without one, the better.

Agreed. 

> [... lots of interesting stuff ...]

Or maintainers and teams could do what the Emacs team does:
provide test plans[0].

Naturally, I'd prefer something automated over the very hands-on
tests that are a bit of a necessity for an application like
Emacs, but they are still better than nothing.

Either way, I think it is preferable to have non-upstream tests
of whatever shape we pick to be separate from the ebuilds and the
source files, for the simple reason that most users won't care
about them. Those who do can retrieve them in the same manner as
the ATs.

And as for my pet peeve, tests that are known to fail, can we
also annotate that somehow so I don't waste hours running a test
suite that gives zero signal on whether I should add the stable
keyword? Even a one-line hin in the stabilization request would
be nice. As it is, I keep a list of known-to-fail packages and my
testing machinery tells me to not bother with FEATURES=test in
those case.

Not ideal, but less time-wasty.

Regards,
Tobias

[0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans


-- 
if (user_specified)
    /* Didn't work, but the user is convinced this is the
     * place. */
        linux-2.4.0-test2/drivers/parport/parport_pc.c

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Kent Fredric]

On 17 May 2016 at 20:46, Tobias Klausmann <klausman@gentoo.org> wrote:
> And as for my pet peeve, tests that are known to fail, can we
> also annotate that somehow so I don't waste hours running a test
> suite that gives zero signal on whether I should add the stable
> keyword? Even a one-line hin in the stabilization request would
> be nice. As it is, I keep a list of known-to-fail packages and my
> testing machinery tells me to not bother with FEATURES=test in
> those case.


IMO: Tests that are "expected to fail" should be killed.

You should either use RESTRICT=test to veto tests entirely ( which I
don't favour ), or more carefully
filter how the test suites get executed.

Tests that fail for non-reasons and are left in that state serve a
disservice to any package that has them, because it encourages people
to not run tests, and that encourages them not to see failures when
the tests identify *real* issues.

There's really no point in a test suite if "Failure is OK" is the
standard you're targeting.

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Pallav Agarwal]

[-- Attachment #1: Type: text/plain, Size: 3741 bytes --]

Hi,
You are right, of course.
The target is to standardize something that would encourage maintainers
to actually provide non-upstream scripts to test packages. At the same
time, it should be possible to use those scripts for automated testing
without
human interference.

Even if they are provided independently, not as part of ebuilds, we need a
to have them associated with a particular ebuild if need be as in case of a
bug fix, or many ebuilds in case of basic functionality for all versions of
a
package.
> And under this scheme, individual projects can define their custom
> hooks and tricks in the ebuild themselves as metadata ( like we
> currently to via eclass variables ), and be handled, not by portage,
> or PMS, not by EAPI, but by the QA tool in conjunction with eclass
> designers.
>
I'm not sure how to deal with eclasses but I guess I'll find out.

On Tue, May 17, 2016 at 2:16 PM, Tobias Klausmann <klausman@gentoo.org>
wrote:

> Hi!
>
> On Tue, 17 May 2016, Kent Fredric wrote:
>
> > On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com>
> wrote:
> > > For normal users we wouldn't. But currently, arch-testers need to make
> a
> > > judgement call on what to test when a stable-req bug is filed. Tests
> run in
> > > src_test are provided by upstream, and does not guarantee that a
> package
> > > that has been merged will actually run on the system.
> > > If the maintainer could add a couple small scripts to check basic
> > > functionality
> > > of the merged package, it would make testing for arch testers much
> easier
> > > and reliable.
> > > Let me give an example. Let's say
> > > app-misc/screenfetch-2.7.7 is the current stable package for
> screenfetch in
> > > the portage tree.
> > > However, on running, it produces an error on the top, along with the
> proper
> > > output.
> > > If screenfetch-3.0.0 happens to fix that error, maintainer can add a
> simple
> > > script
> > >
> > >        if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then
> > >              eerror "Still producing error"
> > >        fi
> > >
> > > To make sure the build is properly updating the screenfetch version,
> and
> > > that
> > > the bug has in fact been fixed. This is the only way I can see to
> reliabily
> > > and automatically test packages that have been merged successfully
> >
> >
> > I don't think this needs to be an EAPI change. And if we can acheive
> > the goal without one, the better.
>
> Agreed.
>
> > [... lots of interesting stuff ...]
>
> Or maintainers and teams could do what the Emacs team does:
> provide test plans[0].
>
> Naturally, I'd prefer something automated over the very hands-on
> tests that are a bit of a necessity for an application like
> Emacs, but they are still better than nothing.
>
> Either way, I think it is preferable to have non-upstream tests
> of whatever shape we pick to be separate from the ebuilds and the
> source files, for the simple reason that most users won't care
> about them. Those who do can retrieve them in the same manner as
> the ATs.
>
> And as for my pet peeve, tests that are known to fail, can we
> also annotate that somehow so I don't waste hours running a test
> suite that gives zero signal on whether I should add the stable
> keyword? Even a one-line hin in the stabilization request would
> be nice. As it is, I keep a list of known-to-fail packages and my
> testing machinery tells me to not bother with FEATURES=test in
> those case.
>
> Not ideal, but less time-wasty.
>
> Regards,
> Tobias
>
> [0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans
>
>
> --
> if (user_specified)
>     /* Didn't work, but the user is convinced this is the
>      * place. */
>         linux-2.4.0-test2/drivers/parport/parport_pc.c
>
>

[-- Attachment #2: Type: text/html, Size: 5034 bytes --]

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Rich Freeman]

On Tue, May 17, 2016 at 5:15 AM, Kent Fredric <kentfredric@gmail.com> wrote:
> On 17 May 2016 at 20:46, Tobias Klausmann <klausman@gentoo.org> wrote:
>> And as for my pet peeve, tests that are known to fail, can we
>> also annotate that somehow so I don't waste hours running a test
>> suite that gives zero signal on whether I should add the stable
>> keyword? Even a one-line hin in the stabilization request would
>> be nice. As it is, I keep a list of known-to-fail packages and my
>> testing machinery tells me to not bother with FEATURES=test in
>> those case.
>
> IMO: Tests that are "expected to fail" should be killed.
>

That makes sense, though ironically the only specific hypothetical use
case to come up so far was an example of just this situation.  A
package is broken in stable, and a test was proposed to detect if
future stable candidates fix the flaw.  There would be no point in
delaying stabilization of a package that contains the same error as
the current stable version.

I don't see any harm in adding support for automated Gentoo-specific
tests, but I am skeptical of how much use they'll actually get.  After
all, we started off with the statement that this is for situations
where upstream doesn't provide test suites, and if upstream can't be
bothered, why would we expect a distro maintainer to care more?

-- 
Rich

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Pallav Agarwal]

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

On Tue, May 17, 2016 at 4:27 PM, Rich Freeman <rich0@gentoo.org> wrote:

> On Tue, May 17, 2016 at 5:15 AM, Kent Fredric <kentfredric@gmail.com>
> wrote:
> > On 17 May 2016 at 20:46, Tobias Klausmann <klausman@gentoo.org> wrote:
> >> And as for my pet peeve, tests that are known to fail, can we
> >> also annotate that somehow so I don't waste hours running a test
> >> suite that gives zero signal on whether I should add the stable
> >> keyword? Even a one-line hin in the stabilization request would
> >> be nice. As it is, I keep a list of known-to-fail packages and my
> >> testing machinery tells me to not bother with FEATURES=test in
> >> those case.
> >
> > IMO: Tests that are "expected to fail" should be killed.
> >
>
> That makes sense, though ironically the only specific hypothetical use
> case to come up so far was an example of just this situation.  A
> package is broken in stable, and a test was proposed to detect if
> future stable candidates fix the flaw.  There would be no point in
> delaying stabilization of a package that contains the same error as
> the current stable version.
>
> I don't see any harm in adding support for automated Gentoo-specific
> tests, but I am skeptical of how much use they'll actually get.  After
> all, we started off with the statement that this is for situations
> where upstream doesn't provide test suites, and if upstream can't be
> bothered, why would we expect a distro maintainer to care more?
>
> --
> Rich
>
> Because we are already expecting an arch tester to conduct tests for the
package. And knowing what to test is something I expect to come more
easily from the maintainer.

[-- Attachment #2: Type: text/html, Size: 2425 bytes --]

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Michael Orlitzky]

On 05/17/2016 06:01 AM, Pallav Agarwal wrote:
> Hi,
> You are right, of course.
> The target is to standardize something that would encourage maintainers
> to actually provide non-upstream scripts to test packages. At the same
> time, it should be possible to use those scripts for automated testing
> without
> human interference.
> 

We already have "emerge --config" which is expected to be run after the
install process has completed, so I don't think that this is too much of
a stretch. Maybe call the phase "pkg_test" analogous to "pkg_config" and
in contrast to "src_test" which runs within the working directory. Then
"emerge --test" could run it.

I stole the Emacs "test plan" idea for a few Haskell packages:

  https://wiki.gentoo.org/wiki/Project:Haskell/Test_plans

These do go a little beyond the scope of unit tests: they check that we
installed the package in the right place, that it's available to your
ghci interpreter, and that we haven't installed some other library with
a name collision.

Some of them have predictable output (like when we hash a string), but
others might cause some trouble. The expected output for hscolour is
"you should see a syntax-highlighted version of Example.hs echoed to
your terminal." I don't care what colors show up, as long as that happens.

Still, even if it doesn't work in 100% of cases, I think it could be
useful. Some people may have inflated ideas of what the stabilization
process currently involves. In many cases, it's a compile/execute test
with no expected results and that could easily be automated even if it's
not the world's best stabilization process.

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Ciaran McCreesh]

On Tue, 17 May 2016 07:26:03 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:
> We already have "emerge --config" which is expected to be run after
> the install process has completed, so I don't think that this is too
> much of a stretch. Maybe call the phase "pkg_test" analogous to
> "pkg_config" and in contrast to "src_test" which runs within the
> working directory. Then "emerge --test" could run it.

For various technical reasons to do with the spec and package manglers
sometimes using phase function names with the src_ or pkg_ stripped
off, calling it that would be a huge pain.

-- 
Ciaran McCreesh

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Rich Freeman]

On Tue, May 17, 2016 at 7:25 AM, Pallav Agarwal
<pallavagarwal07@gmail.com> wrote:
> Because we are already expecting an arch tester to conduct tests for the
> package. And knowing what to test is something I expect to come more
> easily from the maintainer.
>

It would come even more easily from upstream.  My point is that
upstream obviously doesn't consider it a priority, since if they did
they'd be already supplying automated build scripts and we'd already
be using them in the test phase.  Maintainers generally don't consider
them a priority either, since I imagine many existing upstream build
scripts aren't implemented, and when they don't exist maintainers
generally do not provide test plans.

Again, I'm not objecting to the creation of this feature.  I just
think that it is unlikely to have a large impact on how things get
done except perhaps for a couple packages or projects.  For those
packages/projects that do adopt this I'm sure there will be some real
benefits.

I'd also suggest having some way to break out short vs long tests.  It
might be beneficial to users to be able to run the tests on their own
systems and not just limit this to arch testing.  Obviously users are
going to be less likely to want to run test scripts that take hours to
run.  Due to the nature of Gentoo it is unlikely that the environment
used for testing by the AT is identical to the environment the
packages will be run in.  I'm not negating the value of testing prior
to stabilization, but I don't agree that it completely takes the place
of testing at time of installation and if we have automated test
scripts, why not make them available to users?

Of course, that would suggest that there is a potential benefit to be
had from putting this into an EAPI.  My suggestion would be to try to
design this so that it would be appropriate for a future EAPI, but
defer considering it for an EAPI until we see the feature getting
enough use/demand to warrant this.

-- 
Rich

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[M.B.]

[-- Attachment #1: Type: text/plain, Size: 2483 bytes --]



Am 17.05.2016 um 09:37 schrieb Pallav Agarwal:
> For normal users we wouldn't. But currently, arch-testers need to make a
> judgement call on what to test when a stable-req bug is filed. Tests run in
> src_test are provided by upstream, and does not guarantee that a package
> that has been merged will actually run on the system.
> If the maintainer could add a couple small scripts to check basic
> functionality
> of the merged package, it would make testing for arch testers much easier
> and reliable.
> Let me give an example. Let's say
> app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in
> the portage tree.
> However, on running, it produces an error on the top, along with the proper
> output.
> If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple
> script
> 
>        if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then
>              eerror "Still producing error"
>        fi
> 
> To make sure the build is properly updating the screenfetch version, and
> that
> the bug has in fact been fixed. This is the only way I can see to reliabily
> and automatically test packages that have been merged successfully.
> 
> -------
> Regards,
> Pallav
> 
> On Mon, May 16, 2016 at 10:08 PM, Luis Ressel <aranea@aixah.de> wrote:
> 
>> On Mon, 16 May 2016 18:13:33 +0530
>> Pallav Agarwal <pallavagarwal07@gmail.com> wrote:
>>
>>> What I'm suggesting is to add a new function post_install_test. The
>>> function will run only if the build is being run for stabilization
>>> (either as a part of automated stabilization, or manual) which can be
>>> controlled by a USE flag. The function would also require independent
>>> dependencies in case it uses external applications to test the one
>>> being built.
>>
>> Could you please elaborate on this? We already have src_test() for
>> automated tests. Why would we want to run additional tests after the
>> package has been merged?
>>
>> --
>> Regards,
>> Luis Ressel
>>
>> Luis Ressel <aranea@aixah.de>
>> GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD
>>
> 

Good afternoon,

facilities for running post-install (pre-merge) QA-checks are in place.
Please have a look at portage's misc-functions.sh, install_qa_check()
will reveal the locations where you can find the installed checks, along
with a place for local overrides. Perhaps you can design something
around this?

With kind regards,
tomboy64


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 951 bytes --]

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2997 bytes --]

On Tue, 17 May 2016 15:53:34 +0200
"M.B." <tomboy64@sina.cn> wrote:

> Am 17.05.2016 um 09:37 schrieb Pallav Agarwal:
> > For normal users we wouldn't. But currently, arch-testers need to
> > make a judgement call on what to test when a stable-req bug is
> > filed. Tests run in src_test are provided by upstream, and does not
> > guarantee that a package that has been merged will actually run on
> > the system. If the maintainer could add a couple small scripts to
> > check basic functionality
> > of the merged package, it would make testing for arch testers much
> > easier and reliable.
> > Let me give an example. Let's say
> > app-misc/screenfetch-2.7.7 is the current stable package for
> > screenfetch in the portage tree.
> > However, on running, it produces an error on the top, along with
> > the proper output.
> > If screenfetch-3.0.0 happens to fix that error, maintainer can add
> > a simple script
> > 
> >        if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then
> >              eerror "Still producing error"
> >        fi
> > 
> > To make sure the build is properly updating the screenfetch
> > version, and that
> > the bug has in fact been fixed. This is the only way I can see to
> > reliabily and automatically test packages that have been merged
> > successfully.
> > 
> > -------
> > Regards,
> > Pallav
> > 
> > On Mon, May 16, 2016 at 10:08 PM, Luis Ressel <aranea@aixah.de>
> > wrote: 
> >> On Mon, 16 May 2016 18:13:33 +0530
> >> Pallav Agarwal <pallavagarwal07@gmail.com> wrote:
> >>  
> >>> What I'm suggesting is to add a new function post_install_test.
> >>> The function will run only if the build is being run for
> >>> stabilization (either as a part of automated stabilization, or
> >>> manual) which can be controlled by a USE flag. The function would
> >>> also require independent dependencies in case it uses external
> >>> applications to test the one being built.  
> >>
> >> Could you please elaborate on this? We already have src_test() for
> >> automated tests. Why would we want to run additional tests after
> >> the package has been merged?
> >>
> >> --
> >> Regards,
> >> Luis Ressel
> >>
> >> Luis Ressel <aranea@aixah.de>
> >> GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD
> >>  
> >   
> 
> Good afternoon,
> 
> facilities for running post-install (pre-merge) QA-checks are in
> place. Please have a look at portage's misc-functions.sh,
> install_qa_check() will reveal the locations where you can find the
> installed checks, along with a place for local overrides. Perhaps you
> can design something around this?
> 
> With kind regards,
> tomboy64
> 

These tests would be run post-merge, in the normal file system.  Mainly
for stabilization checks that can be automated, so not a QA-checks
qualifier which looks for common problems that can potentially lead to
bugs before it is merged to the normal file system. 

-- 
Brian Dolbec <dolsen>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 951 bytes --]

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Luis Ressel]

On Tue, 17 May 2016 13:07:43 +0530
Pallav Agarwal <pallavagarwal07@gmail.com> wrote:

> Tests run in src_test are provided by upstream, and does not
> guarantee that a package that has been merged will actually run on
> the system. If the maintainer could add a couple small scripts to
> check basic functionality of the merged package, it would make
> testing for arch testers much easier and reliable.

Automated post-merge tests sound kinda dangerous to me. And I don't
think there's any stipulation about src_test() only running
upstream-provided test suites. IMHO, src_test() would be a good place
for most of the maintainter-provided tests you have in mind.

Of course, there are some possible tests for which the src_test()
environment isn't suitable (because they're either interactive or
really need to run post-merge), I just don't expect there'd be many of
them. Therefore, I think we'd be better off providing such tests
out-of-band (test plans in the wiki), or perhaps stuffing them into
pkg_config().

Don't get me wrong, I'm not at all opposed to your idea of easing the
ATs' life, I'm just not convinced of the neccessity of EAPI changes. :)


-- 
Regards,
Luis Ressel

Luis Ressel <aranea@aixah.de>
GPG fpr: F08D 2AF6 655E 25DE 52BC  E53D 08F5 7F90 3029 B5BD

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Sébastien Fabbro]

On 17 May 2016 at 08:34, Luis Ressel <aranea@aixah.de> wrote:

>
> Automated post-merge tests sound kinda dangerous to me. And I don't
> think there's any stipulation about src_test() only running
> upstream-provided test suites. IMHO, src_test() would be a good place
> for most of the maintainter-provided tests you have in mind.
>

The idea of this project is not for a typical Gentoo user to run this
post-merge test, but for an automated arch-tester bot to run them in
batch jobs. The result of which would be an update of the ebuild to
stable.
Basically CI for ebuilds: it could be implemented as a script living
in the package directory, something like a .travis.yml in the GitHub
repositories or may be an EAPI change. Debian has a similar project
[1]. Upstream could provide CI tests and sometimes they do, but we
want to make sure the package integrates well in an installed Gentoo
distribution.

 [1] https://ci.debian.net/doc/

Sebastien

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Rich Freeman]

On Tue, May 17, 2016 at 12:05 PM, Sébastien Fabbro <bicatali@gentoo.org> wrote:
> On 17 May 2016 at 08:34, Luis Ressel <aranea@aixah.de> wrote:
>
>>
>> Automated post-merge tests sound kinda dangerous to me. And I don't
>> think there's any stipulation about src_test() only running
>> upstream-provided test suites. IMHO, src_test() would be a good place
>> for most of the maintainter-provided tests you have in mind.
>>
>
> The idea of this project is not for a typical Gentoo user to run this
> post-merge test, but for an automated arch-tester bot to run them in
> batch jobs. The result of which would be an update of the ebuild to
> stable.

I would suggest that it would make sense to separate tests that are
unsafe for a typical user to run from those which are, to allow users
to run them if they so choose.  As I said earlier, arch testing has
its limitations and I think many would like to be able to run the
automated tests routinely.  Since they're standardized they'd also
provide really good feedback to maintainers when they do have errors
missed by ATs.

-- 
Rich

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Kent Fredric]

On 18 May 2016 at 04:05, Sébastien Fabbro <bicatali@gentoo.org> wrote:
> Basically CI for ebuilds: it could be implemented as a script living
> in the package directory, something like a .travis.yml in the GitHub
> repositories or may be an EAPI change. Debian has a similar project
> [1]. Upstream could provide CI tests and sometimes they do, but we
> want to make sure the package integrates well in an installed Gentoo
> distribution.


Something like $CAT/$PN/maintainers/tests/<*>

or something like that I could live with, the idea being to keep as
much of this content *out* of the main ebuild flow as possible.

I'd much rather however not to require files in $CAT/$PN to be
changed, but to have a schema for code that can be run conditionally
on any suitable package via matching ( CAT, PN, inherit, project=,
maintainer= ) properties.

Mostly, because there are a lot of places where you'd never want to
implement the logic for a single package, you'd want to employ it for
a whole bunch.

Unfortunately, at present, the *sorts* of logic I personally see
myself implementing would require additional dependencies to perform.

This is why we're not already doing it in src_test(), because it would
expand the dependency graph for end users without benefit, ( and in
the way I'm thinking, results in risking a circular dependency, as
some of the tests I'm wanting to do would require Perl modules
installed, but these tests are to check things about Perl modules,
which risks requiring itself to validate itself ...., and exposing
users to that is inexcusable )

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[M. J. Everitt]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On 18/05/16 01:14, Kent Fredric wrote:
> On 18 May 2016 at 04:05, Sébastien Fabbro <bicatali@gentoo.org> wrote:
>> Basically CI for ebuilds: it could be implemented as a script living
>> in the package directory, something like a .travis.yml in the GitHub
>> repositories or may be an EAPI change. Debian has a similar project
>> [1]. Upstream could provide CI tests and sometimes they do, but we
>> want to make sure the package integrates well in an installed Gentoo
>> distribution.
>
> Something like $CAT/$PN/maintainers/tests/<*>
>
> or something like that I could live with, the idea being to keep as
> much of this content *out* of the main ebuild flow as possible.
>
> I'd much rather however not to require files in $CAT/$PN to be
> changed, but to have a schema for code that can be run conditionally
> on any suitable package via matching ( CAT, PN, inherit, project=,
> maintainer= ) properties.
>
> Mostly, because there are a lot of places where you'd never want to
> implement the logic for a single package, you'd want to employ it for
> a whole bunch.
>
> Unfortunately, at present, the *sorts* of logic I personally see
> myself implementing would require additional dependencies to perform.
>
> This is why we're not already doing it in src_test(), because it would
> expand the dependency graph for end users without benefit, ( and in
> the way I'm thinking, results in risking a circular dependency, as
> some of the tests I'm wanting to do would require Perl modules
> installed, but these tests are to check things about Perl modules,
> which risks requiring itself to validate itself ...., and exposing
> users to that is inexcusable )
>
Yes, whilst that's a special case, it would be desirable to collaborate
with another maintainer/team/project to devise a test schedule that was
independent from the target language, if possible. But there will always
be exceptions and issues and such with these things .. :/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[Kent Fredric]

On 18 May 2016 at 12:35, M. J. Everitt <m.j.everitt@iee.org> wrote:
> Yes, whilst that's a special case, it would be desirable to collaborate
> with another maintainer/team/project to devise a test schedule that was
> independent from the target language, if possible. But there will always
> be exceptions and issues and such with these things .. :/


In some of these cases, the things I'd be testing have to rely on Perl
Modules *because* it has to track how those specific modules respond
to the code in question.

For instance, to check we're doing our version normalisation
correctly, we have to use the upstream reference version of Perl's
version handling code directly.

*NOT* doing this results in significant problems, both in our failure
to perfectly map upstreams implementation in a different language, and
in our ability to keep our implementation in consistency with upstream
changes.

And we have already suffered this problem specifically in euscan,
where the euscan project implemented the version interpretation logic
manually, and did so hilariously wrong, and as a result, thinks newer
versions are older versions a lot of the time, and vice versa. I've
attempted my own implementation of upstreams logic *better* than I
think euscan does it, but I'm trapped in the reality where I have *no*
objective way of knowing if it in fact, represents upstreams logic
correctly.

The simplest thing to say here is "implementing it in a non-target
language is often enough the wrong choice".

Similarly, I've made the mistake of trying to understand and interpret
ebuilds statically without using bash .... that's a road to nowhere.
Even using bash is a bit tortured because I can't understand how an
ebuild works without reimplementing all the EAPI parts in bash or
relying on some portage version of the same ( which is extremely not
easy to use outside of the portage tools ).

-- 
Kent

KENTNL - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] Proposal for changes for the next EAPI version

[M. J. Everitt]

[-- Attachment #1: Type: text/plain, Size: 2335 bytes --]

On 18/05/16 01:44, Kent Fredric wrote:
> On 18 May 2016 at 12:35, M. J. Everitt <m.j.everitt@iee.org> wrote:
>> Yes, whilst that's a special case, it would be desirable to collaborate
>> with another maintainer/team/project to devise a test schedule that was
>> independent from the target language, if possible. But there will always
>> be exceptions and issues and such with these things .. :/
>
> In some of these cases, the things I'd be testing have to rely on Perl
> Modules *because* it has to track how those specific modules respond
> to the code in question.
>
> For instance, to check we're doing our version normalisation
> correctly, we have to use the upstream reference version of Perl's
> version handling code directly.
>
> *NOT* doing this results in significant problems, both in our failure
> to perfectly map upstreams implementation in a different language, and
> in our ability to keep our implementation in consistency with upstream
> changes.
>
> And we have already suffered this problem specifically in euscan,
> where the euscan project implemented the version interpretation logic
> manually, and did so hilariously wrong, and as a result, thinks newer
> versions are older versions a lot of the time, and vice versa. I've
> attempted my own implementation of upstreams logic *better* than I
> think euscan does it, but I'm trapped in the reality where I have *no*
> objective way of knowing if it in fact, represents upstreams logic
> correctly.
>
> The simplest thing to say here is "implementing it in a non-target
> language is often enough the wrong choice".
>
> Similarly, I've made the mistake of trying to understand and interpret
> ebuilds statically without using bash .... that's a road to nowhere.
> Even using bash is a bit tortured because I can't understand how an
> ebuild works without reimplementing all the EAPI parts in bash or
> relying on some portage version of the same ( which is extremely not
> easy to use outside of the portage tools ).
>
Yes, I get where you're coming from. I think many of the language and
language-plugin ebuilds are going to suffer from similar problems for
exactly the reasons you describe. It does make the prospect of a good,
over-arching QA/CI system quite challenging (but not impossible) to
achieve .. !!


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

[gentoo-dev] Re: Proposal for changes for the next EAPI version

[Duncan]

Ciaran McCreesh posted on Tue, 17 May 2016 12:29:36 +0100 as excerpted:

> On Tue, 17 May 2016 07:26:03 -0400 Michael Orlitzky <mjo@gentoo.org>
> wrote:
>> We already have "emerge --config" which is expected to be run after the
>> install process has completed, so I don't think that this is too much
>> of a stretch. Maybe call the phase "pkg_test" analogous to "pkg_config"
>> and in contrast to "src_test" which runs within the working directory.
>> Then "emerge --test" could run it.
> 
> For various technical reasons to do with the spec and package manglers
> sometimes using phase function names with the src_ or pkg_ stripped off,
> calling it that would be a huge pain.

What about something like gentoo_test, or more generically, distro_test?

That would also emphasize the difference between distro-level 
functionality and integration testing (which if I'm reading correctly is 
the proposal here) and upstream-level testing, via src_test.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman