[gentoo-dev] Continuous Stabilization (Regarding Post Install Test Scripts)

[gentoo-dev] Continuous Stabilization (Regarding Post Install Test Scripts)

[Pallav Agarwal]

[-- Attachment #1: Type: text/plain, Size: 3368 bytes --]

This is a continuation of "[gentoo-dev] Proposal for changes for the next
EAPI
version" thread. Since the former subject isn't very relevant anymore.

Target:
A place to provide scripts that can be used by an automated arch-tester bot
to
stabilize Gentoo packages. These scripts can be as simple as a one liner

`python -c "import module"`

for example in case of a python module. Or a little complex, for example to
test
different features in case of different USE flags.
The main target is to alleviate the problems and unreliability from ARCH
Testing
and to more thoroughly and automatically test packages.

1 - Some important factors to consider from the previous thread:
a) This must be run POST-MERGE so that problems after merging can be
detected
b) It should be hopefully scalable in both directions - similar tests for
multiple packages
 - different tests for different versions of same package.
c) If a users want, they should be able to run these tests too (If they do,
bug reports
collected will be very valuable.
d) Most common opinion was to not have an EAPI change unless this gains
enough
use or demand.
e) If implemented as a part of ebuild, it can have all benefits of an
ebuild syntax.
Different versions of a package can have different test scripts. Test code
part can
have its own conditional runtime dependencies (only to take effect IF
testing is on).
Also, it can have specific parts of test run or not dependent on USE flags
enabled.

2 - Some suggested solutions
a) The original proposed solution:
To have an extra phase ci_test() or something similar that would be run
ONLY when
emerge is called with ci-test option. That way rest of the ebuild doesn't
have to change,
and for normal users, this wouldn't run unless they explicitly want it to.

PROS:
i) Full use of ebuild syntax
ii) Users can also use.
CONS:
i) Requires EAPI change
ii) Burdens the ebuild with more stuff
iii) Existing ebuilds need to be changed (one more revision per ebuild)

b) Having an extra global USE flag "ci_test" and putting test code in
pkg_postinst
phase under `if use ci_test` condition.

PROS:
i) Full use of ebuild syntax
ii) Users can enable ci-test too
iii) Changes to current ebuild standard is negligible
CONS:
i) Burdens the ebuild with more stuff
ii) Existing ebuilds need to be changed (one more revision per ebuild)

c) Creating a Gentoo Overlay with ebuilds which do test packages. And the
testing
script would be part of the pkg_postinst existing function.
For example, overlay could have a package dev-python/numpy-ci-test with:
```
DEPEND="dev-python/numpy"
pkg_postinstall() {
    python -c "import numpy; numpy.test()" || die "failed"
}
```
as it's content.

PROS:
i) No change to existing ebuilds or EAPI
ii) Can almost use all benefits of ebuild syntax
iii) Anybody can submit tests for ebuilds to the corresponding ebuild on
overlay
iv) Eclasses can define CI tests for multiple packages at once
CONS:
i) Users will almost never use an overlay just for testing
ii) Maintainer will have to submit one script for the portage tree and one
for the overlay.
Many maintainers would prefer to skip that.
iii) Ebuild names need to be different from the ones in portage tree, so
the process isn't
transparent anymore. We first need to check if -ci-test package exists or
not.


Any more suggestions/comments/methods are invited.

---
Regards,
Pallav

[-- Attachment #2: Type: text/html, Size: 5503 bytes --]

Re: [gentoo-dev] Continuous Stabilization (Regarding Post Install Test Scripts)

[Harald Alfred Weiner]

[-- Attachment #1: Type: text/plain, Size: 4349 bytes --]

Dear Pallav,

how about using a dedicated eclass for arch-testing?

E.g., you might provide a function in some eclass
that only runs when called from pkg_postinst [1]
phase AND when some configuration option is set.

Then a maintainer / arch tester might provide
the tests inside the ebuild directory, e.g.,
inside a hidden sub-directory .arch-tests or
something like that, include your eclass inside
the ebuild and call your test-function from pkg_postinst
or something like that...

[1]
https://devmanual.gentoo.org/ebuild-writing/functions/pkg_postinst/index.html


Best wishes,


Harald.

Harald Weiner
Institute of Applied Geometry

Johannes-Kepler-Universität
Altenberger Straße 69
A-4040 Linz
harald.weiner@jku.at
http://www.ag.jku.at

On 05/19/2016 09:45 AM, Pallav Agarwal wrote:
> This is a continuation of "[gentoo-dev] Proposal for changes for the
> next EAPI
> version" thread. Since the former subject isn't very relevant anymore.
>
> Target:
> A place to provide scripts that can be used by an automated
> arch-tester bot to
> stabilize Gentoo packages. These scripts can be as simple as a one liner
>
> `python -c "import module"`
>
> for example in case of a python module. Or a little complex, for
> example to test
> different features in case of different USE flags.
> The main target is to alleviate the problems and unreliability from
> ARCH Testing
> and to more thoroughly and automatically test packages.
>
> 1 - Some important factors to consider from the previous thread:
> a) This must be run POST-MERGE so that problems after merging can be
> detected
> b) It should be hopefully scalable in both directions - similar tests
> for multiple packages
>  - different tests for different versions of same package.
> c) If a users want, they should be able to run these tests too (If
> they do, bug reports
> collected will be very valuable.
> d) Most common opinion was to not have an EAPI change unless this
> gains enough
> use or demand.
> e) If implemented as a part of ebuild, it can have all benefits of an
> ebuild syntax.
> Different versions of a package can have different test scripts. Test
> code part can
> have its own conditional runtime dependencies (only to take effect IF
> testing is on).
> Also, it can have specific parts of test run or not dependent on USE
> flags enabled.
>
> 2 - Some suggested solutions
> a) The original proposed solution:
> To have an extra phase ci_test() or something similar that would be
> run ONLY when
> emerge is called with ci-test option. That way rest of the ebuild
> doesn't have to change,
> and for normal users, this wouldn't run unless they explicitly want it to.
>
> PROS:
> i) Full use of ebuild syntax
> ii) Users can also use.
> CONS:
> i) Requires EAPI change
> ii) Burdens the ebuild with more stuff
> iii) Existing ebuilds need to be changed (one more revision per ebuild)
>
> b) Having an extra global USE flag "ci_test" and putting test code in
> pkg_postinst
> phase under `if use ci_test` condition.
>
> PROS:
> i) Full use of ebuild syntax
> ii) Users can enable ci-test too
> iii) Changes to current ebuild standard is negligible
> CONS:
> i) Burdens the ebuild with more stuff
> ii) Existing ebuilds need to be changed (one more revision per ebuild)
>
> c) Creating a Gentoo Overlay with ebuilds which do test packages. And
> the testing
> script would be part of the pkg_postinst existing function.
> For example, overlay could have a package dev-python/numpy-ci-test with:
> ```
> DEPEND="dev-python/numpy"
> pkg_postinstall() {
>     python -c "import numpy; numpy.test()" || die "failed"
> }
> ```
> as it's content.
>
> PROS:
> i) No change to existing ebuilds or EAPI
> ii) Can almost use all benefits of ebuild syntax
> iii) Anybody can submit tests for ebuilds to the corresponding ebuild
> on overlay
> iv) Eclasses can define CI tests for multiple packages at once
> CONS:
> i) Users will almost never use an overlay just for testing
> ii) Maintainer will have to submit one script for the portage tree and
> one for the overlay.
> Many maintainers would prefer to skip that.
> iii) Ebuild names need to be different from the ones in portage tree,
> so the process isn't
> transparent anymore. We first need to check if -ci-test package exists
> or not.
>
>
> Any more suggestions/comments/methods are invited.
>
> ---
> Regards,
> Pallav


[-- Attachment #2: Type: text/html, Size: 23506 bytes --]