[gentoo-dev] Continuous Stabilization (Regarding Post Install Test Scripts)

[Pallav Agarwal <pallavagarwal07@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3368 bytes --]

This is a continuation of "[gentoo-dev] Proposal for changes for the next
EAPI
version" thread. Since the former subject isn't very relevant anymore.

Target:
A place to provide scripts that can be used by an automated arch-tester bot
to
stabilize Gentoo packages. These scripts can be as simple as a one liner

`python -c "import module"`

for example in case of a python module. Or a little complex, for example to
test
different features in case of different USE flags.
The main target is to alleviate the problems and unreliability from ARCH
Testing
and to more thoroughly and automatically test packages.

1 - Some important factors to consider from the previous thread:
a) This must be run POST-MERGE so that problems after merging can be
detected
b) It should be hopefully scalable in both directions - similar tests for
multiple packages
 - different tests for different versions of same package.
c) If a users want, they should be able to run these tests too (If they do,
bug reports
collected will be very valuable.
d) Most common opinion was to not have an EAPI change unless this gains
enough
use or demand.
e) If implemented as a part of ebuild, it can have all benefits of an
ebuild syntax.
Different versions of a package can have different test scripts. Test code
part can
have its own conditional runtime dependencies (only to take effect IF
testing is on).
Also, it can have specific parts of test run or not dependent on USE flags
enabled.

2 - Some suggested solutions
a) The original proposed solution:
To have an extra phase ci_test() or something similar that would be run
ONLY when
emerge is called with ci-test option. That way rest of the ebuild doesn't
have to change,
and for normal users, this wouldn't run unless they explicitly want it to.

PROS:
i) Full use of ebuild syntax
ii) Users can also use.
CONS:
i) Requires EAPI change
ii) Burdens the ebuild with more stuff
iii) Existing ebuilds need to be changed (one more revision per ebuild)

b) Having an extra global USE flag "ci_test" and putting test code in
pkg_postinst
phase under `if use ci_test` condition.

PROS:
i) Full use of ebuild syntax
ii) Users can enable ci-test too
iii) Changes to current ebuild standard is negligible
CONS:
i) Burdens the ebuild with more stuff
ii) Existing ebuilds need to be changed (one more revision per ebuild)

c) Creating a Gentoo Overlay with ebuilds which do test packages. And the
testing
script would be part of the pkg_postinst existing function.
For example, overlay could have a package dev-python/numpy-ci-test with:
```
DEPEND="dev-python/numpy"
pkg_postinstall() {
    python -c "import numpy; numpy.test()" || die "failed"
}
```
as it's content.

PROS:
i) No change to existing ebuilds or EAPI
ii) Can almost use all benefits of ebuild syntax
iii) Anybody can submit tests for ebuilds to the corresponding ebuild on
overlay
iv) Eclasses can define CI tests for multiple packages at once
CONS:
i) Users will almost never use an overlay just for testing
ii) Maintainer will have to submit one script for the portage tree and one
for the overlay.
Many maintainers would prefer to skip that.
iii) Ebuild names need to be different from the ones in portage tree, so
the process isn't
transparent anymore. We first need to check if -ci-test package exists or
not.


Any more suggestions/comments/methods are invited.

---
Regards,
Pallav

[-- Attachment #2: Type: text/html, Size: 5503 bytes --]