Dear Pallav,

how about using a dedicated eclass for arch-testing?

E.g., you might provide a function in some eclass
that only runs when called from pkg_postinst [1]
phase AND when some configuration option is set.

Then a maintainer / arch tester might provide
the tests inside the ebuild directory, e.g.,
inside a hidden sub-directory .arch-tests or
something like that, include your eclass inside
the ebuild and call your test-function from pkg_postinst
or something like that...

[1]
https://devmanual.gentoo.org/ebuild-writing/functions/pkg_postinst/index.html


Best wishes,


Harald.

Harald Weiner
Institute of Applied Geometry

Johannes-Kepler-Universität
Altenberger Straße 69
A-4040 Linz
harald.weiner@jku.at
http://www.ag.jku.at

On 05/19/2016 09:45 AM, Pallav Agarwal wrote:
> This is a continuation of "[gentoo-dev] Proposal for changes for the
> next EAPI
> version" thread. Since the former subject isn't very relevant anymore.
>
> Target:
> A place to provide scripts that can be used by an automated
> arch-tester bot to
> stabilize Gentoo packages. These scripts can be as simple as a one liner
>
> `python -c "import module"`
>
> for example in case of a python module. Or a little complex, for
> example to test
> different features in case of different USE flags.
> The main target is to alleviate the problems and unreliability from
> ARCH Testing
> and to more thoroughly and automatically test packages.
>
> 1 - Some important factors to consider from the previous thread:
> a) This must be run POST-MERGE so that problems after merging can be
> detected
> b) It should be hopefully scalable in both directions - similar tests
> for multiple packages
>  - different tests for different versions of same package.
> c) If a users want, they should be able to run these tests too (If
> they do, bug reports
> collected will be very valuable.
> d) Most common opinion was to not have an EAPI change unless this
> gains enough
> use or demand.
> e) If implemented as a part of ebuild, it can have all benefits of an
> ebuild syntax.
> Different versions of a package can have different test scripts. Test
> code part can
> have its own conditional runtime dependencies (only to take effect IF
> testing is on).
> Also, it can have specific parts of test run or not dependent on USE
> flags enabled.
>
> 2 - Some suggested solutions
> a) The original proposed solution:
> To have an extra phase ci_test() or something similar that would be
> run ONLY when
> emerge is called with ci-test option. That way rest of the ebuild
> doesn't have to change,
> and for normal users, this wouldn't run unless they explicitly want it to.
>
> PROS:
> i) Full use of ebuild syntax
> ii) Users can also use.
> CONS:
> i) Requires EAPI change
> ii) Burdens the ebuild with more stuff
> iii) Existing ebuilds need to be changed (one more revision per ebuild)
>
> b) Having an extra global USE flag "ci_test" and putting test code in
> pkg_postinst
> phase under `if use ci_test` condition.
>
> PROS:
> i) Full use of ebuild syntax
> ii) Users can enable ci-test too
> iii) Changes to current ebuild standard is negligible
> CONS:
> i) Burdens the ebuild with more stuff
> ii) Existing ebuilds need to be changed (one more revision per ebuild)
>
> c) Creating a Gentoo Overlay with ebuilds which do test packages. And
> the testing
> script would be part of the pkg_postinst existing function.
> For example, overlay could have a package dev-python/numpy-ci-test with:
> ```
> DEPEND="dev-python/numpy"
> pkg_postinstall() {
>     python -c "import numpy; numpy.test()" || die "failed"
> }
> ```
> as it's content.
>
> PROS:
> i) No change to existing ebuilds or EAPI
> ii) Can almost use all benefits of ebuild syntax
> iii) Anybody can submit tests for ebuilds to the corresponding ebuild
> on overlay
> iv) Eclasses can define CI tests for multiple packages at once
> CONS:
> i) Users will almost never use an overlay just for testing
> ii) Maintainer will have to submit one script for the portage tree and
> one for the overlay.
> Many maintainers would prefer to skip that.
> iii) Ebuild names need to be different from the ones in portage tree,
> so the process isn't
> transparent anymore. We first need to check if -ci-test package exists
> or not.
>
>
> Any more suggestions/comments/methods are invited.
>
> ---
> Regards,
> Pallav