From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 165AE138262 for ; Thu, 19 May 2016 08:16:35 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 71EA31412D; Thu, 19 May 2016 08:16:27 +0000 (UTC) Received: from emailsecure.uni-linz.ac.at (emailsecure.uni-linz.ac.at [140.78.3.66]) by pigeon.gentoo.org (Postfix) with ESMTP id E018A21C038 for ; Thu, 19 May 2016 08:16:25 +0000 (UTC) Received: from [140.78.116.82] (unknown [140.78.116.82]) by emailsecure.uni-linz.ac.at (Postfix) with ESMTPSA id AE996FEB95 for ; Thu, 19 May 2016 10:16:24 +0200 (CEST) Subject: Re: [gentoo-dev] Continuous Stabilization (Regarding Post Install Test Scripts) To: gentoo-dev@lists.gentoo.org References: From: Harald Alfred Weiner X-Enigmail-Draft-Status: N1110 Message-ID: <573D7658.8040602@jku.at> Date: Thu, 19 May 2016 10:16:24 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------090005020408010206080202" X-Archives-Salt: 3daf7319-f4e5-4338-bd1c-f3e8359d80e9 X-Archives-Hash: a18fcbf5bf086dbc0761593d46b0f416 This is a multi-part message in MIME format. --------------090005020408010206080202 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Dear Pallav, how about using a dedicated eclass for arch-testing? E.g., you might provide a function in some eclass that only runs when called from pkg_postinst [1] phase AND when some configuration option is set. Then a maintainer / arch tester might provide the tests inside the ebuild directory, e.g., inside a hidden sub-directory .arch-tests or something like that, include your eclass inside the ebuild and call your test-function from pkg_postinst or something like that... [1] https://devmanual.gentoo.org/ebuild-writing/functions/pkg_postinst/index.html Best wishes, Harald. Harald Weiner Institute of Applied Geometry Johannes-Kepler-Universität Altenberger Straße 69 A-4040 Linz harald.weiner@jku.at http://www.ag.jku.at On 05/19/2016 09:45 AM, Pallav Agarwal wrote: > This is a continuation of "[gentoo-dev] Proposal for changes for the > next EAPI > version" thread. Since the former subject isn't very relevant anymore. > > Target: > A place to provide scripts that can be used by an automated > arch-tester bot to > stabilize Gentoo packages. These scripts can be as simple as a one liner > > `python -c "import module"` > > for example in case of a python module. Or a little complex, for > example to test > different features in case of different USE flags. > The main target is to alleviate the problems and unreliability from > ARCH Testing > and to more thoroughly and automatically test packages. > > 1 - Some important factors to consider from the previous thread: > a) This must be run POST-MERGE so that problems after merging can be > detected > b) It should be hopefully scalable in both directions - similar tests > for multiple packages > - different tests for different versions of same package. > c) If a users want, they should be able to run these tests too (If > they do, bug reports > collected will be very valuable. > d) Most common opinion was to not have an EAPI change unless this > gains enough > use or demand. > e) If implemented as a part of ebuild, it can have all benefits of an > ebuild syntax. > Different versions of a package can have different test scripts. Test > code part can > have its own conditional runtime dependencies (only to take effect IF > testing is on). > Also, it can have specific parts of test run or not dependent on USE > flags enabled. > > 2 - Some suggested solutions > a) The original proposed solution: > To have an extra phase ci_test() or something similar that would be > run ONLY when > emerge is called with ci-test option. That way rest of the ebuild > doesn't have to change, > and for normal users, this wouldn't run unless they explicitly want it to. > > PROS: > i) Full use of ebuild syntax > ii) Users can also use. > CONS: > i) Requires EAPI change > ii) Burdens the ebuild with more stuff > iii) Existing ebuilds need to be changed (one more revision per ebuild) > > b) Having an extra global USE flag "ci_test" and putting test code in > pkg_postinst > phase under `if use ci_test` condition. > > PROS: > i) Full use of ebuild syntax > ii) Users can enable ci-test too > iii) Changes to current ebuild standard is negligible > CONS: > i) Burdens the ebuild with more stuff > ii) Existing ebuilds need to be changed (one more revision per ebuild) > > c) Creating a Gentoo Overlay with ebuilds which do test packages. And > the testing > script would be part of the pkg_postinst existing function. > For example, overlay could have a package dev-python/numpy-ci-test with: > ``` > DEPEND="dev-python/numpy" > pkg_postinstall() { > python -c "import numpy; numpy.test()" || die "failed" > } > ``` > as it's content. > > PROS: > i) No change to existing ebuilds or EAPI > ii) Can almost use all benefits of ebuild syntax > iii) Anybody can submit tests for ebuilds to the corresponding ebuild > on overlay > iv) Eclasses can define CI tests for multiple packages at once > CONS: > i) Users will almost never use an overlay just for testing > ii) Maintainer will have to submit one script for the portage tree and > one for the overlay. > Many maintainers would prefer to skip that. > iii) Ebuild names need to be different from the ones in portage tree, > so the process isn't > transparent anymore. We first need to check if -ci-test package exists > or not. > > > Any more suggestions/comments/methods are invited. > > --- > Regards, > Pallav --------------090005020408010206080202 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Dear Pallav,

how about using a dedicated eclass for arch-testing?

E.g., you might provide a function in some eclass
that only runs when called from pkg_postinst [1]
phase AND when some configuration option is set.

Then a maintainer / arch tester might provide
the tests inside the ebuild directory, e.g.,
inside a hidden sub-directory .arch-tests or
something like that, include your eclass inside
the ebuild and call your test-function from pkg_postinst
or something like that...

[1] https://devmanual.gentoo.org/ebuild-writing/functions/pkg_postinst/index.html


Best wishes,


Harald.
Harald Weiner
Institute of Applied Geometry

Johannes-Kepler-Universität
Altenberger Straße 69
A-4040 Linz
harald.weiner@jku.at
http://www.ag.jku.at
On 05/19/2016 09:45 AM, Pallav Agarwal wrote:
This is a continuation of "[gentoo-dev] Proposal for changes for the next EAPI
version" thread. Since the former subject isn't very relevant anymore.

Target:
A place to provide scripts that can be used by an automated arch-tester bot to
stabilize Gentoo packages. These scripts can be as simple as a one liner

`python -c "import module"`

for example in case of a python module. Or a little complex, for example to test
different features in case of different USE flags.
The main target is to alleviate the problems and unreliability from ARCH Testing
and to more thoroughly and automatically test packages.

1 - Some important factors to consider from the previous thread:
a) This must be run POST-MERGE so that problems after merging can be detected
b) It should be hopefully scalable in both directions - similar tests for multiple packages
 - different tests for different versions of same package.
c) If a users want, they should be able to run these tests too (If they do, bug reports
collected will be very valuable.
d) Most common opinion was to not have an EAPI change unless this gains enough
use or demand.
e) If implemented as a part of ebuild, it can have all benefits of an ebuild syntax.
Different versions of a package can have different test scripts. Test code part can
have its own conditional runtime dependencies (only to take effect IF testing is on).
Also, it can have specific parts of test run or not dependent on USE flags enabled.

2 - Some suggested solutions
a) The original proposed solution:
To have an extra phase ci_test() or something similar that would be run ONLY when
emerge is called with ci-test option. That way rest of the ebuild doesn't have to change,
and for normal users, this wouldn't run unless they explicitly want it to.

PROS:
i) Full use of ebuild syntax
ii) Users can also use.
CONS:
i) Requires EAPI change
ii) Burdens the ebuild with more stuff
iii) Existing ebuilds need to be changed (one more revision per ebuild)

b) Having an extra global USE flag "ci_test" and putting test code in pkg_postinst
phase under `if use ci_test` condition.

PROS:
i) Full use of ebuild syntax
ii) Users can enable ci-test too
iii) Changes to current ebuild standard is negligible
CONS:
i) Burdens the ebuild with more stuff
ii) Existing ebuilds need to be changed (one more revision per ebuild)

c) Creating a Gentoo Overlay with ebuilds which do test packages. And the testing
script would be part of the pkg_postinst existing function.
For example, overlay could have a package dev-python/numpy-ci-test with:
```
DEPEND="dev-python/numpy"
pkg_postinstall() {
    python -c "import numpy; numpy.test()" || die "failed"
}
```
as it's content.

PROS:
i) No change to existing ebuilds or EAPI
ii) Can almost use all benefits of ebuild syntax
iii) Anybody can submit tests for ebuilds to the corresponding ebuild on overlay
iv) Eclasses can define CI tests for multiple packages at once
CONS:
i) Users will almost never use an overlay just for testing
ii) Maintainer will have to submit one script for the portage tree and one for the overlay.
Many maintainers would prefer to skip that.
iii) Ebuild names need to be different from the ones in portage tree, so the process isn't
transparent anymore. We first need to check if -ci-test package exists or not.


Any more suggestions/comments/methods are invited.

---
Regards,
Pallav

--------------090005020408010206080202--