* [gentoo-dev] Proposal for changes for the next EAPI version @ 2016-05-16 12:43 Pallav Agarwal 2016-05-16 16:38 ` Luis Ressel 0 siblings, 1 reply; 22+ messages in thread From: Pallav Agarwal @ 2016-05-16 12:43 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1024 bytes --] Hi, I am a student selected for GSoC 2016. One of the things in my proposal requires the ebuilds to carry a mechanism to test the built software by running some script provided by the maintainer of the ebuild. This would be basically similar to whatever tests an Arch Tester would do, however made easier by the fact that it would be written by the maintainer. This could be to test whatever bug the ebuild fixes, or the basic functionality of the software. This would help to automate the stabilization of the built packages (which my project on continuous stabilization wishes to accomplish). What I'm suggesting is to add a new function post_install_test. The function will run only if the build is being run for stabilization (either as a part of automated stabilization, or manual) which can be controlled by a USE flag. The function would also require independent dependencies in case it uses external applications to test the one being built. I wanted to know other opinions on this matter. Thanks, Pallav Agarwal [-- Attachment #2: Type: text/html, Size: 1165 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-16 12:43 [gentoo-dev] Proposal for changes for the next EAPI version Pallav Agarwal @ 2016-05-16 16:38 ` Luis Ressel 2016-05-17 7:37 ` Pallav Agarwal 0 siblings, 1 reply; 22+ messages in thread From: Luis Ressel @ 2016-05-16 16:38 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 752 bytes --] On Mon, 16 May 2016 18:13:33 +0530 Pallav Agarwal <pallavagarwal07@gmail.com> wrote: > What I'm suggesting is to add a new function post_install_test. The > function will run only if the build is being run for stabilization > (either as a part of automated stabilization, or manual) which can be > controlled by a USE flag. The function would also require independent > dependencies in case it uses external applications to test the one > being built. Could you please elaborate on this? We already have src_test() for automated tests. Why would we want to run additional tests after the package has been merged? -- Regards, Luis Ressel Luis Ressel <aranea@aixah.de> GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 949 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-16 16:38 ` Luis Ressel @ 2016-05-17 7:37 ` Pallav Agarwal 2016-05-17 8:02 ` Kent Fredric ` (2 more replies) 0 siblings, 3 replies; 22+ messages in thread From: Pallav Agarwal @ 2016-05-17 7:37 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1943 bytes --] For normal users we wouldn't. But currently, arch-testers need to make a judgement call on what to test when a stable-req bug is filed. Tests run in src_test are provided by upstream, and does not guarantee that a package that has been merged will actually run on the system. If the maintainer could add a couple small scripts to check basic functionality of the merged package, it would make testing for arch testers much easier and reliable. Let me give an example. Let's say app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in the portage tree. However, on running, it produces an error on the top, along with the proper output. If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple script if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then eerror "Still producing error" fi To make sure the build is properly updating the screenfetch version, and that the bug has in fact been fixed. This is the only way I can see to reliabily and automatically test packages that have been merged successfully. ------- Regards, Pallav On Mon, May 16, 2016 at 10:08 PM, Luis Ressel <aranea@aixah.de> wrote: > On Mon, 16 May 2016 18:13:33 +0530 > Pallav Agarwal <pallavagarwal07@gmail.com> wrote: > > > What I'm suggesting is to add a new function post_install_test. The > > function will run only if the build is being run for stabilization > > (either as a part of automated stabilization, or manual) which can be > > controlled by a USE flag. The function would also require independent > > dependencies in case it uses external applications to test the one > > being built. > > Could you please elaborate on this? We already have src_test() for > automated tests. Why would we want to run additional tests after the > package has been merged? > > -- > Regards, > Luis Ressel > > Luis Ressel <aranea@aixah.de> > GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD > [-- Attachment #2: Type: text/html, Size: 2811 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 7:37 ` Pallav Agarwal @ 2016-05-17 8:02 ` Kent Fredric 2016-05-17 8:46 ` Tobias Klausmann 2016-05-17 13:53 ` [gentoo-dev] " M.B. 2016-05-17 15:34 ` Luis Ressel 2 siblings, 1 reply; 22+ messages in thread From: Kent Fredric @ 2016-05-17 8:02 UTC (permalink / raw To: gentoo-dev On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com> wrote: > For normal users we wouldn't. But currently, arch-testers need to make a > judgement call on what to test when a stable-req bug is filed. Tests run in > src_test are provided by upstream, and does not guarantee that a package > that has been merged will actually run on the system. > If the maintainer could add a couple small scripts to check basic > functionality > of the merged package, it would make testing for arch testers much easier > and reliable. > Let me give an example. Let's say > app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in > the portage tree. > However, on running, it produces an error on the top, along with the proper > output. > If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple > script > > if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then > eerror "Still producing error" > fi > > To make sure the build is properly updating the screenfetch version, and > that > the bug has in fact been fixed. This is the only way I can see to reliabily > and automatically test packages that have been merged successfully I don't think this needs to be an EAPI change. And if we can acheive the goal without one, the better. For instance, repoman is progressing in a way that repoman is becomming pluggable, and that will hopefully be more useful. Mostly, because it will allow us to define checks based on "inherit" lines, and based on project/maintainer values in metadata.xml, in theory. You *could* theoretically augment this on an ebuild-by-ebuild basis, but I would encourage doing as much as we can in this sphere outside that workflow, because it will burden ebuilds with unhelpful stuff, and it will increase commit churn on gentoo simply for QA reasons. But the reality is, you have to think not on ebuild-by-ebuild basis, you have to think how this scales in volume, both against large repositories of ebuilds, and how it scales in regards to 1-maintainer-thousands-of-ebuilds and 1-project-thousands-of-ebuilds and 1-eclass-thousands-of-ebuilds scales. For instance, there's very obvious applications for sets of gentoo defined quality tests in dev-perl/, but anything we devise is going to apply to all of them. _And_ to achieve our goal, we will likely need external dependencies satisfied. And I'd hate to be burdening those hundreds of ebuilds with that overhead, its much better implemented out-of-band first, and then only supplemented on an ebuild-by-ebuild basis. And under this scheme, individual projects can define their custom hooks and tricks in the ebuild themselves as metadata ( like we currently to via eclass variables ), and be handled, not by portage, or PMS, not by EAPI, but by the QA tool in conjunction with eclass designers. Then, no need for formal EAPI is required. The QA tool just sources the ebuild in bash, hands the metadata to the plugin, and the plugin can then execute whatever functions it wanted. And the other benefit of doing this outside EAPI: It doesn't have to be part of PMS, and maintainers of other packaging tools are not compelled to implement a feature that is not relevant to end users. And this also means our velocity with regards to changing what we do here is much quicker, as we're no longer bound by the tight QA controls we have to make sure we don't break users systems, because we are *only* targeting devs. -- Kent KENTNL - https://metacpan.org/author/KENTNL ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 8:02 ` Kent Fredric @ 2016-05-17 8:46 ` Tobias Klausmann 2016-05-17 9:15 ` Kent Fredric 2016-05-17 10:01 ` Pallav Agarwal 0 siblings, 2 replies; 22+ messages in thread From: Tobias Klausmann @ 2016-05-17 8:46 UTC (permalink / raw To: gentoo-dev Hi! On Tue, 17 May 2016, Kent Fredric wrote: > On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com> wrote: > > For normal users we wouldn't. But currently, arch-testers need to make a > > judgement call on what to test when a stable-req bug is filed. Tests run in > > src_test are provided by upstream, and does not guarantee that a package > > that has been merged will actually run on the system. > > If the maintainer could add a couple small scripts to check basic > > functionality > > of the merged package, it would make testing for arch testers much easier > > and reliable. > > Let me give an example. Let's say > > app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in > > the portage tree. > > However, on running, it produces an error on the top, along with the proper > > output. > > If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple > > script > > > > if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then > > eerror "Still producing error" > > fi > > > > To make sure the build is properly updating the screenfetch version, and > > that > > the bug has in fact been fixed. This is the only way I can see to reliabily > > and automatically test packages that have been merged successfully > > > I don't think this needs to be an EAPI change. And if we can acheive > the goal without one, the better. Agreed. > [... lots of interesting stuff ...] Or maintainers and teams could do what the Emacs team does: provide test plans[0]. Naturally, I'd prefer something automated over the very hands-on tests that are a bit of a necessity for an application like Emacs, but they are still better than nothing. Either way, I think it is preferable to have non-upstream tests of whatever shape we pick to be separate from the ebuilds and the source files, for the simple reason that most users won't care about them. Those who do can retrieve them in the same manner as the ATs. And as for my pet peeve, tests that are known to fail, can we also annotate that somehow so I don't waste hours running a test suite that gives zero signal on whether I should add the stable keyword? Even a one-line hin in the stabilization request would be nice. As it is, I keep a list of known-to-fail packages and my testing machinery tells me to not bother with FEATURES=test in those case. Not ideal, but less time-wasty. Regards, Tobias [0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans -- if (user_specified) /* Didn't work, but the user is convinced this is the * place. */ linux-2.4.0-test2/drivers/parport/parport_pc.c ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 8:46 ` Tobias Klausmann @ 2016-05-17 9:15 ` Kent Fredric 2016-05-17 10:57 ` Rich Freeman 2016-05-17 10:01 ` Pallav Agarwal 1 sibling, 1 reply; 22+ messages in thread From: Kent Fredric @ 2016-05-17 9:15 UTC (permalink / raw To: gentoo-dev On 17 May 2016 at 20:46, Tobias Klausmann <klausman@gentoo.org> wrote: > And as for my pet peeve, tests that are known to fail, can we > also annotate that somehow so I don't waste hours running a test > suite that gives zero signal on whether I should add the stable > keyword? Even a one-line hin in the stabilization request would > be nice. As it is, I keep a list of known-to-fail packages and my > testing machinery tells me to not bother with FEATURES=test in > those case. IMO: Tests that are "expected to fail" should be killed. You should either use RESTRICT=test to veto tests entirely ( which I don't favour ), or more carefully filter how the test suites get executed. Tests that fail for non-reasons and are left in that state serve a disservice to any package that has them, because it encourages people to not run tests, and that encourages them not to see failures when the tests identify *real* issues. There's really no point in a test suite if "Failure is OK" is the standard you're targeting. -- Kent KENTNL - https://metacpan.org/author/KENTNL ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 9:15 ` Kent Fredric @ 2016-05-17 10:57 ` Rich Freeman 2016-05-17 11:25 ` Pallav Agarwal 0 siblings, 1 reply; 22+ messages in thread From: Rich Freeman @ 2016-05-17 10:57 UTC (permalink / raw To: gentoo-dev On Tue, May 17, 2016 at 5:15 AM, Kent Fredric <kentfredric@gmail.com> wrote: > On 17 May 2016 at 20:46, Tobias Klausmann <klausman@gentoo.org> wrote: >> And as for my pet peeve, tests that are known to fail, can we >> also annotate that somehow so I don't waste hours running a test >> suite that gives zero signal on whether I should add the stable >> keyword? Even a one-line hin in the stabilization request would >> be nice. As it is, I keep a list of known-to-fail packages and my >> testing machinery tells me to not bother with FEATURES=test in >> those case. > > IMO: Tests that are "expected to fail" should be killed. > That makes sense, though ironically the only specific hypothetical use case to come up so far was an example of just this situation. A package is broken in stable, and a test was proposed to detect if future stable candidates fix the flaw. There would be no point in delaying stabilization of a package that contains the same error as the current stable version. I don't see any harm in adding support for automated Gentoo-specific tests, but I am skeptical of how much use they'll actually get. After all, we started off with the statement that this is for situations where upstream doesn't provide test suites, and if upstream can't be bothered, why would we expect a distro maintainer to care more? -- Rich ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 10:57 ` Rich Freeman @ 2016-05-17 11:25 ` Pallav Agarwal 2016-05-17 11:42 ` Rich Freeman 0 siblings, 1 reply; 22+ messages in thread From: Pallav Agarwal @ 2016-05-17 11:25 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1648 bytes --] On Tue, May 17, 2016 at 4:27 PM, Rich Freeman <rich0@gentoo.org> wrote: > On Tue, May 17, 2016 at 5:15 AM, Kent Fredric <kentfredric@gmail.com> > wrote: > > On 17 May 2016 at 20:46, Tobias Klausmann <klausman@gentoo.org> wrote: > >> And as for my pet peeve, tests that are known to fail, can we > >> also annotate that somehow so I don't waste hours running a test > >> suite that gives zero signal on whether I should add the stable > >> keyword? Even a one-line hin in the stabilization request would > >> be nice. As it is, I keep a list of known-to-fail packages and my > >> testing machinery tells me to not bother with FEATURES=test in > >> those case. > > > > IMO: Tests that are "expected to fail" should be killed. > > > > That makes sense, though ironically the only specific hypothetical use > case to come up so far was an example of just this situation. A > package is broken in stable, and a test was proposed to detect if > future stable candidates fix the flaw. There would be no point in > delaying stabilization of a package that contains the same error as > the current stable version. > > I don't see any harm in adding support for automated Gentoo-specific > tests, but I am skeptical of how much use they'll actually get. After > all, we started off with the statement that this is for situations > where upstream doesn't provide test suites, and if upstream can't be > bothered, why would we expect a distro maintainer to care more? > > -- > Rich > > Because we are already expecting an arch tester to conduct tests for the package. And knowing what to test is something I expect to come more easily from the maintainer. [-- Attachment #2: Type: text/html, Size: 2425 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 11:25 ` Pallav Agarwal @ 2016-05-17 11:42 ` Rich Freeman 0 siblings, 0 replies; 22+ messages in thread From: Rich Freeman @ 2016-05-17 11:42 UTC (permalink / raw To: gentoo-dev On Tue, May 17, 2016 at 7:25 AM, Pallav Agarwal <pallavagarwal07@gmail.com> wrote: > Because we are already expecting an arch tester to conduct tests for the > package. And knowing what to test is something I expect to come more > easily from the maintainer. > It would come even more easily from upstream. My point is that upstream obviously doesn't consider it a priority, since if they did they'd be already supplying automated build scripts and we'd already be using them in the test phase. Maintainers generally don't consider them a priority either, since I imagine many existing upstream build scripts aren't implemented, and when they don't exist maintainers generally do not provide test plans. Again, I'm not objecting to the creation of this feature. I just think that it is unlikely to have a large impact on how things get done except perhaps for a couple packages or projects. For those packages/projects that do adopt this I'm sure there will be some real benefits. I'd also suggest having some way to break out short vs long tests. It might be beneficial to users to be able to run the tests on their own systems and not just limit this to arch testing. Obviously users are going to be less likely to want to run test scripts that take hours to run. Due to the nature of Gentoo it is unlikely that the environment used for testing by the AT is identical to the environment the packages will be run in. I'm not negating the value of testing prior to stabilization, but I don't agree that it completely takes the place of testing at time of installation and if we have automated test scripts, why not make them available to users? Of course, that would suggest that there is a potential benefit to be had from putting this into an EAPI. My suggestion would be to try to design this so that it would be appropriate for a future EAPI, but defer considering it for an EAPI until we see the feature getting enough use/demand to warrant this. -- Rich ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 8:46 ` Tobias Klausmann 2016-05-17 9:15 ` Kent Fredric @ 2016-05-17 10:01 ` Pallav Agarwal 2016-05-17 11:26 ` Michael Orlitzky 1 sibling, 1 reply; 22+ messages in thread From: Pallav Agarwal @ 2016-05-17 10:01 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 3741 bytes --] Hi, You are right, of course. The target is to standardize something that would encourage maintainers to actually provide non-upstream scripts to test packages. At the same time, it should be possible to use those scripts for automated testing without human interference. Even if they are provided independently, not as part of ebuilds, we need a to have them associated with a particular ebuild if need be as in case of a bug fix, or many ebuilds in case of basic functionality for all versions of a package. > And under this scheme, individual projects can define their custom > hooks and tricks in the ebuild themselves as metadata ( like we > currently to via eclass variables ), and be handled, not by portage, > or PMS, not by EAPI, but by the QA tool in conjunction with eclass > designers. > I'm not sure how to deal with eclasses but I guess I'll find out. On Tue, May 17, 2016 at 2:16 PM, Tobias Klausmann <klausman@gentoo.org> wrote: > Hi! > > On Tue, 17 May 2016, Kent Fredric wrote: > > > On 17 May 2016 at 19:37, Pallav Agarwal <pallavagarwal07@gmail.com> > wrote: > > > For normal users we wouldn't. But currently, arch-testers need to make > a > > > judgement call on what to test when a stable-req bug is filed. Tests > run in > > > src_test are provided by upstream, and does not guarantee that a > package > > > that has been merged will actually run on the system. > > > If the maintainer could add a couple small scripts to check basic > > > functionality > > > of the merged package, it would make testing for arch testers much > easier > > > and reliable. > > > Let me give an example. Let's say > > > app-misc/screenfetch-2.7.7 is the current stable package for > screenfetch in > > > the portage tree. > > > However, on running, it produces an error on the top, along with the > proper > > > output. > > > If screenfetch-3.0.0 happens to fix that error, maintainer can add a > simple > > > script > > > > > > if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then > > > eerror "Still producing error" > > > fi > > > > > > To make sure the build is properly updating the screenfetch version, > and > > > that > > > the bug has in fact been fixed. This is the only way I can see to > reliabily > > > and automatically test packages that have been merged successfully > > > > > > I don't think this needs to be an EAPI change. And if we can acheive > > the goal without one, the better. > > Agreed. > > > [... lots of interesting stuff ...] > > Or maintainers and teams could do what the Emacs team does: > provide test plans[0]. > > Naturally, I'd prefer something automated over the very hands-on > tests that are a bit of a necessity for an application like > Emacs, but they are still better than nothing. > > Either way, I think it is preferable to have non-upstream tests > of whatever shape we pick to be separate from the ebuilds and the > source files, for the simple reason that most users won't care > about them. Those who do can retrieve them in the same manner as > the ATs. > > And as for my pet peeve, tests that are known to fail, can we > also annotate that somehow so I don't waste hours running a test > suite that gives zero signal on whether I should add the stable > keyword? Even a one-line hin in the stabilization request would > be nice. As it is, I keep a list of known-to-fail packages and my > testing machinery tells me to not bother with FEATURES=test in > those case. > > Not ideal, but less time-wasty. > > Regards, > Tobias > > [0] https://wiki.gentoo.org/wiki/Project:Emacs/Test_plans > > > -- > if (user_specified) > /* Didn't work, but the user is convinced this is the > * place. */ > linux-2.4.0-test2/drivers/parport/parport_pc.c > > [-- Attachment #2: Type: text/html, Size: 5034 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 10:01 ` Pallav Agarwal @ 2016-05-17 11:26 ` Michael Orlitzky 2016-05-17 11:29 ` Ciaran McCreesh 0 siblings, 1 reply; 22+ messages in thread From: Michael Orlitzky @ 2016-05-17 11:26 UTC (permalink / raw To: gentoo-dev On 05/17/2016 06:01 AM, Pallav Agarwal wrote: > Hi, > You are right, of course. > The target is to standardize something that would encourage maintainers > to actually provide non-upstream scripts to test packages. At the same > time, it should be possible to use those scripts for automated testing > without > human interference. > We already have "emerge --config" which is expected to be run after the install process has completed, so I don't think that this is too much of a stretch. Maybe call the phase "pkg_test" analogous to "pkg_config" and in contrast to "src_test" which runs within the working directory. Then "emerge --test" could run it. I stole the Emacs "test plan" idea for a few Haskell packages: https://wiki.gentoo.org/wiki/Project:Haskell/Test_plans These do go a little beyond the scope of unit tests: they check that we installed the package in the right place, that it's available to your ghci interpreter, and that we haven't installed some other library with a name collision. Some of them have predictable output (like when we hash a string), but others might cause some trouble. The expected output for hscolour is "you should see a syntax-highlighted version of Example.hs echoed to your terminal." I don't care what colors show up, as long as that happens. Still, even if it doesn't work in 100% of cases, I think it could be useful. Some people may have inflated ideas of what the stabilization process currently involves. In many cases, it's a compile/execute test with no expected results and that could easily be automated even if it's not the world's best stabilization process. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 11:26 ` Michael Orlitzky @ 2016-05-17 11:29 ` Ciaran McCreesh 2016-05-18 8:18 ` [gentoo-dev] " Duncan 0 siblings, 1 reply; 22+ messages in thread From: Ciaran McCreesh @ 2016-05-17 11:29 UTC (permalink / raw To: gentoo-dev On Tue, 17 May 2016 07:26:03 -0400 Michael Orlitzky <mjo@gentoo.org> wrote: > We already have "emerge --config" which is expected to be run after > the install process has completed, so I don't think that this is too > much of a stretch. Maybe call the phase "pkg_test" analogous to > "pkg_config" and in contrast to "src_test" which runs within the > working directory. Then "emerge --test" could run it. For various technical reasons to do with the spec and package manglers sometimes using phase function names with the src_ or pkg_ stripped off, calling it that would be a huge pain. -- Ciaran McCreesh ^ permalink raw reply [flat|nested] 22+ messages in thread
* [gentoo-dev] Re: Proposal for changes for the next EAPI version 2016-05-17 11:29 ` Ciaran McCreesh @ 2016-05-18 8:18 ` Duncan 0 siblings, 0 replies; 22+ messages in thread From: Duncan @ 2016-05-18 8:18 UTC (permalink / raw To: gentoo-dev Ciaran McCreesh posted on Tue, 17 May 2016 12:29:36 +0100 as excerpted: > On Tue, 17 May 2016 07:26:03 -0400 Michael Orlitzky <mjo@gentoo.org> > wrote: >> We already have "emerge --config" which is expected to be run after the >> install process has completed, so I don't think that this is too much >> of a stretch. Maybe call the phase "pkg_test" analogous to "pkg_config" >> and in contrast to "src_test" which runs within the working directory. >> Then "emerge --test" could run it. > > For various technical reasons to do with the spec and package manglers > sometimes using phase function names with the src_ or pkg_ stripped off, > calling it that would be a huge pain. What about something like gentoo_test, or more generically, distro_test? That would also emphasize the difference between distro-level functionality and integration testing (which if I'm reading correctly is the proposal here) and upstream-level testing, via src_test. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 7:37 ` Pallav Agarwal 2016-05-17 8:02 ` Kent Fredric @ 2016-05-17 13:53 ` M.B. 2016-05-17 14:02 ` Brian Dolbec 2016-05-17 15:34 ` Luis Ressel 2 siblings, 1 reply; 22+ messages in thread From: M.B. @ 2016-05-17 13:53 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 2483 bytes --] Am 17.05.2016 um 09:37 schrieb Pallav Agarwal: > For normal users we wouldn't. But currently, arch-testers need to make a > judgement call on what to test when a stable-req bug is filed. Tests run in > src_test are provided by upstream, and does not guarantee that a package > that has been merged will actually run on the system. > If the maintainer could add a couple small scripts to check basic > functionality > of the merged package, it would make testing for arch testers much easier > and reliable. > Let me give an example. Let's say > app-misc/screenfetch-2.7.7 is the current stable package for screenfetch in > the portage tree. > However, on running, it produces an error on the top, along with the proper > output. > If screenfetch-3.0.0 happens to fix that error, maintainer can add a simple > script > > if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then > eerror "Still producing error" > fi > > To make sure the build is properly updating the screenfetch version, and > that > the bug has in fact been fixed. This is the only way I can see to reliabily > and automatically test packages that have been merged successfully. > > ------- > Regards, > Pallav > > On Mon, May 16, 2016 at 10:08 PM, Luis Ressel <aranea@aixah.de> wrote: > >> On Mon, 16 May 2016 18:13:33 +0530 >> Pallav Agarwal <pallavagarwal07@gmail.com> wrote: >> >>> What I'm suggesting is to add a new function post_install_test. The >>> function will run only if the build is being run for stabilization >>> (either as a part of automated stabilization, or manual) which can be >>> controlled by a USE flag. The function would also require independent >>> dependencies in case it uses external applications to test the one >>> being built. >> >> Could you please elaborate on this? We already have src_test() for >> automated tests. Why would we want to run additional tests after the >> package has been merged? >> >> -- >> Regards, >> Luis Ressel >> >> Luis Ressel <aranea@aixah.de> >> GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD >> > Good afternoon, facilities for running post-install (pre-merge) QA-checks are in place. Please have a look at portage's misc-functions.sh, install_qa_check() will reveal the locations where you can find the installed checks, along with a place for local overrides. Perhaps you can design something around this? With kind regards, tomboy64 [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 951 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 13:53 ` [gentoo-dev] " M.B. @ 2016-05-17 14:02 ` Brian Dolbec 0 siblings, 0 replies; 22+ messages in thread From: Brian Dolbec @ 2016-05-17 14:02 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 2997 bytes --] On Tue, 17 May 2016 15:53:34 +0200 "M.B." <tomboy64@sina.cn> wrote: > Am 17.05.2016 um 09:37 schrieb Pallav Agarwal: > > For normal users we wouldn't. But currently, arch-testers need to > > make a judgement call on what to test when a stable-req bug is > > filed. Tests run in src_test are provided by upstream, and does not > > guarantee that a package that has been merged will actually run on > > the system. If the maintainer could add a couple small scripts to > > check basic functionality > > of the merged package, it would make testing for arch testers much > > easier and reliable. > > Let me give an example. Let's say > > app-misc/screenfetch-2.7.7 is the current stable package for > > screenfetch in the portage tree. > > However, on running, it produces an error on the top, along with > > the proper output. > > If screenfetch-3.0.0 happens to fix that error, maintainer can add > > a simple script > > > > if [ "$(screenfetch 2>&1 1>/dev/null)" != "" ] then > > eerror "Still producing error" > > fi > > > > To make sure the build is properly updating the screenfetch > > version, and that > > the bug has in fact been fixed. This is the only way I can see to > > reliabily and automatically test packages that have been merged > > successfully. > > > > ------- > > Regards, > > Pallav > > > > On Mon, May 16, 2016 at 10:08 PM, Luis Ressel <aranea@aixah.de> > > wrote: > >> On Mon, 16 May 2016 18:13:33 +0530 > >> Pallav Agarwal <pallavagarwal07@gmail.com> wrote: > >> > >>> What I'm suggesting is to add a new function post_install_test. > >>> The function will run only if the build is being run for > >>> stabilization (either as a part of automated stabilization, or > >>> manual) which can be controlled by a USE flag. The function would > >>> also require independent dependencies in case it uses external > >>> applications to test the one being built. > >> > >> Could you please elaborate on this? We already have src_test() for > >> automated tests. Why would we want to run additional tests after > >> the package has been merged? > >> > >> -- > >> Regards, > >> Luis Ressel > >> > >> Luis Ressel <aranea@aixah.de> > >> GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD > >> > > > > Good afternoon, > > facilities for running post-install (pre-merge) QA-checks are in > place. Please have a look at portage's misc-functions.sh, > install_qa_check() will reveal the locations where you can find the > installed checks, along with a place for local overrides. Perhaps you > can design something around this? > > With kind regards, > tomboy64 > These tests would be run post-merge, in the normal file system. Mainly for stabilization checks that can be automated, so not a QA-checks qualifier which looks for common problems that can potentially lead to bugs before it is merged to the normal file system. -- Brian Dolbec <dolsen> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 951 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 7:37 ` Pallav Agarwal 2016-05-17 8:02 ` Kent Fredric 2016-05-17 13:53 ` [gentoo-dev] " M.B. @ 2016-05-17 15:34 ` Luis Ressel 2016-05-17 16:05 ` Sébastien Fabbro 2 siblings, 1 reply; 22+ messages in thread From: Luis Ressel @ 2016-05-17 15:34 UTC (permalink / raw To: gentoo-dev On Tue, 17 May 2016 13:07:43 +0530 Pallav Agarwal <pallavagarwal07@gmail.com> wrote: > Tests run in src_test are provided by upstream, and does not > guarantee that a package that has been merged will actually run on > the system. If the maintainer could add a couple small scripts to > check basic functionality of the merged package, it would make > testing for arch testers much easier and reliable. Automated post-merge tests sound kinda dangerous to me. And I don't think there's any stipulation about src_test() only running upstream-provided test suites. IMHO, src_test() would be a good place for most of the maintainter-provided tests you have in mind. Of course, there are some possible tests for which the src_test() environment isn't suitable (because they're either interactive or really need to run post-merge), I just don't expect there'd be many of them. Therefore, I think we'd be better off providing such tests out-of-band (test plans in the wiki), or perhaps stuffing them into pkg_config(). Don't get me wrong, I'm not at all opposed to your idea of easing the ATs' life, I'm just not convinced of the neccessity of EAPI changes. :) -- Regards, Luis Ressel Luis Ressel <aranea@aixah.de> GPG fpr: F08D 2AF6 655E 25DE 52BC E53D 08F5 7F90 3029 B5BD ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 15:34 ` Luis Ressel @ 2016-05-17 16:05 ` Sébastien Fabbro 2016-05-17 16:42 ` Rich Freeman 2016-05-18 0:14 ` Kent Fredric 0 siblings, 2 replies; 22+ messages in thread From: Sébastien Fabbro @ 2016-05-17 16:05 UTC (permalink / raw To: gentoo-dev On 17 May 2016 at 08:34, Luis Ressel <aranea@aixah.de> wrote: > > Automated post-merge tests sound kinda dangerous to me. And I don't > think there's any stipulation about src_test() only running > upstream-provided test suites. IMHO, src_test() would be a good place > for most of the maintainter-provided tests you have in mind. > The idea of this project is not for a typical Gentoo user to run this post-merge test, but for an automated arch-tester bot to run them in batch jobs. The result of which would be an update of the ebuild to stable. Basically CI for ebuilds: it could be implemented as a script living in the package directory, something like a .travis.yml in the GitHub repositories or may be an EAPI change. Debian has a similar project [1]. Upstream could provide CI tests and sometimes they do, but we want to make sure the package integrates well in an installed Gentoo distribution. [1] https://ci.debian.net/doc/ Sebastien ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 16:05 ` Sébastien Fabbro @ 2016-05-17 16:42 ` Rich Freeman 2016-05-18 0:14 ` Kent Fredric 1 sibling, 0 replies; 22+ messages in thread From: Rich Freeman @ 2016-05-17 16:42 UTC (permalink / raw To: gentoo-dev On Tue, May 17, 2016 at 12:05 PM, Sébastien Fabbro <bicatali@gentoo.org> wrote: > On 17 May 2016 at 08:34, Luis Ressel <aranea@aixah.de> wrote: > >> >> Automated post-merge tests sound kinda dangerous to me. And I don't >> think there's any stipulation about src_test() only running >> upstream-provided test suites. IMHO, src_test() would be a good place >> for most of the maintainter-provided tests you have in mind. >> > > The idea of this project is not for a typical Gentoo user to run this > post-merge test, but for an automated arch-tester bot to run them in > batch jobs. The result of which would be an update of the ebuild to > stable. I would suggest that it would make sense to separate tests that are unsafe for a typical user to run from those which are, to allow users to run them if they so choose. As I said earlier, arch testing has its limitations and I think many would like to be able to run the automated tests routinely. Since they're standardized they'd also provide really good feedback to maintainers when they do have errors missed by ATs. -- Rich ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-17 16:05 ` Sébastien Fabbro 2016-05-17 16:42 ` Rich Freeman @ 2016-05-18 0:14 ` Kent Fredric 2016-05-18 0:35 ` M. J. Everitt 1 sibling, 1 reply; 22+ messages in thread From: Kent Fredric @ 2016-05-18 0:14 UTC (permalink / raw To: gentoo-dev On 18 May 2016 at 04:05, Sébastien Fabbro <bicatali@gentoo.org> wrote: > Basically CI for ebuilds: it could be implemented as a script living > in the package directory, something like a .travis.yml in the GitHub > repositories or may be an EAPI change. Debian has a similar project > [1]. Upstream could provide CI tests and sometimes they do, but we > want to make sure the package integrates well in an installed Gentoo > distribution. Something like $CAT/$PN/maintainers/tests/<*> or something like that I could live with, the idea being to keep as much of this content *out* of the main ebuild flow as possible. I'd much rather however not to require files in $CAT/$PN to be changed, but to have a schema for code that can be run conditionally on any suitable package via matching ( CAT, PN, inherit, project=, maintainer= ) properties. Mostly, because there are a lot of places where you'd never want to implement the logic for a single package, you'd want to employ it for a whole bunch. Unfortunately, at present, the *sorts* of logic I personally see myself implementing would require additional dependencies to perform. This is why we're not already doing it in src_test(), because it would expand the dependency graph for end users without benefit, ( and in the way I'm thinking, results in risking a circular dependency, as some of the tests I'm wanting to do would require Perl modules installed, but these tests are to check things about Perl modules, which risks requiring itself to validate itself ...., and exposing users to that is inexcusable ) -- Kent KENTNL - https://metacpan.org/author/KENTNL ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-18 0:14 ` Kent Fredric @ 2016-05-18 0:35 ` M. J. Everitt 2016-05-18 0:44 ` Kent Fredric 0 siblings, 1 reply; 22+ messages in thread From: M. J. Everitt @ 2016-05-18 0:35 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1979 bytes --] On 18/05/16 01:14, Kent Fredric wrote: > On 18 May 2016 at 04:05, Sébastien Fabbro <bicatali@gentoo.org> wrote: >> Basically CI for ebuilds: it could be implemented as a script living >> in the package directory, something like a .travis.yml in the GitHub >> repositories or may be an EAPI change. Debian has a similar project >> [1]. Upstream could provide CI tests and sometimes they do, but we >> want to make sure the package integrates well in an installed Gentoo >> distribution. > > Something like $CAT/$PN/maintainers/tests/<*> > > or something like that I could live with, the idea being to keep as > much of this content *out* of the main ebuild flow as possible. > > I'd much rather however not to require files in $CAT/$PN to be > changed, but to have a schema for code that can be run conditionally > on any suitable package via matching ( CAT, PN, inherit, project=, > maintainer= ) properties. > > Mostly, because there are a lot of places where you'd never want to > implement the logic for a single package, you'd want to employ it for > a whole bunch. > > Unfortunately, at present, the *sorts* of logic I personally see > myself implementing would require additional dependencies to perform. > > This is why we're not already doing it in src_test(), because it would > expand the dependency graph for end users without benefit, ( and in > the way I'm thinking, results in risking a circular dependency, as > some of the tests I'm wanting to do would require Perl modules > installed, but these tests are to check things about Perl modules, > which risks requiring itself to validate itself ...., and exposing > users to that is inexcusable ) > Yes, whilst that's a special case, it would be desirable to collaborate with another maintainer/team/project to devise a test schedule that was independent from the target language, if possible. But there will always be exceptions and issues and such with these things .. :/ [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 901 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-18 0:35 ` M. J. Everitt @ 2016-05-18 0:44 ` Kent Fredric 2016-05-18 0:48 ` M. J. Everitt 0 siblings, 1 reply; 22+ messages in thread From: Kent Fredric @ 2016-05-18 0:44 UTC (permalink / raw To: gentoo-dev On 18 May 2016 at 12:35, M. J. Everitt <m.j.everitt@iee.org> wrote: > Yes, whilst that's a special case, it would be desirable to collaborate > with another maintainer/team/project to devise a test schedule that was > independent from the target language, if possible. But there will always > be exceptions and issues and such with these things .. :/ In some of these cases, the things I'd be testing have to rely on Perl Modules *because* it has to track how those specific modules respond to the code in question. For instance, to check we're doing our version normalisation correctly, we have to use the upstream reference version of Perl's version handling code directly. *NOT* doing this results in significant problems, both in our failure to perfectly map upstreams implementation in a different language, and in our ability to keep our implementation in consistency with upstream changes. And we have already suffered this problem specifically in euscan, where the euscan project implemented the version interpretation logic manually, and did so hilariously wrong, and as a result, thinks newer versions are older versions a lot of the time, and vice versa. I've attempted my own implementation of upstreams logic *better* than I think euscan does it, but I'm trapped in the reality where I have *no* objective way of knowing if it in fact, represents upstreams logic correctly. The simplest thing to say here is "implementing it in a non-target language is often enough the wrong choice". Similarly, I've made the mistake of trying to understand and interpret ebuilds statically without using bash .... that's a road to nowhere. Even using bash is a bit tortured because I can't understand how an ebuild works without reimplementing all the EAPI parts in bash or relying on some portage version of the same ( which is extremely not easy to use outside of the portage tools ). -- Kent KENTNL - https://metacpan.org/author/KENTNL ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Proposal for changes for the next EAPI version 2016-05-18 0:44 ` Kent Fredric @ 2016-05-18 0:48 ` M. J. Everitt 0 siblings, 0 replies; 22+ messages in thread From: M. J. Everitt @ 2016-05-18 0:48 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 2335 bytes --] On 18/05/16 01:44, Kent Fredric wrote: > On 18 May 2016 at 12:35, M. J. Everitt <m.j.everitt@iee.org> wrote: >> Yes, whilst that's a special case, it would be desirable to collaborate >> with another maintainer/team/project to devise a test schedule that was >> independent from the target language, if possible. But there will always >> be exceptions and issues and such with these things .. :/ > > In some of these cases, the things I'd be testing have to rely on Perl > Modules *because* it has to track how those specific modules respond > to the code in question. > > For instance, to check we're doing our version normalisation > correctly, we have to use the upstream reference version of Perl's > version handling code directly. > > *NOT* doing this results in significant problems, both in our failure > to perfectly map upstreams implementation in a different language, and > in our ability to keep our implementation in consistency with upstream > changes. > > And we have already suffered this problem specifically in euscan, > where the euscan project implemented the version interpretation logic > manually, and did so hilariously wrong, and as a result, thinks newer > versions are older versions a lot of the time, and vice versa. I've > attempted my own implementation of upstreams logic *better* than I > think euscan does it, but I'm trapped in the reality where I have *no* > objective way of knowing if it in fact, represents upstreams logic > correctly. > > The simplest thing to say here is "implementing it in a non-target > language is often enough the wrong choice". > > Similarly, I've made the mistake of trying to understand and interpret > ebuilds statically without using bash .... that's a road to nowhere. > Even using bash is a bit tortured because I can't understand how an > ebuild works without reimplementing all the EAPI parts in bash or > relying on some portage version of the same ( which is extremely not > easy to use outside of the portage tools ). > Yes, I get where you're coming from. I think many of the language and language-plugin ebuilds are going to suffer from similar problems for exactly the reasons you describe. It does make the prospect of a good, over-arching QA/CI system quite challenging (but not impossible) to achieve .. !! [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 901 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2016-05-18 8:19 UTC | newest] Thread overview: 22+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-05-16 12:43 [gentoo-dev] Proposal for changes for the next EAPI version Pallav Agarwal 2016-05-16 16:38 ` Luis Ressel 2016-05-17 7:37 ` Pallav Agarwal 2016-05-17 8:02 ` Kent Fredric 2016-05-17 8:46 ` Tobias Klausmann 2016-05-17 9:15 ` Kent Fredric 2016-05-17 10:57 ` Rich Freeman 2016-05-17 11:25 ` Pallav Agarwal 2016-05-17 11:42 ` Rich Freeman 2016-05-17 10:01 ` Pallav Agarwal 2016-05-17 11:26 ` Michael Orlitzky 2016-05-17 11:29 ` Ciaran McCreesh 2016-05-18 8:18 ` [gentoo-dev] " Duncan 2016-05-17 13:53 ` [gentoo-dev] " M.B. 2016-05-17 14:02 ` Brian Dolbec 2016-05-17 15:34 ` Luis Ressel 2016-05-17 16:05 ` Sébastien Fabbro 2016-05-17 16:42 ` Rich Freeman 2016-05-18 0:14 ` Kent Fredric 2016-05-18 0:35 ` M. J. Everitt 2016-05-18 0:44 ` Kent Fredric 2016-05-18 0:48 ` M. J. Everitt
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox