[gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Michał Górny]

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
 glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 132 insertions(+)
 create mode 100644 glep-9999.ebuild

diff --git a/glep-9999.ebuild b/glep-9999.ebuild
new file mode 100644
index 0000000..9ee18ca
--- /dev/null
+++ b/glep-9999.ebuild
@@ -0,0 +1,132 @@
+---
+GLEP: 9999
+Title: TEST_SUITE_PRESENT variable
+Author: Michał Górny <mgorny@gentoo.org>
+Type: Standards Track
+Status: Draft
+Version: 1
+Created: 2023-02-19
+Last-Modified: 2023-02-19
+Post-History: 2023-02-19
+Content-Type: text/x-rst
+---
+
+
+Abstract
+========
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
+the package features a test suite.  It can be set either by the ebuild,
+the eclass or the default ``src_test`` implementation, and afterwards
+included in the package manager logs.  This can aid in analyzing
+the results of automated package testing.
+
+
+Motivation
+==========
+
+The deployment of new Python targets in Gentoo currently involves
+testing of a large number of Gentoo packages against the said target.
+This is currently done manually for the most part.  It can be
+particularly time consuming if multiple individuals repeatedly test
+the same package only to determine that it remains incompatible with
+the new interpreter.
+
+The Python team wanted to explore the use of automation to aid this
+testing.  Unfortunately, this faces a major problem: for the vast
+of majority of packages, the incompatibilities with new Python versions
+do not exhibit during the installation and can only be detected through
+running the test suite.  The results of automated testing are therefore
+only meaningful if the package features a test phase.
+
+For packages using ``distutils-r1`` eclass, the presence of test suite
+can usually be easily determined through grepping for
+``distutils_enable_tests`` call or an explicit ``python_test()``
+function.  Even then, it seems sensible to work towards a more generic
+approach to tell whether a package had a test suite or not,
+and therefore whether a particular successful automated testing result
+means that the package actually passed tests or only confirmed that
+the Python files were copied successfully.
+
+An explicit indication whether a test suite was present can be presented
+by the package manager as part of logs, along with the result of running
+the test phase.  Afterwards, these logs can be used to determine which
+packages were actually tested.
+
+
+Specification
+=============
+
+A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
+by a ``src_test()`` implementation to indicate whether the package
+featured a test suite.  It can take three values:
+
+- ``yes`` indicating that a test suite was run
+- ``indeterminate`` indicating that it was not possible to clearly
+  determine whether the test suite was present or not (this could be
+  a case e.g. when a generic test command is run and it does not
+  indicate whether any tests were found)
+- ``no`` indicating that no test suite was run
+
+This variable *should* be set by eclasses defining the ``src_test()``
+phase.  If the package in question is using ``src_test()`` defined
+by an eclass that does not declare it explicitly, the PM must assume
+``indeterminate``.
+
+The variable *may* be set by an ebuild defining the ``src_test()``
+phase.  If the ebuild does not define it explicitly, the PM must assume
+``yes``.
+
+The default ``src_test()`` implementation as defined by the PMS sets
+the value to ``indeterminate`` if it runs a ``check`` or ``test``
+target, and to ``no`` if neither of the targets is found.
+
+
+Rationale
+=========
+
+The use of ternary flag makes it possible to clearly represent all three
+possible outcomes while navigating the defaults defined in the GLEP.
+The flag is set in ``src_test()``, so that runtime conditions (such
+as the results obtained from the actual test runner) can be used to
+determine the actual value.
+
+The defaults were defined based on the following assumptions:
+
+1. The presence of ``check`` target is common in autotools projects but
+   it does not guarantee that the target actually does anything, let
+   alone run a proper test suite.  However, the lack of any test target
+   clearly indicates that no tests were run.
+
+2. Eclass ``src_test`` implementations can be very generic and succeed
+   without actually performing any testing.  It is therefore reasonable
+   to default to ``indeterminate`` result when they are used,
+   and recommend them to explicitly override the variable.
+
+3. Explicit ``src_test`` declared in ebuild can generally be assumed
+   to actually run tests, with the exception of declaring the function
+   to prevent ``default_src_test`` from running.  It therefore makes
+   sense to default to ``yes`` but allow ebuilds to override the value
+   in the latter case.
+
+
+Backwards Compatibility
+=======================
+
+This GLEP is entirely optional.  The package managers not implementing
+it will treat the variable as a regular bash variable set by the test
+phase and ignore it.
+
+
+Reference Implementation
+========================
+
+TODO
+
+
+Copyright
+=========
+
+This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
+International License.  To view a copy of this license, visit
+https://creativecommons.org/licenses/by-sa/4.0/.
-- 
2.39.2

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Anna (cybertailor) Vyalkova]

Is it better than

	RESTRICT="test"

?

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Michał Górny]

On Sun, 2023-02-19 at 22:35 +0500, Anna (cybertailor) Vyalkova wrote:
> Is it better than
> 
> 	RESTRICT="test"
> 
> ?

Yes.  RESTRICT=test is only workable if everyone reliably set it on all
ebuilds not having any tests.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 6055 bytes --]

Hi,

Michał Górny <mgorny@gentoo.org> writes:

> Signed-off-by: Michał Górny <mgorny@gentoo.org>
> ---
>  glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 132 insertions(+)
>  create mode 100644 glep-9999.ebuild
>
> diff --git a/glep-9999.ebuild b/glep-9999.ebuild
> new file mode 100644
> index 0000000..9ee18ca
> --- /dev/null
> +++ b/glep-9999.ebuild
> @@ -0,0 +1,132 @@
> +---
> +GLEP: 9999
> +Title: TEST_SUITE_PRESENT variable
> +Author: Michał Górny <mgorny@gentoo.org>
> +Type: Standards Track
> +Status: Draft
> +Version: 1
> +Created: 2023-02-19
> +Last-Modified: 2023-02-19
> +Post-History: 2023-02-19
> +Content-Type: text/x-rst
> +---
> +
> +
> +Abstract
> +========
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> +the package features a test suite.  It can be set either by the ebuild,
> +the eclass or the default ``src_test`` implementation, and afterwards
> +included in the package manager logs.  This can aid in analyzing
> +the results of automated package testing.
> +
> +
> +Motivation
> +==========
> +
> +The deployment of new Python targets in Gentoo currently involves
> +testing of a large number of Gentoo packages against the said target.
> +This is currently done manually for the most part.  It can be
> +particularly time consuming if multiple individuals repeatedly test
> +the same package only to determine that it remains incompatible with
> +the new interpreter.
> +
> +The Python team wanted to explore the use of automation to aid this
> +testing.  Unfortunately, this faces a major problem: for the vast
> +of majority of packages, the incompatibilities with new Python versions
> +do not exhibit during the installation and can only be detected through
> +running the test suite.  The results of automated testing are therefore
> +only meaningful if the package features a test phase.
> +
> +For packages using ``distutils-r1`` eclass, the presence of test suite
> +can usually be easily determined through grepping for
> +``distutils_enable_tests`` call or an explicit ``python_test()``
> +function.  Even then, it seems sensible to work towards a more generic
> +approach to tell whether a package had a test suite or not,
> +and therefore whether a particular successful automated testing result
> +means that the package actually passed tests or only confirmed that
> +the Python files were copied successfully.
> +
> +An explicit indication whether a test suite was present can be presented
> +by the package manager as part of logs, along with the result of running
> +the test phase.  Afterwards, these logs can be used to determine which
> +packages were actually tested.
> +

+1.  I have had similar aspirations lately, so I'm glad someone else
beat me to it :)

> +
> +Specification
> +=============
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
> +by a ``src_test()`` implementation to indicate whether the package
> +featured a test suite.  It can take three values:
> +
> +- ``yes`` indicating that a test suite was run
> +- ``indeterminate`` indicating that it was not possible to clearly
> +  determine whether the test suite was present or not (this could be
> +  a case e.g. when a generic test command is run and it does not
> +  indicate whether any tests were found)
> +- ``no`` indicating that no test suite was run
> +
> +This variable *should* be set by eclasses defining the ``src_test()``
> +phase.  If the package in question is using ``src_test()`` defined
> +by an eclass that does not declare it explicitly, the PM must assume
> +``indeterminate``.
> +
> +The variable *may* be set by an ebuild defining the ``src_test()``
> +phase.  If the ebuild does not define it explicitly, the PM must assume
> +``yes``.
> +
> +The default ``src_test()`` implementation as defined by the PMS sets
> +the value to ``indeterminate`` if it runs a ``check`` or ``test``
> +target, and to ``no`` if neither of the targets is found.
> +
> +
> +Rationale
> +=========
> +
> +The use of ternary flag makes it possible to clearly represent all three
> +possible outcomes while navigating the defaults defined in the GLEP.
> +The flag is set in ``src_test()``, so that runtime conditions (such
> +as the results obtained from the actual test runner) can be used to
> +determine the actual value.
> +
> +The defaults were defined based on the following assumptions:
> +
> +1. The presence of ``check`` target is common in autotools projects but
> +   it does not guarantee that the target actually does anything, let
> +   alone run a proper test suite.  However, the lack of any test target
> +   clearly indicates that no tests were run.
> +
> +2. Eclass ``src_test`` implementations can be very generic and succeed
> +   without actually performing any testing.  It is therefore reasonable
> +   to default to ``indeterminate`` result when they are used,
> +   and recommend them to explicitly override the variable.
> +
> +3. Explicit ``src_test`` declared in ebuild can generally be assumed
> +   to actually run tests, with the exception of declaring the function
> +   to prevent ``default_src_test`` from running.  It therefore makes
> +   sense to default to ``yes`` but allow ebuilds to override the value
> +   in the latter case.
> +
> +
> +Backwards Compatibility
> +=======================
> +
> +This GLEP is entirely optional.  The package managers not implementing
> +it will treat the variable as a regular bash variable set by the test
> +phase and ignore it.
> +
> +
> +Reference Implementation
> +========================
> +
> +TODO
> +
> +
> +Copyright
> +=========
> +
> +This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
> +International License.  To view a copy of this license, visit
> +https://creativecommons.org/licenses/by-sa/4.0/.

This approach looks OK to me.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Alec Warner]

On Sun, Feb 19, 2023 at 9:32 AM Michał Górny <mgorny@gentoo.org> wrote:
>
> Signed-off-by: Michał Górny <mgorny@gentoo.org>
> ---
>  glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 132 insertions(+)
>  create mode 100644 glep-9999.ebuild
>
> diff --git a/glep-9999.ebuild b/glep-9999.ebuild
> new file mode 100644
> index 0000000..9ee18ca
> --- /dev/null
> +++ b/glep-9999.ebuild
> @@ -0,0 +1,132 @@
> +---
> +GLEP: 9999
> +Title: TEST_SUITE_PRESENT variable
> +Author: Michał Górny <mgorny@gentoo.org>
> +Type: Standards Track
> +Status: Draft
> +Version: 1
> +Created: 2023-02-19
> +Last-Modified: 2023-02-19
> +Post-History: 2023-02-19
> +Content-Type: text/x-rst
> +---
> +
> +
> +Abstract
> +========
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> +the package features a test suite.  It can be set either by the ebuild,
> +the eclass or the default ``src_test`` implementation, and afterwards
> +included in the package manager logs.  This can aid in analyzing
> +the results of automated package testing.
> +
> +
> +Motivation
> +==========
> +
> +The deployment of new Python targets in Gentoo currently involves
> +testing of a large number of Gentoo packages against the said target.
> +This is currently done manually for the most part.  It can be
> +particularly time consuming if multiple individuals repeatedly test
> +the same package only to determine that it remains incompatible with
> +the new interpreter.
> +
> +The Python team wanted to explore the use of automation to aid this
> +testing.  Unfortunately, this faces a major problem: for the vast
> +of majority of packages, the incompatibilities with new Python versions
> +do not exhibit during the installation and can only be detected through
> +running the test suite.  The results of automated testing are therefore
> +only meaningful if the package features a test phase.
> +
> +For packages using ``distutils-r1`` eclass, the presence of test suite
> +can usually be easily determined through grepping for
> +``distutils_enable_tests`` call or an explicit ``python_test()``
> +function.  Even then, it seems sensible to work towards a more generic
> +approach to tell whether a package had a test suite or not,
> +and therefore whether a particular successful automated testing result
> +means that the package actually passed tests or only confirmed that
> +the Python files were copied successfully.
> +
> +An explicit indication whether a test suite was present can be presented
> +by the package manager as part of logs, along with the result of running
> +the test phase.  Afterwards, these logs can be used to determine which
> +packages were actually tested.

So for my own edification:
 A package has no test suite: src_test always returns true; and the
output is useless for automation purposes.
 A package has a test suite: src_test may return true; and the output
is useless because it's not distinguishable from the first case.
 A package has a test suite: src_test returns false (because the tests
failed.) This is currently the case where all the value out of
src_test is gained for automation.

Here this proposal intends to distinguish between the first two cases;
by basically annotating packages so that if src_test passes and
TEST_SUITE_DETECTED=true, we can take a positive inference; and if
TEST_SUITE_DETECTED=no or indeterminate, we cannot take an inference
either way.

If so +1 to this.

-A

> +
> +
> +Specification
> +=============
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
> +by a ``src_test()`` implementation to indicate whether the package
> +featured a test suite.  It can take three values:
> +
> +- ``yes`` indicating that a test suite was run
> +- ``indeterminate`` indicating that it was not possible to clearly
> +  determine whether the test suite was present or not (this could be
> +  a case e.g. when a generic test command is run and it does not
> +  indicate whether any tests were found)
> +- ``no`` indicating that no test suite was run
> +
> +This variable *should* be set by eclasses defining the ``src_test()``
> +phase.  If the package in question is using ``src_test()`` defined
> +by an eclass that does not declare it explicitly, the PM must assume
> +``indeterminate``.
> +
> +The variable *may* be set by an ebuild defining the ``src_test()``
> +phase.  If the ebuild does not define it explicitly, the PM must assume
> +``yes``.
> +
> +The default ``src_test()`` implementation as defined by the PMS sets
> +the value to ``indeterminate`` if it runs a ``check`` or ``test``
> +target, and to ``no`` if neither of the targets is found.
> +
> +
> +Rationale
> +=========
> +
> +The use of ternary flag makes it possible to clearly represent all three
> +possible outcomes while navigating the defaults defined in the GLEP.
> +The flag is set in ``src_test()``, so that runtime conditions (such
> +as the results obtained from the actual test runner) can be used to
> +determine the actual value.
> +
> +The defaults were defined based on the following assumptions:
> +
> +1. The presence of ``check`` target is common in autotools projects but
> +   it does not guarantee that the target actually does anything, let
> +   alone run a proper test suite.  However, the lack of any test target
> +   clearly indicates that no tests were run.
> +
> +2. Eclass ``src_test`` implementations can be very generic and succeed
> +   without actually performing any testing.  It is therefore reasonable
> +   to default to ``indeterminate`` result when they are used,
> +   and recommend them to explicitly override the variable.
> +
> +3. Explicit ``src_test`` declared in ebuild can generally be assumed
> +   to actually run tests, with the exception of declaring the function
> +   to prevent ``default_src_test`` from running.  It therefore makes
> +   sense to default to ``yes`` but allow ebuilds to override the value
> +   in the latter case.
> +
> +
> +Backwards Compatibility
> +=======================
> +
> +This GLEP is entirely optional.  The package managers not implementing
> +it will treat the variable as a regular bash variable set by the test
> +phase and ignore it.
> +
> +
> +Reference Implementation
> +========================
> +
> +TODO
> +
> +
> +Copyright
> +=========
> +
> +This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
> +International License.  To view a copy of this license, visit
> +https://creativecommons.org/licenses/by-sa/4.0/.
> --
> 2.39.2
>
>

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Maciej Barć]

[-- Attachment #1.1.1: Type: text/plain, Size: 6343 bytes --]

What if developer configured an ebuild in a way that it downloads the 
test suite/files/data with USE=test?

IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should 
be true (exists).

W dniu 19.02.2023 o 18:32, Michał Górny pisze:
> Signed-off-by: Michał Górny <mgorny@gentoo.org>
> ---
>   glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 132 insertions(+)
>   create mode 100644 glep-9999.ebuild
> 
> diff --git a/glep-9999.ebuild b/glep-9999.ebuild
> new file mode 100644
> index 0000000..9ee18ca
> --- /dev/null
> +++ b/glep-9999.ebuild
> @@ -0,0 +1,132 @@
> +---
> +GLEP: 9999
> +Title: TEST_SUITE_PRESENT variable
> +Author: Michał Górny <mgorny@gentoo.org>
> +Type: Standards Track
> +Status: Draft
> +Version: 1
> +Created: 2023-02-19
> +Last-Modified: 2023-02-19
> +Post-History: 2023-02-19
> +Content-Type: text/x-rst
> +---
> +
> +
> +Abstract
> +========
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> +the package features a test suite.  It can be set either by the ebuild,
> +the eclass or the default ``src_test`` implementation, and afterwards
> +included in the package manager logs.  This can aid in analyzing
> +the results of automated package testing.
> +
> +
> +Motivation
> +==========
> +
> +The deployment of new Python targets in Gentoo currently involves
> +testing of a large number of Gentoo packages against the said target.
> +This is currently done manually for the most part.  It can be
> +particularly time consuming if multiple individuals repeatedly test
> +the same package only to determine that it remains incompatible with
> +the new interpreter.
> +
> +The Python team wanted to explore the use of automation to aid this
> +testing.  Unfortunately, this faces a major problem: for the vast
> +of majority of packages, the incompatibilities with new Python versions
> +do not exhibit during the installation and can only be detected through
> +running the test suite.  The results of automated testing are therefore
> +only meaningful if the package features a test phase.
> +
> +For packages using ``distutils-r1`` eclass, the presence of test suite
> +can usually be easily determined through grepping for
> +``distutils_enable_tests`` call or an explicit ``python_test()``
> +function.  Even then, it seems sensible to work towards a more generic
> +approach to tell whether a package had a test suite or not,
> +and therefore whether a particular successful automated testing result
> +means that the package actually passed tests or only confirmed that
> +the Python files were copied successfully.
> +
> +An explicit indication whether a test suite was present can be presented
> +by the package manager as part of logs, along with the result of running
> +the test phase.  Afterwards, these logs can be used to determine which
> +packages were actually tested.
> +
> +
> +Specification
> +=============
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
> +by a ``src_test()`` implementation to indicate whether the package
> +featured a test suite.  It can take three values:
> +
> +- ``yes`` indicating that a test suite was run
> +- ``indeterminate`` indicating that it was not possible to clearly
> +  determine whether the test suite was present or not (this could be
> +  a case e.g. when a generic test command is run and it does not
> +  indicate whether any tests were found)
> +- ``no`` indicating that no test suite was run
> +
> +This variable *should* be set by eclasses defining the ``src_test()``
> +phase.  If the package in question is using ``src_test()`` defined
> +by an eclass that does not declare it explicitly, the PM must assume
> +``indeterminate``.
> +
> +The variable *may* be set by an ebuild defining the ``src_test()``
> +phase.  If the ebuild does not define it explicitly, the PM must assume
> +``yes``.
> +
> +The default ``src_test()`` implementation as defined by the PMS sets
> +the value to ``indeterminate`` if it runs a ``check`` or ``test``
> +target, and to ``no`` if neither of the targets is found.
> +
> +
> +Rationale
> +=========
> +
> +The use of ternary flag makes it possible to clearly represent all three
> +possible outcomes while navigating the defaults defined in the GLEP.
> +The flag is set in ``src_test()``, so that runtime conditions (such
> +as the results obtained from the actual test runner) can be used to
> +determine the actual value.
> +
> +The defaults were defined based on the following assumptions:
> +
> +1. The presence of ``check`` target is common in autotools projects but
> +   it does not guarantee that the target actually does anything, let
> +   alone run a proper test suite.  However, the lack of any test target
> +   clearly indicates that no tests were run.
> +
> +2. Eclass ``src_test`` implementations can be very generic and succeed
> +   without actually performing any testing.  It is therefore reasonable
> +   to default to ``indeterminate`` result when they are used,
> +   and recommend them to explicitly override the variable.
> +
> +3. Explicit ``src_test`` declared in ebuild can generally be assumed
> +   to actually run tests, with the exception of declaring the function
> +   to prevent ``default_src_test`` from running.  It therefore makes
> +   sense to default to ``yes`` but allow ebuilds to override the value
> +   in the latter case.
> +
> +
> +Backwards Compatibility
> +=======================
> +
> +This GLEP is entirely optional.  The package managers not implementing
> +it will treat the variable as a regular bash variable set by the test
> +phase and ignore it.
> +
> +
> +Reference Implementation
> +========================
> +
> +TODO
> +
> +
> +Copyright
> +=========
> +
> +This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
> +International License.  To view a copy of this license, visit
> +https://creativecommons.org/licenses/by-sa/4.0/.

-- 
Have a great day!

~ Maciej XGQT Barć

xgqt@gentoo.org
Gentoo Linux developer
(dotnet, emacs, math, ml, nim, scheme, sci)
https://wiki.gentoo.org/wiki/User:Xgqt
9B0A 4C5D 02A3 B43C 9D6F D6B1 14D7 4A1F 43A6 AC3C

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 10875 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Michał Górny]

On Mon, 2023-02-20 at 00:10 +0100, Maciej Barć wrote:
> What if developer configured an ebuild in a way that it downloads the 
> test suite/files/data with USE=test?
> 
> IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should 
> be true (exists).
> 

How does that change anything?  If tests were run, it's a "yes". 
Otherwise, it's "no".

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Alec Warner]

On Sun, Feb 19, 2023 at 3:11 PM Maciej Barć <xgqt@gentoo.org> wrote:
>
> What if developer configured an ebuild in a way that it downloads the
> test suite/files/data with USE=test?
>
> IMO it should be added to the GLEP that then TEST_SUITE_PRESENT should
> be true (exists).

This is what I was afraid of with the name; it's not only about a
testsuite being there or not, it's about trusting that:
 - There is a test suite.
 - It was run.
 - It passed, and passing is valuable.

Just existing is not sufficient, because we cannot gain any
information from an src_test that just downloaded but didn't run the
suite. That is not valuable information.

-A

>
> W dniu 19.02.2023 o 18:32, Michał Górny pisze:
> > Signed-off-by: Michał Górny <mgorny@gentoo.org>
> > ---
> >   glep-9999.ebuild | 132 +++++++++++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 132 insertions(+)
> >   create mode 100644 glep-9999.ebuild
> >
> > diff --git a/glep-9999.ebuild b/glep-9999.ebuild
> > new file mode 100644
> > index 0000000..9ee18ca
> > --- /dev/null
> > +++ b/glep-9999.ebuild
> > @@ -0,0 +1,132 @@
> > +---
> > +GLEP: 9999
> > +Title: TEST_SUITE_PRESENT variable
> > +Author: Michał Górny <mgorny@gentoo.org>
> > +Type: Standards Track
> > +Status: Draft
> > +Version: 1
> > +Created: 2023-02-19
> > +Last-Modified: 2023-02-19
> > +Post-History: 2023-02-19
> > +Content-Type: text/x-rst
> > +---
> > +
> > +
> > +Abstract
> > +========
> > +
> > +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> > +the package features a test suite.  It can be set either by the ebuild,
> > +the eclass or the default ``src_test`` implementation, and afterwards
> > +included in the package manager logs.  This can aid in analyzing
> > +the results of automated package testing.
> > +
> > +
> > +Motivation
> > +==========
> > +
> > +The deployment of new Python targets in Gentoo currently involves
> > +testing of a large number of Gentoo packages against the said target.
> > +This is currently done manually for the most part.  It can be
> > +particularly time consuming if multiple individuals repeatedly test
> > +the same package only to determine that it remains incompatible with
> > +the new interpreter.
> > +
> > +The Python team wanted to explore the use of automation to aid this
> > +testing.  Unfortunately, this faces a major problem: for the vast
> > +of majority of packages, the incompatibilities with new Python versions
> > +do not exhibit during the installation and can only be detected through
> > +running the test suite.  The results of automated testing are therefore
> > +only meaningful if the package features a test phase.
> > +
> > +For packages using ``distutils-r1`` eclass, the presence of test suite
> > +can usually be easily determined through grepping for
> > +``distutils_enable_tests`` call or an explicit ``python_test()``
> > +function.  Even then, it seems sensible to work towards a more generic
> > +approach to tell whether a package had a test suite or not,
> > +and therefore whether a particular successful automated testing result
> > +means that the package actually passed tests or only confirmed that
> > +the Python files were copied successfully.
> > +
> > +An explicit indication whether a test suite was present can be presented
> > +by the package manager as part of logs, along with the result of running
> > +the test phase.  Afterwards, these logs can be used to determine which
> > +packages were actually tested.
> > +
> > +
> > +Specification
> > +=============
> > +
> > +A new ``TEST_SUITE_PRESENT`` variable is introduced that can be set
> > +by a ``src_test()`` implementation to indicate whether the package
> > +featured a test suite.  It can take three values:
> > +
> > +- ``yes`` indicating that a test suite was run
> > +- ``indeterminate`` indicating that it was not possible to clearly
> > +  determine whether the test suite was present or not (this could be
> > +  a case e.g. when a generic test command is run and it does not
> > +  indicate whether any tests were found)
> > +- ``no`` indicating that no test suite was run
> > +
> > +This variable *should* be set by eclasses defining the ``src_test()``
> > +phase.  If the package in question is using ``src_test()`` defined
> > +by an eclass that does not declare it explicitly, the PM must assume
> > +``indeterminate``.
> > +
> > +The variable *may* be set by an ebuild defining the ``src_test()``
> > +phase.  If the ebuild does not define it explicitly, the PM must assume
> > +``yes``.
> > +
> > +The default ``src_test()`` implementation as defined by the PMS sets
> > +the value to ``indeterminate`` if it runs a ``check`` or ``test``
> > +target, and to ``no`` if neither of the targets is found.
> > +
> > +
> > +Rationale
> > +=========
> > +
> > +The use of ternary flag makes it possible to clearly represent all three
> > +possible outcomes while navigating the defaults defined in the GLEP.
> > +The flag is set in ``src_test()``, so that runtime conditions (such
> > +as the results obtained from the actual test runner) can be used to
> > +determine the actual value.
> > +
> > +The defaults were defined based on the following assumptions:
> > +
> > +1. The presence of ``check`` target is common in autotools projects but
> > +   it does not guarantee that the target actually does anything, let
> > +   alone run a proper test suite.  However, the lack of any test target
> > +   clearly indicates that no tests were run.
> > +
> > +2. Eclass ``src_test`` implementations can be very generic and succeed
> > +   without actually performing any testing.  It is therefore reasonable
> > +   to default to ``indeterminate`` result when they are used,
> > +   and recommend them to explicitly override the variable.
> > +
> > +3. Explicit ``src_test`` declared in ebuild can generally be assumed
> > +   to actually run tests, with the exception of declaring the function
> > +   to prevent ``default_src_test`` from running.  It therefore makes
> > +   sense to default to ``yes`` but allow ebuilds to override the value
> > +   in the latter case.
> > +
> > +
> > +Backwards Compatibility
> > +=======================
> > +
> > +This GLEP is entirely optional.  The package managers not implementing
> > +it will treat the variable as a regular bash variable set by the test
> > +phase and ignore it.
> > +
> > +
> > +Reference Implementation
> > +========================
> > +
> > +TODO
> > +
> > +
> > +Copyright
> > +=========
> > +
> > +This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
> > +International License.  To view a copy of this license, visit
> > +https://creativecommons.org/licenses/by-sa/4.0/.
>
> --
> Have a great day!
>
> ~ Maciej XGQT Barć
>
> xgqt@gentoo.org
> Gentoo Linux developer
> (dotnet, emacs, math, ml, nim, scheme, sci)
> https://wiki.gentoo.org/wiki/User:Xgqt
> 9B0A 4C5D 02A3 B43C 9D6F D6B1 14D7 4A1F 43A6 AC3C

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Florian Schmaus]

On 19/02/2023 18.32, Michał Górny wrote:
> +Abstract
> +========
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> +the package features a test suite.
I wonder if we could not simply use IUSE="test" for this purpose? That 
is, allow declaring the 'test' use flag, even though it is not used in a 
USE conditional, to indicate that the package has a test suite.

With that said, I don't have any strong objections against this 
proposal. But I'd like to mention that with RESTRICT="test", IUSE="test" 
and then TEST_SUITE_PRESENT="true", we end up with an awful lot of test 
related metadata and complexity, which potentially contributes to confusion.

- Flow

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]



> On 20 Feb 2023, at 08:37, Florian Schmaus <flow@gentoo.org> wrote:
> 
> On 19/02/2023 18.32, Michał Górny wrote:
>> +Abstract
>> +========
>> +
>> +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
>> +the package features a test suite.
> I wonder if we could not simply use IUSE="test" for this purpose? That is, allow declaring the 'test' use flag, even though it is not used in a USE conditional, to indicate that the package has a test suite.
> 
This might make the confusion wrt "USE=test should enable running tests" worse, as users will see every package with tests has it.

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 51 bytes --]

How about using a new token in PROPERTIES instead?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Michał Górny]

On Mon, 2023-02-20 at 12:42 +0100, Ulrich Mueller wrote:
> How about using a new token in PROPERTIES instead?

I don't see how you could alter PROPERTIES at runtime, or how you'd
express three possible outcomes with a binary flag.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [PATCH] New pre-GLEP: TEST_SUITE_PRESENT variable

[Michał Górny]

On Sun, 2023-02-19 at 18:32 +0100, Michał Górny wrote:
> +Abstract
> +========
> +
> +A new ``TEST_SUITE_PRESENT`` variable is introduced to indicate whether
> +the package features a test suite.  It can be set either by the ebuild,
> +the eclass or the default ``src_test`` implementation, and afterwards
> +included in the package manager logs.  This can aid in analyzing
> +the results of automated package testing.
> 

I've decided to withdraw this proposal.  After giving it more thought,
it doesn't really solve the problem it was intended to solve, and it is
unnecessarily complex for a non-solution.

Long story short, a package may use Python AND have a test suite, yet
not test anything actually written in Python.  The proposal would
qualify these packages as having the test suite, yet in the end it
wouldn't provide useful input for PYTHON_COMPAT testing.

I think a domain-specific solution such as checking for python_test() or
distutils_enable_tests will work better here, and other packages will
still require manual verification.

-- 
Best regards,
Michał Górny