[gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1934 bytes --]

Hi, everyone.

Per long-standing unwritten (or rather, written but not really
officially approved) policy unmasked ebuilds are not allowed to fiddle
with Internet. While this normally makes a lot of sense (except for
special cases like live ebuilds), this also means that for some poorly
written packages we end up either disabling a fair number of tests, or
even restricting tests completely.

The Internet-based tests are of course mostly unreliable in the long
run, and should normally be replaced by some kind of mocking, local
servers etc., we simply do not have the manpower to fix all
the packages. All we can do is disable them.

Sadly, for some packages this means that we're left with no tests at
all. As developers, we can play around to run the tests manually, or
comment out needed ebuild bits for extra local testing. However, I think
it would make our work easier if we had a more uniform solutions for
detecting whenever the developer needs to disable networked tests,
and how to enable them.

The obvious solution would be to use a global USE flag with explanatory
description for that. For example:

internet-test - Enable running tests that access the Internet. Those
tests can be unreliable, result in data transfer fees and cause privacy
concerns (potentially exposing which packages are being installed). Use
at your own discretion.


The advantages of that would be:

a. tests requiring Internet are exposed in the standard ebuild metadata,
making it easy to grab it using standard tools,

b. those tests can easily be enabled, and that fact is recorded
in the installed package metadata,

c. the flag can easily be used in RESTRICT="" constraint to easily
disable all the tests.

The disadvantage is that we're introducing yet another special flag that
does not affect installed files.


What do you think? Any other ideas?


-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Alexis Ballier]

On Thu, 27 Apr 2017 16:14:13 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> b. those tests can easily be enabled, and that fact is recorded
> in the installed package metadata,
> 
> c. the flag can easily be used in RESTRICT="" constraint to easily
> disable all the tests.


I see that as a rather important disadvantage. Why would we want to
record that in VDB ?


> What do you think? Any other ideas?

why not simply a new RESTRICT value ?

Re: [gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Mike Gilbert]

On Thu, Apr 27, 2017 at 5:42 PM, Alexis Ballier <aballier@gentoo.org> wrote:
> On Thu, 27 Apr 2017 16:14:13 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
>
>> b. those tests can easily be enabled, and that fact is recorded
>> in the installed package metadata,
>>
>> c. the flag can easily be used in RESTRICT="" constraint to easily
>> disable all the tests.
>
>
> I see that as a rather important disadvantage. Why would we want to
> record that in VDB ?
>
>
>> What do you think? Any other ideas?
>
> why not simply a new RESTRICT value ?
>

How would a new RESTRICT value help? There's no way for a
user/developer to disable a RESTRICT setting unless it is declared
behind a USE flag.

Re: [gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Alexis Ballier]

On Thu, 27 Apr 2017 17:57:30 -0400
Mike Gilbert <floppym@gentoo.org> wrote:

> On Thu, Apr 27, 2017 at 5:42 PM, Alexis Ballier <aballier@gentoo.org>
> wrote:
> > On Thu, 27 Apr 2017 16:14:13 +0200
> > Michał Górny <mgorny@gentoo.org> wrote:
> >  
> >> b. those tests can easily be enabled, and that fact is recorded
> >> in the installed package metadata,
> >>
> >> c. the flag can easily be used in RESTRICT="" constraint to easily
> >> disable all the tests.  
> >
> >
> > I see that as a rather important disadvantage. Why would we want to
> > record that in VDB ?
> >
> >  
> >> What do you think? Any other ideas?  
> >
> > why not simply a new RESTRICT value ?
> >  
> 
> How would a new RESTRICT value help? There's no way for a
> user/developer to disable a RESTRICT setting unless it is declared
> behind a USE flag.
> 

What I was thinking is RESTRICT="internet-tests" letting PM skip
src_test if users disabled it, just like RESTRICT=test. But this fails
when one only wants to skip part of the tests.

Re: [gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Michał Górny]

Dnia 27 kwietnia 2017 23:42:34 CEST, Alexis Ballier <aballier@gentoo.org> napisał(a):
>On Thu, 27 Apr 2017 16:14:13 +0200
>Michał Górny <mgorny@gentoo.org> wrote:
>
>> b. those tests can easily be enabled, and that fact is recorded
>> in the installed package metadata,
>> 
>> c. the flag can easily be used in RESTRICT="" constraint to easily
>> disable all the tests.
>
>
>I see that as a rather important disadvantage. Why would we want to
>record that in VDB ?

To know whether the tests were run.

>
>
>> What do you think? Any other ideas?
>
>why not simply a new RESTRICT value ?

How would you apply it to part of the tests? How would you prevent users from running networked tests by default?


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Alexis Ballier]

On Fri, 28 Apr 2017 07:07:26 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 27 kwietnia 2017 23:42:34 CEST, Alexis Ballier
> <aballier@gentoo.org> napisał(a):
> >On Thu, 27 Apr 2017 16:14:13 +0200
> >Michał Górny <mgorny@gentoo.org> wrote:
> >  
> >> b. those tests can easily be enabled, and that fact is recorded
> >> in the installed package metadata,
> >> 
> >> c. the flag can easily be used in RESTRICT="" constraint to easily
> >> disable all the tests.  
> >
> >
> >I see that as a rather important disadvantage. Why would we want to
> >record that in VDB ?  
> 
> To know whether the tests were run.

I don't think it belongs there. Unless running tests or not makes a
difference on the installed package, in which case it is a bug in the
package. Remember: test useflag is a hack that portage auto-enables
when running tests and is not taken into account with --newuse. Plus,
I've learnt recently there are some subtle differences between portage
and PMS, like you have to manually feed RESTRICT="test? ( test )" in
order for the desired behavior to be PMS compliant.

> >
> >  
> >> What do you think? Any other ideas?  
> >
> >why not simply a new RESTRICT value ?  
> 
> How would you apply it to part of the tests? How would you prevent
> users from running networked tests by default?

Running part of the tests can get messy with RESTRICT, indeed. 
Maybe a more manual approach could work then: If ENABLE_NETWORK_TESTS
variable is set in environment then run networked tests otherwise
display a warning.

Re: [gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2282 bytes --]

On Thu, 27 Apr 2017 16:14:13 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> What do you think? Any other ideas?

I personally think that "test" being exposed as a useflag is a hack that shouldn't be perpetuated further.

I'd rather have the ability to check for the flag the same way as we check for arch, but without necessitating
that the test driver use flags are exposed visibly even to people who aren't using FEATURES=test

Similarly, there's irritation derived from having USE flags that *only* pertain to tests, because they
turn into null-options when toggled, or pointless REQUIRED_USE rubbish.

I'd rather we have something that side-steps the need for futher USE flag abuse, and
made testing flags only visible to users who were *actually* doing testing.

Something akin to USE_EXPAND but just for test flags seems pertinent.

Because there's plenty of *other* classes of testing options that we might want
in the future.

For instance, regression tests, which by necessity create circular dependencies, *should* be
a visible option.

And I can imagine something like TESTS="-* gentoo" being something that could eventually come into
existence, which would be a flag that toggled on gentoo supplied tests.

Large complex tools like databases also need graduated control of tests, because
the current approach is very much "all or nothing" most of the time, when in reality:

-> I want to run *some* sensible tests for all packages
-> But I don't want to be running test suites that could run for 50+ hours ( sys-libs/db )

But for people who don't use FEATURES=test, none of those toggles should be something they even
have to know about, the flags should simply evaporate without it.

And TEST-specific flags should be discouraged from RDEPEND, just like "test" is.

TEST-specific flags in DEPEND is also somewhat dubious when it comes to --with-beps=y 

I think in the long term, I'd want a TDEPEND or something instead.

I know you're vying for "a small incremental step that just fixes this one niche",
but this is basically a problem that's been known ( at least by me ) for many years,
and we're in growing need of a real solution, not a small incremental hack that is
a disservice to the future we know exists.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

On sob, 2017-04-29 at 01:25 +1200, Kent Fredric wrote:
> I know you're vying for "a small incremental step that just fixes this one niche",
> but this is basically a problem that's been known ( at least by me ) for many years,
> and we're in growing need of a real solution, not a small incremental hack that is
> a disservice to the future we know exists.

Congratulations, you've just completely discouraged me from pursuing
this any further. I don't have the time or the resources to solve all
the problems in the world, especially if I don't give a damn about them.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] [RFC] Providing consistent means to enable tests requiring Internet access

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

On Fri, 28 Apr 2017 15:37:15 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Congratulations, you've just completely discouraged me from pursuing
> this any further. I don't have the time or the resources to solve all
> the problems in the world, especially if I don't give a damn about
> them.

Its not that I don't want a solution here, but this has been something
that has long bothered me, sufficient I want to eventually see this
turn into a future-EAPI proposal.

I think if I was to say there is something we can do today for this,
I'd use a non-use variable to regulate it, expressly so it *cant* be
used in SRC_URI or DEPEND

perl-module.eclass has such a feature already, if an end user specifies
DIST_TEST_OVERRIDE="do parallel network", certain tests stop being
disabled ( though doing this globally will turn on a lot more tests
that are currently disabled for non-network related reasons, and force
parallelism where its not warranted, so said feature is still a poor
model to adopt verbatim in a global way )

But trying to decide on a shared ENV var that has default mechanics and
default implications and is enforced by portage quickly gets into
GLEP/EAPI territory anyway.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]