[gentoo-dev] tests

[gentoo-dev] tests

[Piotr Jaroszyński]

Hello,

There was some discussion about forcing/not forcing tests in EAPI-1, but there 
was clearly no compromise. Imho, tests are very important and thus I want to 
discuss them a little more, but in more sensible fashion.

Firstly each test can be(not all categories are mutually exclusive):
- not existant
- non-functional
- not runnable from ebuild
- useful but unreasonable resource-wise
- useful and reasonable resource-wise
- necessary
- known to partially fail but with a way of skipping failing tests
- known to partially fail but with no easy way of skipping failing tests
Is that list comprehensive?

Secondly we must answer the question how precisely we want to distinguish 
them, so users/dev can choose which categories of tests they want to run. 
What comes to mind is:
- run all tests
- run only necessary tests
- run only reasonable tests
- don't run tests at all
Again, is that list comprehensive?

Please don't post solutions unless we figure out which options we really want 
to deliver.

-- 
Best Regards,
Piotr Jaroszyński
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Josh Sled]

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

On Tue, 2007-05-01 at 15:08 +0200, Piotr Jaroszyński wrote:
> Firstly each test can be(not all categories are mutually exclusive):
[...]
> - necessary

Could you qualify, please?  Is this "necessary for the (non-test) build
artifact"?

If so, I'd not call it a test, just part of the build that's invoked via
`make check`. ;)

-- 
...jsled
http://asynchronous.org/ - a=jsled;b=asynchronous.org; echo ${a}@${b}

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]

On Tue, 01 May 2007 09:24:34 -0400
Josh Sled <jsled@asynchronous.org> wrote:
> On Tue, 2007-05-01 at 15:08 +0200, Piotr Jaroszyński wrote:
> > Firstly each test can be(not all categories are mutually exclusive):
> [...]
> > - necessary
> 
> Could you qualify, please?  Is this "necessary for the (non-test)
> build artifact"?

There are packages where upstream says that running 'make check' is an
essential part of the build process, and that you shouldn't install
without having done so.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Alec Warner]

Piotr Jaroszyński wrote:
> Hello,
> 
> There was some discussion about forcing/not forcing tests in EAPI-1, but there 
> was clearly no compromise. Imho, tests are very important and thus I want to 
> discuss them a little more, but in more sensible fashion.
> 
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing tests
> Is that list comprehensive?

There are some tests that require root, and thus only fails with
userpriv.  Kernel modules that attempt to load themselves to make sure
there are no problems are an example.  I forget the exact package offhand.

Otherwise I like where this discussion is going ;)
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Daniel Gryniewicz]

On Tue, 2007-05-01 at 15:08 +0200, Piotr Jaroszyński wrote:
> Hello,
> 
> There was some discussion about forcing/not forcing tests in EAPI-1, but there 
> was clearly no compromise. Imho, tests are very important and thus I want to 
> discuss them a little more, but in more sensible fashion.
> 
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing tests
> Is that list comprehensive?
> 
> Secondly we must answer the question how precisely we want to distinguish 
> them, so users/dev can choose which categories of tests they want to run. 
> What comes to mind is:
> - run all tests
> - run only necessary tests
> - run only reasonable tests
> - don't run tests at all
> Again, is that list comprehensive?
> 

Don't forget tests that have heavy requirements to run.  Many gnome
tests, for example, need a virtual X to run, which puts a new set of
DEPENDS requirements on your system.

Daniel

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Vlastimil Babka]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Daniel Gryniewicz wrote:
> On Tue, 2007-05-01 at 15:08 +0200, Piotr Jaroszyński wrote:
>> Hello,
>>
>> There was some discussion about forcing/not forcing tests in EAPI-1, but there 
>> was clearly no compromise. Imho, tests are very important and thus I want to 
>> discuss them a little more, but in more sensible fashion.
>>
>> Firstly each test can be(not all categories are mutually exclusive):
>> - not existant
>> - non-functional
>> - not runnable from ebuild
>> - useful but unreasonable resource-wise
>> - useful and reasonable resource-wise
>> - necessary
>> - known to partially fail but with a way of skipping failing tests
>> - known to partially fail but with no easy way of skipping failing tests
>> Is that list comprehensive?
>>
>> Secondly we must answer the question how precisely we want to distinguish 
>> them, so users/dev can choose which categories of tests they want to run. 
>> What comes to mind is:
>> - run all tests
>> - run only necessary tests
>> - run only reasonable tests
>> - don't run tests at all
>> Again, is that list comprehensive?
>>
> 
> Don't forget tests that have heavy requirements to run.  Many gnome
> tests, for example, need a virtual X to run, which puts a new set of
> DEPENDS requirements on your system.
> 
> Daniel
> 
And sometimes these extra deps can result in circular deps.
- --
Vlastimil Babka (Caster)
Gentoo/Java
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGN2mKtbrAj05h3oQRAsIPAJ40sOQV97jBCUFAIcKZFHJ1mRiu4QCggfz6
Toh/ZYYpU7lJfmVOqQclaWo=
=ULIL
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Maurice van der Pot]

[-- Attachment #1: Type: text/plain, Size: 2717 bytes --]

On Tue, May 01, 2007 at 03:08:56PM +0200, Piotr Jaroszyński wrote:
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing tests
> Is that list comprehensive?

Isn't it easier to list a set of boolean properties of _individual_
tests?  
- We don't need "non existent". 
- Non-functional and known to partially fail come down to "known to
  fail" for individual tests.  
- I have no idea what "not runnable from ebuild" is.  
- Unreasonable could be "resource hungry" or "needs additional deps"
  (those are two different things).
- If a test is "necessary" I don't see why we should allow it to be
  skipped.
- And about skipping failing tests. There is always a way to skip
  failing tests: not running any of them. It's just the granularity that
  is different, but the user doesn't care about that.

> Secondly we must answer the question how precisely we want to distinguish 
> them, so users/dev can choose which categories of tests they want to run. 
> What comes to mind is:
> - run all tests
> - run only necessary tests
> - run only reasonable tests
> - don't run tests at all
> Again, is that list comprehensive?

I'd say, let the user decide based on the properties, fex:

run known to fail  : no
run resource hungry: yes
run additional deps: no

if ( (known to fail   == false || run known to fail   == true) &&
     (resource hungry == false || run resource hungry == true) &&
     (additional deps == false || run additional deps == true) ) 
{
  run the test
}

So for each test/set of tests and for each possible reason not to run
it, either the reason does not apply to this test or the user explicitly
said to run it anyway.

You don't see the "way of skipping failing tests" in this last part.
This is because if the decision is made not to run a test, the smallest
set that can be skipped (possibly all tests for an ebuild) are skipped.

> Please don't post solutions unless we figure out which options we really want 
> to deliver.

I'm sorry if this counts as a solution, but I wasn't sure the list you
gave was the kind of list we need and I didn't know another way of
explaining the use of the properties I listed.

Regards,
Maurice.

-- 
Maurice van der Pot

Gentoo Linux Developer   griffon26@gentoo.org     http://www.gentoo.org
Creator of BiteMe!       griffon26@kfk4ever.com   http://www.kfk4ever.com


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Tue, 1 May 2007 19:18:28 +0200
Maurice van der Pot <griffon26@gentoo.org> wrote:
> I'd say, let the user decide based on the properties

Too complicated. Bombarding the user with pointless alternatives is not
the same as giving the user choice.

I'm also highly sceptical that the properties you listed are boolean.
Resource hungry on an IP22 could be a walk in the park for an X16...

> fex:

Please don't abuse the English language in that manner.

> I'm sorry if this counts as a solution, but I wasn't sure the list you
> gave was the kind of list we need and I didn't know another way of
> explaining the use of the properties I listed.

Well, every solution offered so far is at least moderately icky... I
believe the idea is to look at the problem in more detail before going
off in what's probably the wrong direction again.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Maurice van der Pot]

[-- Attachment #1: Type: text/plain, Size: 1936 bytes --]

On Tue, May 01, 2007 at 06:35:22PM +0100, Ciaran McCreesh wrote:
> On Tue, 1 May 2007 19:18:28 +0200
> Maurice van der Pot <griffon26@gentoo.org> wrote:
> > I'd say, let the user decide based on the properties
> 
> Too complicated. Bombarding the user with pointless alternatives is not
> the same as giving the user choice.

I'm not sure why this is a reply to my message instead of the message I
replied to. They both provide more or less the same choice to the user.

> I'm also highly sceptical that the properties you listed are boolean.
> Resource hungry on an IP22 could be a walk in the park for an X16...

I suppose that's possible, but if you look at it like that probably
everything can be called resource hungry on some machines. And if you
own a Blue Gene, you probably don't worry too much and enable
everything.

Or do you mean that for instance tests involving lots of floating point
calculations are a big deal for cpus that use FP emulation?  Isn't that
peanuts compared to the tests that would be called resource hungry here?

We wouldn't have to prove to anyone that a test is resource hungry. We
would just have to put each set of tests into one of two groups. If
you're not sure in which group it belongs, it probably doesn't matter
that much anyway. 

Look at merge times... everybody agrees openoffice, mozilla firefox,
gcc and qt take quite some time to emerge and that vim, bash and
iptables do not. That's the kind of distinction that would be useful.

> > fex:
> Please don't abuse the English language in that manner.

Since you took the time to highlight this apparently grave injustice to
the English language, would you please explain it to me so I can do
better next time?

Maurice.

-- 
Maurice van der Pot

Gentoo Linux Developer   griffon26@gentoo.org     http://www.gentoo.org
Creator of BiteMe!       griffon26@kfk4ever.com   http://www.kfk4ever.com


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Piotr Jaroszyński]

On Tuesday 01 of May 2007 21:53:36 Maurice van der Pot wrote:
> I'm not sure why this is a reply to my message instead of the message I
> replied to. They both provide more or less the same choice to the user.

Err I wasn't providing any choices for users yet, I only thought about the 
below as things that can be wanted by users/devs and asked whether I missed 
something. How we will end up distinguishing them is another story...
> - run all tests
> - run only reasonable tests
> - run only necessary tests
> - don't run tests at all

-- 
Best Regards,
Piotr Jaroszyński
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Rémi Cardona]

Piotr Jaroszyński a écrit :
> On Tuesday 01 of May 2007 21:53:36 Maurice van der Pot wrote:
>> I'm not sure why this is a reply to my message instead of the message I
>> replied to. They both provide more or less the same choice to the user.
> 
> Err I wasn't providing any choices for users yet, I only thought about the 
> below as things that can be wanted by users/devs and asked whether I missed 
> something. How we will end up distinguishing them is another story...
>> - run all tests
>> - run only reasonable tests
>> - run only necessary tests
>> - don't run tests at all

Philosophical question :

What's "reasonable"? Where do you draw the line? Same question for 
"necessary".

These two notions are very much subjective, both in terms of required 
time, dependencies, but also in terms of perceived usefulness.

Cheers,

Rémi
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Danny van Dyk]

Am Mittwoch, 2. Mai 2007 schrieb Rémi Cardona:
> Piotr Jaroszyński a écrit :
> > On Tuesday 01 of May 2007 21:53:36 Maurice van der Pot wrote:
> >> I'm not sure why this is a reply to my message instead of the
> >> message I replied to. They both provide more or less the same
> >> choice to the user.
> >
> > Err I wasn't providing any choices for users yet, I only thought
> > about the below as things that can be wanted by users/devs and
> > asked whether I missed something. How we will end up distinguishing
> > them is another story...
> >
> >> - run all tests
> >> - run only reasonable tests
> >> - run only necessary tests
> >> - don't run tests at all
>
> Philosophical question :
>
> What's "reasonable"? Where do you draw the line? Same question for
> "necessary".
Re 'necessary': Tests for scientific packages are surely necessary, i'd 
even claim they're mandatory. Nothing is as bad as having a program 
that yields unreproducable (read: wrong) results.

Been there, seen that, had the primordial urge to kill things.

Danny
-- 
Danny van Dyk <kugelfang@gentoo.org>
Gentoo/AMD64 Project, Gentoo Scientific Project
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Josh Saddler]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

Maurice van der Pot wrote:
>>> fex:
>> Please don't abuse the English language in that manner.
> 
> Since you took the time to highlight this apparently grave injustice to
> the English language, would you please explain it to me so I can do
> better next time?

he just doesn't like it because it's ferringb-speak and it makes him cranky.

anyway, on the subject of tests...as others have covered the *first*
time this was discussed on the lists, mandatory tests being run every
time the user installs a package? no. oh hell no. we don't seem to do
that much with the packages in our tree now, do we?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Stephen Bennett]

On Tue, 01 May 2007 14:52:30 -0700
Josh Saddler <nightmorph@gentoo.org> wrote:

> anyway, on the subject of tests...as others have covered the *first*
> time this was discussed on the lists, mandatory tests being run every
> time the user installs a package? no. oh hell no. we don't seem to do
> that much with the packages in our tree now, do we?

Care to turn that into a reasoned argument rather than what appears to
be a knee-jerk reaction?
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Josh Saddler]

[-- Attachment #1: Type: text/plain, Size: 1227 bytes --]

Stephen Bennett wrote:
> On Tue, 01 May 2007 14:52:30 -0700
> Josh Saddler <nightmorph@gentoo.org> wrote:
> 
>> anyway, on the subject of tests...as others have covered the *first*
>> time this was discussed on the lists, mandatory tests being run every
>> time the user installs a package? no. oh hell no. we don't seem to do
>> that much with the packages in our tree now, do we?
> 
> Care to turn that into a reasoned argument rather than what appears to
> be a knee-jerk reaction?

Not a knee jerk reaction, just a strong one. One of the key reasons why
mandatory tests were not desired was the fact that sometimes much more
stuff will be installed than what you'd normally get. Exhibit A:
robbat2's message just sent on diradm that normally just needs openldap
with USE=minimal, but building for tests requires all of openldap,
samba, etc.

I'd like to think we aren't in the practice of forcing users to install
cruft on their systems. If you need more examples from the last thread,
assuming you don't still have local archives, I could scrounge 'em from
gmane I suppose, though we're both equally capable of typing in search
phrases. The tests subject wasn't brought up that long ago, either.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Piotr Jaroszyński]

On Wednesday 02 of May 2007 00:28:42 Josh Saddler wrote:
> Not a knee jerk reaction, just a strong one. One of the key reasons why
> mandatory tests were not desired was the fact that sometimes much more
> stuff will be installed than what you'd normally get. Exhibit A:
> robbat2's message just sent on diradm that normally just needs openldap
> with USE=minimal, but building for tests requires all of openldap,
> samba, etc.
Where did you read that this thread is about forcing tests? That was the black 
and white approach and we all know how it failed... The purpose of this 
discussion is to figure out a compromise between the current state and force 
all, because neither of them is good.

-- 
Best Regards,
Piotr Jaroszyński
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

On Tue, 01 May 2007 14:52:30 -0700
Josh Saddler <nightmorph@gentoo.org> wrote:
> anyway, on the subject of tests...as others have covered the *first*
> time this was discussed on the lists, mandatory tests being run every
> time the user installs a package? no. oh hell no. we don't seem to do
> that much with the packages in our tree now, do we?

The first time this was discussed didn't come up with an ideal
solution, and the existing situation is severely broken. Hence why this
thread is going back to first principles -- once we've established what
the problem really is, *then* we can start discussing solutions again.

If you don't have anything constructive to say about the former, please
keep quiet about the latter. That *you* cannot see an adequate
solution does not mean that others won't after proper discussion.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2712 bytes --]

On Tue, 1 May 2007 21:53:36 +0200
Maurice van der Pot <griffon26@gentoo.org> wrote:
> > Too complicated. Bombarding the user with pointless alternatives is
> > not the same as giving the user choice.
> 
> I'm not sure why this is a reply to my message instead of the message
> I replied to. They both provide more or less the same choice to the
> user.

This thread is not about what's to be presented to the user. This
thread is about the tests. Discussing what's to be presented to the
user at this stage is premature.

> > I'm also highly sceptical that the properties you listed are
> > boolean. Resource hungry on an IP22 could be a walk in the park for
> > an X16...
> 
> I suppose that's possible, but if you look at it like that probably
> everything can be called resource hungry on some machines. And if you
> own a Blue Gene, you probably don't worry too much and enable
> everything.
> 
> Or do you mean that for instance tests involving lots of floating
> point calculations are a big deal for cpus that use FP emulation?
> Isn't that peanuts compared to the tests that would be called
> resource hungry here?

The point is, on some archs it is reasonable to expect that many users
will have sixteen plus logical CPUs with frequencies measured in at
least hundreds of MHz and memory measured in gigabytes, and many other
users will have a single sub-hundred MHz CPU and maybe 128MBytes RAM if
we're lucky. Test suites like, say, Boost's, are trivial to run on the
former and impossible on the latter.

> We wouldn't have to prove to anyone that a test is resource hungry. We
> would just have to put each set of tests into one of two groups. If
> you're not sure in which group it belongs, it probably doesn't matter
> that much anyway. 
> 
> Look at merge times... everybody agrees openoffice, mozilla firefox,
> gcc and qt take quite some time to emerge and that vim, bash and
> iptables do not. That's the kind of distinction that would be useful.

It's not that simple. You're forgetting that many archs routinely deal
with systems with eight or sixteen way CPUs. If a package parallelises,
it's fast on such systems. If it doesn't, it's immensely slow.

> > > fex:
> > Please don't abuse the English language in that manner.
> 
> Since you took the time to highlight this apparently grave injustice
> to the English language, would you please explain it to me so I can do
> better next time?

'fex' isn't English, and it comes across as extremely annoying and
unprofessional to many native speakers. It's worse than AOL style 'r u
2' things because 'e.g' is a similarly short and entirely correct
alternative.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Piotr Jaroszyński]

On Tuesday 01 of May 2007 19:18:28 Maurice van der Pot wrote:
> Isn't it easier to list a set of boolean properties of _individual_
> tests?
It was just a list of different test classes, which came to mind. The 
question, which still persist, was how precisely we want to divide them into 
groups as current boolean choice seems to be not enough.

> I'd say, let the user decide based on the properties, fex:
It seems to be too early for that. Firstly we should figure out 
the "properties" and then we can think how to deliver them for end-users.

-- 
Best Regards,
Piotr Jaroszyński
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Rémi Cardona]

Piotr Jaroszyński wrote:
> Hello,
> 
> There was some discussion about forcing/not forcing tests in EAPI-1, but there 
> was clearly no compromise. Imho, tests are very important and thus I want to 
> discuss them a little more, but in more sensible fashion.
> 
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing tests
  - require other and bigger deps than what the actual package requires

Rémi
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Jure Varlec]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]

On Tuesday 01 of May 2007 21:24:17 Rémi Cardona wrote:
> - require other and bigger deps than what the actual package requires

Hm, perhaps this one should be split into:

 -- additional deps are already installed
 -- additional deps are not yet installed

Regards

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

On Tue, May 01, 2007 at 10:10:28PM +0200, Jure Varlec wrote:
> On Tuesday 01 of May 2007 21:24:17 R??mi Cardona wrote:
> > - require other and bigger deps than what the actual package requires
> Hm, perhaps this one should be split into:
>  -- additional deps are already installed
>  -- additional deps are not yet installed
No, it's more complicated than that.

One of my packages, app-admin/diradm, needs deps built a certain way.

Building it for normal usage just requires openldap built with
USE=minimal.

Building it for tests requires a full OpenLDAP, and additionally
requires all of samba.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Council Member
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

[gentoo-dev] Re: tests

[Piotr Jaroszyński]

> Firstly each test can be(not all categories are mutually exclusive):
> (...)
How many of these we can find is not really that important. I mentioned the 
different categories just to show that tests are not black and white and we 
need more then boolean choice to make good use of them.
What we need to figure out is the categories we want to distinguish between in 
ebuilds and *then* how to implement that sanely.

-- 
Best Regards,
Piotr Jaroszyński
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 2689 bytes --]

On Tue, 1 May 2007 15:08:56 +0200
Piotr Jaroszyński <peper@gentoo.org> wrote:

> Hello,
> 
> There was some discussion about forcing/not forcing tests in EAPI-1,
> but there was clearly no compromise. Imho, tests are very important
> and thus I want to discuss them a little more, but in more sensible
> fashion.
> 
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing
> tests Is that list comprehensive?

I'd approach it a bit different: Before creating fixed classification
groups I'd first identify the attributes of tests that should be used
for those classifications.
a) cost (in terms of runtime, resource usage, additional deps)
b) effectiveness (does a failing/working test mean the package is
broken/working?)
c) importance (is there a realistic chance for the test to be useful?)
d) correctness (does the test match the implementation? overlaps a bit
with effectiveness)
e) others?

Each of these needs to be considered if we want to find a good
compromise of which tests to run and which not. A test with high cost
can still be worth running if effectiveness, correctness and importance
are also high, on the other hand a test with little effectiveness,
correctness and/or importance probably isn't worth running even with
zero cost.
Now the tricky question is how to actually measure those attributes.

> Secondly we must answer the question how precisely we want to
> distinguish them, so users/dev can choose which categories of tests
> they want to run. What comes to mind is:
> - run all tests
> - run only necessary tests
> - run only reasonable tests
> - don't run tests at all
> Again, is that list comprehensive?

Problem is that terms like "reasonable" or "necessary" are quite
subjective (regarding both humans and machines), and in this special
context even "all" could be interpreted in different ways (btw, could
someone give some real examples for packages with "necessary" tests?).

So I think a more fine grained classification is needed that can be
adopted for specific use cases (e.g. the mips+embedded profiles might
want different defaults than the amd64+desktop profiles).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Daniel Gryniewicz]

On Wed, 2007-05-02 at 01:32 +0200, Marius Mauch wrote:

> 
> I'd approach it a bit different: Before creating fixed classification
> groups I'd first identify the attributes of tests that should be used
> for those classifications.
> a) cost (in terms of runtime, resource usage, additional deps)
> b) effectiveness (does a failing/working test mean the package is
> broken/working?)
> c) importance (is there a realistic chance for the test to be useful?)
> d) correctness (does the test match the implementation? overlaps a bit
> with effectiveness)
> e) others?

There is one serious problem with this:  Who's going to do the work to
figure all this out for the 11,000 odd packages in the tree?  This seems
like a *huge* amount of work, work that I have no plan on doing for the
100-odd packages I (help) maintain, let alone the 4-10 different
versions of each package.  I highly doubt other maintainers want to do
this kind of work either.

Daniel

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

On Tue, 01 May 2007 19:46:56 -0400
Daniel Gryniewicz <dang@gentoo.org> wrote:
> There is one serious problem with this:  Who's going to do the work to
> figure all this out for the 11,000 odd packages in the tree?  This
> seems like a *huge* amount of work, work that I have no plan on doing
> for the 100-odd packages I (help) maintain, let alone the 4-10
> different versions of each package.  I highly doubt other maintainers
> want to do this kind of work either.

You're talking implementation details. This isn't the time for that!
No-one has worked out what, if anything, is to be done, so you can't
know how much work whatever it is is.

Having said that, there's no need to figure it out for the whole tree
in one go if it's an EAPI change.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

On Wed, May 02, 2007 at 12:55:05AM +0100, Ciaran McCreesh wrote:
> You're talking implementation details. This isn't the time for that!
> No-one has worked out what, if anything, is to be done, so you can't
> know how much work whatever it is is.
> 
> Having said that, there's no need to figure it out for the whole tree
> in one go if it's an EAPI change.

Please stop stating that fallacy; binding it into an EAPI version 
means you *have* to plan for the whole tree (meaning figure it out up 
front).

Skipping the whole "plan the fall out of it" bit of EAPI versioning 
means you wind up pushing multiple versions out (fragmenting the 
format), or forcing changes on folks while claiming "yeah, won't 
affect ya".

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

On Tue, 1 May 2007 17:34:07 -0700
Brian Harring <ferringb@gmail.com> wrote:
> On Wed, May 02, 2007 at 12:55:05AM +0100, Ciaran McCreesh wrote:
> > You're talking implementation details. This isn't the time for that!
> > No-one has worked out what, if anything, is to be done, so you can't
> > know how much work whatever it is is.
> > 
> > Having said that, there's no need to figure it out for the whole
> > tree in one go if it's an EAPI change.
> 
> Please stop stating that fallacy; binding it into an EAPI version 
> means you *have* to plan for the whole tree (meaning figure it out up 
> front).

It means figuring it out up front, but not changing the whole tree at
once, which is where the problem is.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]

On Tue, May 01, 2007 at 07:46:56PM -0400, Daniel Gryniewicz wrote:
> On Wed, 2007-05-02 at 01:32 +0200, Marius Mauch wrote:
> > I'd approach it a bit different: Before creating fixed classification
> > groups I'd first identify the attributes of tests that should be used
> > for those classifications.
> > a) cost (in terms of runtime, resource usage, additional deps)
> > b) effectiveness (does a failing/working test mean the package is
> > broken/working?)
> > c) importance (is there a realistic chance for the test to be useful?)
> > d) correctness (does the test match the implementation? overlaps a bit
> > with effectiveness)
> > e) others?
> There is one serious problem with this:  Who's going to do the work to
> figure all this out for the 11,000 odd packages in the tree?  This seems
> like a *huge* amount of work, work that I have no plan on doing for the
> 100-odd packages I (help) maintain, let alone the 4-10 different
> versions of each package.  I highly doubt other maintainers want to do
> this kind of work either.
This wouldn't be an instant transition, and a lot of packages would be
covered under the 'importance' side.

Using crypto packages as an example, the costs are low (compare known
input+output pairs), the effectiveness is high, the importance is high
(witness the checksum problems caused in the tree some months ago), and
the correctness is very high.

The mysql testcases on the other hand have a low effectiveness, there
have been lots of cases where they break due to userpriv or sandbox and
high cost.

For the packages I maintain, I'd definitely implement test stuff for the
crypto and system-admin packages where feasible, but for a lot of others
I wouldn't bother - the cost/benefit ratio is not high enough.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Council Member
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

Re: [gentoo-dev] tests

[Stephen Bennett]

On Tue, 01 May 2007 19:46:56 -0400
Daniel Gryniewicz <dang@gentoo.org> wrote:

> There is one serious problem with this:  Who's going to do the work to
> figure all this out for the 11,000 odd packages in the tree?  This
> seems like a *huge* amount of work, work that I have no plan on doing
> for the 100-odd packages I (help) maintain, let alone the 4-10
> different versions of each package.  I highly doubt other maintainers
> want to do this kind of work either.

Last I heard the intention was to tie it to the EAPI=1 bump, so that
packages can be updated one by one as they move to the newer eapi.
Current (ie EAPI=0) ebuilds will continue to function as they have done.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Daniel Gryniewicz]

On Wed, 2007-05-02 at 01:12 +0100, Stephen Bennett wrote:
> On Tue, 01 May 2007 19:46:56 -0400
> Daniel Gryniewicz <dang@gentoo.org> wrote:
> 
> > There is one serious problem with this:  Who's going to do the work to
> > figure all this out for the 11,000 odd packages in the tree?  This
> > seems like a *huge* amount of work, work that I have no plan on doing
> > for the 100-odd packages I (help) maintain, let alone the 4-10
> > different versions of each package.  I highly doubt other maintainers
> > want to do this kind of work either.
> 
> Last I heard the intention was to tie it to the EAPI=1 bump, so that
> packages can be updated one by one as they move to the newer eapi.
> Current (ie EAPI=0) ebuilds will continue to function as they have done.

Sure, but now you're requiring me to go through all that extra work if I
want any of the benefits of EAPI=1.  Or alternatively, dooming us to
support EAPI=0 forever, since I don't want to do that work.  Or, third
option, is that everyone marks their packages as "low priority tests,
don't run them" just to switch to EAPI=1, and we have no gain over what
we have now.

Honestly, tests are nice, but too many of them are broken upstream, and
we are not (and should not be, IMO) in the position of fixing them all.
If a developer wants to work with her upstream to fix the tests in her
packages, great and more power to her.  Most of us are swamped just
supporting them, let alone fixing test cases.  You really need an
upstream who cares a lot about tests for the tests to be meaningful and
work.  Lots of upstreams don't currently care, and have inherited
obsolete and (now) broken tests from previous maintainers.

I think this thread in general overestimates the value of tests in
packages.  I think we will find, if we go through the effort, that more
of them are useless and/or broken than are useful.  My 2 cents.

Daniel

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Danny van Dyk]

Hi Daniel,

Am Mittwoch, 2. Mai 2007 schrieb Daniel Gryniewicz:
> Honestly, tests are nice, but too many of them are broken upstream,
> and we are not (and should not be, IMO) in the position of fixing
> them all. If a developer wants to work with her upstream to fix the
> tests in her packages, great and more power to her.  Most of us are
> swamped just supporting them, let alone fixing test cases.  You
> really need an upstream who cares a lot about tests for the tests to
> be meaningful and work.  Lots of upstreams don't currently care, and
> have inherited obsolete and (now) broken tests from previous
> maintainers.
When you read Piotr's original mail carefully, you will see that he 
lists 'non-functional' as category, and nobody keeps you from declaring 
your packages' test-suites as such. However, keep in mind that several 
other maintainers don't have so many problems with their test-suites.

> I think this thread in general overestimates the value of tests in
> packages.  I think we will find, if we go through the effort, that
> more of them are useless and/or broken than are useful.  My 2 cents.
As a member of the sci team I have to say I completely disagree with you 
here. sci-* packages mostly have reasonable test suites, the importance 
to run them is very high (you do want reproducable and correct results, 
don't you?). However, sometimes you cannot run those tests from an 
ebuild's environment, for example when you need a running x-server.

Danny
-- 
Danny van Dyk <kugelfang@gentoo.org>
Gentoo/AMD64 Project, Gentoo Scientific Project
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 812 bytes --]

On Tue, 01 May 2007 21:51:17 -0400
Daniel Gryniewicz <dang@gentoo.org> wrote:
> Sure, but now you're requiring me to go through all that extra work
> if I want any of the benefits of EAPI=1.

It is likely that EAPI-1 will be stricter in quite a few areas...

> Or, third option, is that everyone marks their packages as "low
> priority tests, don't run them" just to switch to EAPI=1, and we have
> no gain over what we have now.

No, even that's a gain. It means that arch teams *know* when a test
failure isn't a problem. But 'everyone' won't do that.

> I think this thread in general overestimates the value of tests in
> packages.

To maintainers, possibly. Not to arch teams. The way test suites are
now makes arch teams' jobs a lot harder than they should be.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

On Wed, 2 May 2007 01:32:20 +0200
Marius Mauch <genone@gentoo.org> wrote:
> (btw, could someone give some real examples for packages with
> "necessary" tests?)

There're two groups of packages with necessary tests that come to mind:
those that are very compiler / system sensitive (certain scientific
apps), and those where upstream says that 'make check' is a vital part
of the build process.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Philipp Riegger]

On 02.05.2007, at 02:32, Marius Mauch wrote:

> a) cost (in terms of runtime, resource usage, additional deps)

Tools for this could be implemented in the package manager. The  
package has to be installed and tested by the developer, so if  
portage would show the times for each stage or the times for the test  
and the rest or something like that, the developer could get an idea:  
If test time is smaller than build time (or less than half of  
complete time), than it's not that much cost. It test time is less  
then 1 hour (or whatever), than it's not that much cost. In any other  
case it's much cost.

> b) effectiveness (does a failing/working test mean the package is
> broken/working?)

To figure this out before releasing a package to the tree might be  
lots of work. so this could be figured out later. If there are bugs  
about tests failing, try to reproduce it or ask the reporter to do  
some tests if everything is working as expected.

> c) importance (is there a realistic chance for the test to be useful?)

This can easily be decided, as mentioned in other posts (scientific  
packages, core packages, cryptographic packages,...)

> d) correctness (does the test match the implementation? overlaps a bit
> with effectiveness)

This might be a lot of work. I think this cannot be tested in a sane  
way for every package. So it's probably up to the maintainer/herd or  
upstream to decide if he/they sould take care of this

Philipp


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] tests

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

On Tuesday 01 May 2007, Piotr Jaroszyński wrote:
> There was some discussion about forcing/not forcing tests in EAPI-1, but
> there was clearly no compromise.

the compromise was that requiring in spec is wrong ... default handling of 
tests is up to the package manager / profiles / teams

> Imho, tests are very important and thus I want to discuss them a little
> more, but in more sensible fashion.

might be useful to see if there's a better way of tracking them ... bugzilla 
doesnt seem to give proper visibility here
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] tests

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

On Wed, 2 May 2007 16:05:06 -0400
Mike Frysinger <vapier@gentoo.org> wrote:
> On Tuesday 01 May 2007, Piotr Jaroszyński wrote:
> > There was some discussion about forcing/not forcing tests in
> > EAPI-1, but there was clearly no compromise.
> 
> the compromise was that requiring in spec is wrong ... default
> handling of tests is up to the package manager / profiles / teams

That's a lousy solution. *Something* needs to change. There are several
solutions better than the current situation; the question is whether
anyone can find one that is perfect rather than one that is merely a
lot better.

-- 
Ciaran McCreesh


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] tests

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

On Wednesday 02 May 2007, Ciaran McCreesh wrote:
> Mike Frysinger <vapier@gentoo.org> wrote:
> > On Tuesday 01 May 2007, Piotr Jaroszyński wrote:
> > > There was some discussion about forcing/not forcing tests in
> > > EAPI-1, but there was clearly no compromise.
> >
> > the compromise was that requiring in spec is wrong ... default
> > handling of tests is up to the package manager / profiles / teams
>
> That's a lousy solution.

all ive said is that it's better than what you proposed: making src_test 
mandatory.  that is not nor ever will be a viable solution.

> *Something* needs to change. There are several 
> solutions better than the current situation; the question is whether
> anyone can find one that is perfect rather than one that is merely a
> lot better.

the only feasible thing that should go into EAPI is to change the *default* 
behavior of src_test from not being run to being run ... thus all existing 
methods for controlling test behavior will continue to work the same
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

[gentoo-dev] Re: tests

[Ryan Hill]

Piotr Jaroszyński wrote:
> Hello,
> 
> There was some discussion about forcing/not forcing tests in EAPI-1, but there 
> was clearly no compromise. Imho, tests are very important and thus I want to 
> discuss them a little more, but in more sensible fashion.
> 
> Firstly each test can be(not all categories are mutually exclusive):
> - not existant
> - non-functional
> - not runnable from ebuild
> - useful but unreasonable resource-wise
> - useful and reasonable resource-wise
> - necessary
> - known to partially fail but with a way of skipping failing tests
> - known to partially fail but with no easy way of skipping failing tests
> Is that list comprehensive?

I've been running with FEATURES=test for a long time now.  Here's some
of the more interesting cases:

- fail only on little/big-endian archs
- fail only with/without root privs
- fail only if dependencies are / are not compiled with certain optional
support
- fail only with GCC >=4
- are expected to fail and are only meant as a regression test
- take 3 minutes on x86 and 3 hours on mips
- fail on hardened
- fail with/without new tar versions
- fail with/without new flex versions (etc.)
- fail if a kernel component is a module instead of built-in
- fail if certain environment variables are set
- fail if compiled with certain (safe) CFLAGS/CXXFLAGS

Can we qualify each of these into one of your categories?  (NB: I
realize there are solutions for each of these examples.  I'm pointing
out that not only is the situation not black and white, it often ranges
in the ultra-violet.)

> Secondly we must answer the question how precisely we want to distinguish 
> them, so users/dev can choose which categories of tests they want to run. 
> What comes to mind is:
> - run all tests
> - run only necessary tests
> - run only reasonable tests
> - don't run tests at all
> Again, is that list comprehensive?

- run only tests that don't require extra deps
- run only tests that work on hardened
- run only tests that work on my arch

> Please don't post solutions unless we figure out which options we really want 
> to deliver.

Sorry. (neener neener) ;)  Would people accept running src_test() by
default only on packages in the system set?  There are some that we
might want to turn off - glibc, gcc, binutils, autoconf, and automake
are on my current short list.  coreutils is also a lot of fun.  db takes
six hours.

Anyone, however, who is of the opinion that tests for their packages are
so important that they should never be skipped, and who is willing to
deal with the bug reports that will undeniably be generated as a result,
should IMO be allowed to shoot themselves in the foot, while those
uninterested can go about their business without further interruption.

In no way should this be tied to EAPI.


-- 
                                where to now? if i had to guess
dirtyepic gentoo org        i'm afraid to say antarctica's next
9B81 6C9F E791 83BB 3AB3  5B2D E625 A073 8379 37E8 (0x837937E8)

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: tests

[Alistair John Bush]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ryan Hill wrote:

> Can we qualify each of these into one of your categories?  (NB: I
> realize there are solutions for each of these examples.  I'm pointing
> out that not only is the situation not black and white, it often ranges
> in the ultra-violet.)
> 

++

> 
> Anyone, however, who is of the opinion that tests for their packages are
> so important that they should never be skipped, and who is willing to
> deal with the bug reports that will undeniably be generated as a result,
> should IMO be allowed to shoot themselves in the foot, while those
> uninterested can go about their business without further interruption.
> 
> In no way should this be tied to EAPI.
> 
> 

This should be appended with "A wise man once said ..."

Unless there is some overwhelming evidence that a majority of users have
FEATURES="test" and USE="test" set I can only see the costs of this far
outweighing the benefits.


- --
Alistair John Bush
Developer Gentoo Java

OpenPGP key 0x4900CFB7
www.gentoo.org
www.gentoo.org/proj/en/java
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGPVk0RJGZcUkAz7cRAvLxAKCMmVsmTYxboAnUg7BoElZab2K6uwCcDvLo
IXdW+e12W/yMlq7WkDXhE2E=
=o83O
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list