[gentoo-dev] RFC: USE=qa-test

[gentoo-dev] RFC: USE=qa-test

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

Some packages, like dbus[1], have testing features that, while useful for
developers and arch-testers, aren't something that should be foisted on
users.  Dbus' case is extreme, as it builds-in functions that are useful for
unit testing, but result in an insecure and unstable package (I just "fixed" a
bunch of testsuite failures i've been seeing in dbus-using packages by
disabling USE=test). Other packages have testsuites that take an unreasonable
amount of time to build/run (db, ppl, boost, that faad/faac one that takes
six hours), are pretty much guaranteed to fail (gcc, binutils), have strange
dependency quirks (can't run the tests unless the package is already
installed, create circular dependencies), or a dozen other situations I can't
think of right now.

I'd like to propose a new USE flag, qa-test or a better name, to handle these
cases in a consistent way.  This would give us a way to differentiate between
tests that everyone should run and tests that only devs and arch-testers
would be interested in, making enabling FEATURES=test by default in a future
EAPI a little more palatable.    Use of this flag would be up to the
maintainer, of course.


[1] http://bugs.gentoo.org/287722

-- 
fonts,                             Character is what you are in the dark.
gcc-porting,
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFC: USE=qa-test

[Tobias Klausmann]

Hi! 

On Tue, 06 Oct 2009, Ryan Hill wrote:
> [... separate testing flag/feature for
> complicated/long/somehwat broken test suites ...]
> [1] http://bugs.gentoo.org/287722

I agree. However, some of the cases aren't quite clear-cut. Take,
for example fftw. Its test suite takes about half an hour on a
modern machine (or so the ebuild claims). That's still several
times the normal build time. And in the case of slower arches,
the time can skyrocket: even on our quad-cpu Alpha dev machine
(definitely on the beefy end), the test suite takes over three
hours. 

Bottom line: at least for the build time, some sort of guideline
("increases buildtime by more than a factor of X or more than Y
minutes") might be useful. The same goes for ballooning
installed-package size. 

I think the "breaks a lot" is more clear cut, and "causes
security problems" definitely is.

Regards,
Tobias



-- 
We are sorry, but the number you have dialed is imaginary.
Please rotate your phone 90 degrees and try again.

[gentoo-dev] Re: RFC: USE=qa-test

[Duncan]

Ryan Hill posted on Tue, 06 Oct 2009 20:38:18 -0600 as excerpted:

> I'd like to propose a new USE flag, qa-test or a better name, to handle
> these cases in a consistent way.  This would give us a way to
> differentiate between tests that everyone should run and tests that only
> devs and arch-testers would be interested in, making enabling
> FEATURES=test by default in a future EAPI a little more palatable.   
> Use of this flag would be up to the maintainer, of course.

I'm confused.  I'm sure there's a good reason for the different proposal 
here, but I don't know what it is, and I'd like to know:

Previously there was discussion of RESTRICT=test, to be set for just such 
"user probably doesn't want this on" type situations.  The proposal then 
was to turn FEATURES=test on by default for a specific EAPI, and then as 
packages moved to it, they'd add RESTRICT=test where appropriate (with a 
comment documenting why, of course), and anything of that EAPI or later 
that didn't have RESTRICT=test would automatically get built for and have 
its tests run by default.

Now I see this proposal for making it a USE flag, which may or may not be 
more appropriate, I don't know.  I do know I'd love to see someone 
explain the differences.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: RFC: USE=qa-test

[Patrick Lauer]

On Wednesday 07 October 2009 13:13:31 Duncan wrote:
> Ryan Hill posted on Tue, 06 Oct 2009 20:38:18 -0600 as excerpted:
> > I'd like to propose a new USE flag, qa-test or a better name, to handle
> > these cases in a consistent way.  This would give us a way to
> > differentiate between tests that everyone should run and tests that only
> > devs and arch-testers would be interested in, making enabling
> > FEATURES=test by default in a future EAPI a little more palatable.
> > Use of this flag would be up to the maintainer, of course.
> 
> I'm confused.  I'm sure there's a good reason for the different proposal
> here, but I don't know what it is, and I'd like to know:
> 
> Previously there was discussion of RESTRICT=test, to be set for just such
> "user probably doesn't want this on" type situations. 
That force-disables tests. I'm not aware of an override for it, 
RESTRICT="test" basically means "don't bother with tests, they are suicidal"

> The proposal then
> was to turn FEATURES=test on by default for a specific EAPI,
... which is never more than a proposal by people not fixing the packages.
Just FYI, building gnome in a chroot I hit 9 test-failures out of ~150 
packages. That quota of failure is roughly stable all across the tree, so 
unless someone starts fixing the ~1000 failures it's just not viable.

Oh, and someone needs to maintain and forward-port the fixes. And someone 
needs to send me some hardware so I can find the time to run all those tests - 
on my quadcore boost takes ~4h with tests, which is just prohibitive.

> and then as
> packages moved to it, they'd add RESTRICT=test where appropriate (with a
> comment documenting why, of course), and anything of that EAPI or later
> that didn't have RESTRICT=test would automatically get built for and have
> its tests run by default.
Feel free to have FEATURES="test" enabled in a dev profile or in your 
make.conf. Forcing it on users is just insane.

> Now I see this proposal for making it a USE flag, which may or may not be
> more appropriate, I don't know.  I do know I'd love to see someone
> explain the differences.
It's a bit more selective. RESTRICT is a global "no, this is broken", the new 
useflag (or more fine-grained restrict?) says "there are tests, but they might 
make things explode" so that those that do want the tests can easily use them 
without making users sad puppied. And if there's anything we don't want it is 
more sad puppies. 

[gentoo-dev] Re: RFC: USE=qa-test

[Duncan]

Patrick Lauer posted on Wed, 07 Oct 2009 13:36:47 +0200 as excerpted:

>> Now I see this proposal for making it a USE flag, which may or may not
>> be more appropriate, I don't know.  I do know I'd love to see someone
>> explain the differences.
> It's a bit more selective. RESTRICT is a global "no, this is broken",
> the new useflag (or more fine-grained restrict?) says "there are tests,
> but they might make things explode" so that those that do want the tests
> can easily use them without making users sad puppied. And if there's
> anything we don't want it is more sad puppies.

Thanks.  That makes things /much/ clearer! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] RFC: USE=qa-test

[Gilles Dartiguelongue]

Le mardi 06 octobre 2009 à 20:38 -0600, Ryan Hill a écrit :
> Some packages, like dbus[1], have testing features that, while useful for
> developers and arch-testers, aren't something that should be foisted on
> users.  Dbus' case is extreme, as it builds-in functions that are useful for
> unit testing, but result in an insecure and unstable package (I just "fixed" a
> bunch of testsuite failures i've been seeing in dbus-using packages by
> disabling USE=test). Other packages have testsuites that take an unreasonable
> amount of time to build/run (db, ppl, boost, that faad/faac one that takes
> six hours), are pretty much guaranteed to fail (gcc, binutils), have strange
> dependency quirks (can't run the tests unless the package is already
> installed, create circular dependencies), or a dozen other situations I can't
> think of right now.
> 
> I'd like to propose a new USE flag, qa-test or a better name, to handle these
> cases in a consistent way.  This would give us a way to differentiate between
> tests that everyone should run and tests that only devs and arch-testers
> would be interested in, making enabling FEATURES=test by default in a future
> EAPI a little more palatable.    Use of this flag would be up to the
> maintainer, of course.

while it might sound sane, I think this proposal covers too much cases,
most of which should actually be filled as bugs to the maintainers of
the packages for not fixing the testsuite (or not filling an upstream
bug) before commiting to the tree.

For gnome ebuilds as someone commented out, the test failure rate is
quite stable, and we are slowly trying to get around them, or at least
not commiting ebuilds with new regressions in the testsuite.

Use of RESTRICT="test" shouldn't be encouraged as it disables tests
completely while part of them might still work and be relevant.

-- 
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo

Re: [gentoo-dev] RFC: USE=qa-test

[Jeroen Roovers]

On Tue, 6 Oct 2009 20:38:18 -0600
Ryan Hill <dirtyepic@gentoo.org> wrote:

> Some packages, like dbus[1], have testing features that, while useful
> for developers and arch-testers, aren't something that should be
> foisted on users.  Dbus' case is extreme, as it builds-in functions
> that are useful for unit testing, but result in an insecure and
> unstable package 

You could build it twice in the same ebuild... Of course, you still
wouldn't be testing the same thing as you're installing, but then again
upstream releases it that way.


Regards,
     jer

[gentoo-dev] Re: RFC: USE=qa-test

[Peter Hjalmarsson]

ons 2009-10-07 klockan 16:46 +0200 skrev Gilles Dartiguelongue:
> Le mardi 06 octobre 2009 à 20:38 -0600, Ryan Hill a écrit :
> > Some packages, like dbus[1], have testing features that, while useful for
> > developers and arch-testers, aren't something that should be foisted on
> > users.  Dbus' case is extreme, as it builds-in functions that are useful for
> > unit testing, but result in an insecure and unstable package (I just "fixed" a
> > bunch of testsuite failures i've been seeing in dbus-using packages by
> > disabling USE=test). Other packages have testsuites that take an unreasonable
> > amount of time to build/run (db, ppl, boost, that faad/faac one that takes
> > six hours), are pretty much guaranteed to fail (gcc, binutils), have strange
> > dependency quirks (can't run the tests unless the package is already
> > installed, create circular dependencies), or a dozen other situations I can't
> > think of right now.
> > 
> > I'd like to propose a new USE flag, qa-test or a better name, to handle these
> > cases in a consistent way.  This would give us a way to differentiate between
> > tests that everyone should run and tests that only devs and arch-testers
> > would be interested in, making enabling FEATURES=test by default in a future
> > EAPI a little more palatable.    Use of this flag would be up to the
> > maintainer, of course.
> 
> while it might sound sane, I think this proposal covers too much cases,
> most of which should actually be filled as bugs to the maintainers of
> the packages for not fixing the testsuite (or not filling an upstream
> bug) before commiting to the tree.
> 
> For gnome ebuilds as someone commented out, the test failure rate is
> quite stable, and we are slowly trying to get around them, or at least
> not commiting ebuilds with new regressions in the testsuite.
> 
> Use of RESTRICT="test" shouldn't be encouraged as it disables tests
> completely while part of them might still work and be relevant.
> 


The problem comes with packages like dbus where the testsuit is pretty
useful for ATs, but if dbus is merged with FEATURES="test" then other
testsuits for other packages depending on dbus breaks, which AFAICS
tells us that a dbus built with tests is pretty broken for general
usage.

Example:

FEATURES="test" emerge -1q dbus && FEATURES="test" emerge -1q
dbus-python
and dbus-python will FAIL its testsuit.

However
FEATURES="-test" USE="-test" emerge -1q dbus && FEATURES="test" emerge
-1q dbus-python
and dbus-python will compile, test and install without problems.

Now how do we ensure that the ATs still can test a package even under
these conditions?
Because as I see it RESTRICT is only for packages whose testsuits are
broken by design and never meant to be run on a working system (yeah
*drm*, I am looking at you).

I think we first hand should decide over these packages, after a
decision is made about these problems with MORE then a compile-time
impact, we can start talking about less important problems (i.e.
problems that do not break stuff) like how long time a testsuit is
supposed to run.

And FEATURES="test" IS sometimes a good thing, I learned this the hard
way once when something broke grub for me, and one of the things that
made my system unbootable was one of those things the grub testsuit
checks for.

[gentoo-dev] Re: RFC: USE=qa-test

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Wed, 7 Oct 2009 13:36:47 +0200
Patrick Lauer <patrick@gentoo.org> wrote:

> On Wednesday 07 October 2009 13:13:31 Duncan wrote:

> > The proposal then
> > was to turn FEATURES=test on by default for a specific EAPI,

> ... which is never more than a proposal by people not fixing the packages.
> Just FYI, building gnome in a chroot I hit 9 test-failures out of ~150 
> packages. That quota of failure is roughly stable all across the tree, so 
> unless someone starts fixing the ~1000 failures it's just not viable.

Yeah, this is why I want FEATURES=test turned on for dev profiles.  By the
sheer number of packages that fail, it's obvious that most devs don't enable
it.  I know that some people will still disable it again, but I'm hoping it
will at least raise the visibility of these problems.

I try to file bugs for packages that fail, especially at gcc-porting time
since we rely on testsuites to detect miscompiled code or other problems.  So
a failure-free tree is something I'd like to work towards, even if it is a
pipe dream.


-- 
fonts,                             Character is what you are in the dark.
gcc-porting,
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: RFC: USE=qa-test

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 2082 bytes --]

On Wed, 07 Oct 2009 16:46:36 +0200
Gilles Dartiguelongue <eva@gentoo.org> wrote:

> Le mardi 06 octobre 2009 à 20:38 -0600, Ryan Hill a écrit :
> > Some packages, like dbus[1], have testing features that, while useful for
> > developers and arch-testers, aren't something that should be foisted on
> > users.  Dbus' case is extreme, as it builds-in functions that are useful for
> > unit testing, but result in an insecure and unstable package (I just "fixed" a
> > bunch of testsuite failures i've been seeing in dbus-using packages by
> > disabling USE=test). Other packages have testsuites that take an unreasonable
> > amount of time to build/run (db, ppl, boost, that faad/faac one that takes
> > six hours), are pretty much guaranteed to fail (gcc, binutils), have strange
> > dependency quirks (can't run the tests unless the package is already
> > installed, create circular dependencies), or a dozen other situations I can't
> > think of right now.
> > 
> > I'd like to propose a new USE flag, qa-test or a better name, to handle these
> > cases in a consistent way.  This would give us a way to differentiate between
> > tests that everyone should run and tests that only devs and arch-testers
> > would be interested in, making enabling FEATURES=test by default in a future
> > EAPI a little more palatable.    Use of this flag would be up to the
> > maintainer, of course.
> 
> while it might sound sane, I think this proposal covers too much cases,
> most of which should actually be filled as bugs to the maintainers of
> the packages for not fixing the testsuite (or not filling an upstream
> bug) before commiting to the tree.

I don't mean for this to include broken testsuites; those should be fixed as
you said.  My main motivation is to provide an option for packages like dbus
that have useful testsuites that really shouldn't be enabled for everyday use.


-- 
fonts,                             Character is what you are in the dark.
gcc-porting,
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: RFC: USE=qa-test

[Peter Hjalmarsson]

Sorry for reviving an old thread, but was there any progress on this
topic?

With packages as dbus that breaks with FEATURES/USE="test" hand in hand
with packages like dev-libs/gmp[1] there would really be nice to know if
you are supposed to be screwed or helped by FEATURES/USE="test"...




[1](from http://gmplib.org/)
IMPORTANT INFORMATION FOR ALL GMP USERS:

GMP is very often miscompiled! Please never use your newly compiled
libgmp.a or libgmp.so without first running make check. If it doesn't
complete without errors, don't trust the library. Please try another
compiler release, or change optimisation flags until it works. If you
don't have the skill to isolate the problem, please report it to us as
if it was a GMP bug; else to the compiler vendor. (The compilers that
cause most problems are HP's unbundled compilers and gcc, in particular
Apple's gcc releases.)

N.B. gcc 4.3.2 miscompiles GMP 4.3.x on 64-bit machines.

[gentoo-dev] Re: RFC: USE=qa-test

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

On Sun, 11 Oct 2009 13:11:14 +0200
Peter Hjalmarsson <xake@rymdraket.net> wrote:

> Sorry for reviving an old thread, but was there any progress on this
> topic?
> 
> With packages as dbus that breaks with FEATURES/USE="test" hand in hand
> with packages like dev-libs/gmp[1] there would really be nice to know if
> you are supposed to be screwed or helped by FEATURES/USE="test"...

A week isn't old. ;)  I'll add it once the flu and homework let up for a
minute.

-- 
fonts,                             Character is what you are in the dark.
gcc-porting,
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: RFC: USE=qa-test

[Peter Volkov]

В Срд, 14/10/2009 в 22:12 -0600, Ryan Hill пишет:
> On Sun, 11 Oct 2009 13:11:14 +0200
> Peter Hjalmarsson <xake@rymdraket.net> wrote:
> 
> > Sorry for reviving an old thread, but was there any progress on this
> > topic?

> A week isn't old. ;)  I'll add it once the flu and homework let up for a
> minute.

Why don't you like Jeroen's suggestion? It looks more much more sane.
Currently dbus ebuild allows installation of package upstream never
intended to be installed on end-user stations and this is bug in ebuild
that should be fixed. And I don't think it's good idea to workaround
this bug with yet another USE flag which, when enabled, will still allow
users to install broken package on the system. Not speaking that this
USE flag could be easily overused.

Twice build (one for installation and one for testsuite run) is really
not hard since dbus allows out of sources build (/me just checked this).

-- 
Peter.

[gentoo-dev] Re: RFC: USE=qa-test

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

On Sat, 17 Oct 2009 21:30:20 +0400
Peter Volkov <pva@gentoo.org> wrote:

> В Срд, 14/10/2009 в 22:12 -0600, Ryan Hill пишет:
> > On Sun, 11 Oct 2009 13:11:14 +0200
> > Peter Hjalmarsson <xake@rymdraket.net> wrote:
> > 
> > > Sorry for reviving an old thread, but was there any progress on this
> > > topic?
> 
> > A week isn't old. ;)  I'll add it once the flu and homework let up for a
> > minute.
> 
> Why don't you like Jeroen's suggestion? It looks more much more sane.
> Currently dbus ebuild allows installation of package upstream never
> intended to be installed on end-user stations and this is bug in ebuild
> that should be fixed. And I don't think it's good idea to workaround
> this bug with yet another USE flag which, when enabled, will still allow
> users to install broken package on the system. Not speaking that this
> USE flag could be easily overused.
> 
> Twice build (one for installation and one for testsuite run) is really
> not hard since dbus allows out of sources build (/me just checked this).

I want a more general solution, as eva asked for in the bug.  I also want a
testing feature that allows more control than always on or off.  Other
packages already employ USE flags for testing or debug features that
shouldn't normally be enabled (even w/ USE=test).  I'd just like to make it
consistent.

But, that said, I'm happy to let the dbus guys what they want to do.


-- 
fonts,                             Character is what you are in the dark.
gcc-porting,
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]