[gentoo-dev] Make FEATURES=test the default

[gentoo-dev] Make FEATURES=test the default

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

I'd like to suggest we make FEATURES=test (and therefore USE=test) the
default behaviour, rather than the opt-in we currently have.  Far too
many packages fail their test phase.

Since we encourage users to set CFLAGS in ways that upstream may not
have predicted, if upstream go to the trouble of providing tests it
seems sensible to me that they should be run, unless there is a very
good reason not to run them.  This will help pick up faults that are due
to the compiler version, CFLAGS etc much more consistently.

At the very least, ebuild maintainers and ATs should be running with
tests switched on.  If the tests are known to fail then the ebuild can
either RESTRICT=test, or just return successfully from src_test()
where the test report is useful even if some tests fail.

Thoughts?
-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Ciaran McCreesh]

On Sat, 5 Aug 2006 02:11:58 +0200 "Kevin F. Quinn"
<kevquinn@gentoo.org> wrote:
| I'd like to suggest we make FEATURES=test (and therefore USE=test) the
| default behaviour, rather than the opt-in we currently have.  Far too
| many packages fail their test phase.

Paludis does this. It's nice in theory. Unfortunately, it means system,
Gnome and KDE can't be installed, so it'd probably be wise to start
actually fixing all those "fails test" bugs first...

-- 
Ciaran McCreesh
Mail            : ciaran dot mccreesh at blueyonder.co.uk


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Alec Warner]

Kevin F. Quinn wrote:
> I'd like to suggest we make FEATURES=test (and therefore USE=test) the
> default behaviour, rather than the opt-in we currently have.  Far too
> many packages fail their test phase.
> 
> Since we encourage users to set CFLAGS in ways that upstream may not
> have predicted, if upstream go to the trouble of providing tests it
> seems sensible to me that they should be run, unless there is a very
> good reason not to run them.  This will help pick up faults that are due
> to the compiler version, CFLAGS etc much more consistently.
> 
> At the very least, ebuild maintainers and ATs should be running with
> tests switched on.  If the tests are known to fail then the ebuild can
> either RESTRICT=test, or just return successfully from src_test()
> where the test report is useful even if some tests fail.
> 
> Thoughts?

Give me some numbers on how many things still fail with that enabled 
because I would be concerned if the number is too high.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

On Fri, 04 Aug 2006 20:18:40 -0400
Alec Warner <antarus@gentoo.org> wrote:

> Give me some numbers on how many things still fail with that enabled 
> because I would be concerned if the number is too high.

I don't have numbers, but if you have FEATURES=test set yourself you
should know how many fail. It's not insignificant.

-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Saturday 05 August 2006 11:05, Kevin F. Quinn wrote:
> On Fri, 04 Aug 2006 20:18:40 -0400
>
> Alec Warner <antarus@gentoo.org> wrote:
> > Give me some numbers on how many things still fail with that enabled
> > because I would be concerned if the number is too high.
>
> I don't have numbers, but if you have FEATURES=test set yourself you
> should know how many fail. It's not insignificant.

Part of the problem is that many test suites themselves are broken. Or broken 
on some architectures. Other times the tests fail because of broken 
dependencies.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 200 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Joshua Jackson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kevin F. Quinn wrote:
> I'd like to suggest we make FEATURES=test (and therefore USE=test) the
> default behaviour, rather than the opt-in we currently have.  Far too
> many packages fail their test phase.
>
> Since we encourage users to set CFLAGS in ways that upstream may not
> have predicted, if upstream go to the trouble of providing tests it
> seems sensible to me that they should be run, unless there is a very
> good reason not to run them.  This will help pick up faults that are due
> to the compiler version, CFLAGS etc much more consistently.
>
> At the very least, ebuild maintainers and ATs should be running with
> tests switched on.  If the tests are known to fail then the ebuild can
> either RESTRICT=test, or just return successfully from src_test()
> where the test report is useful even if some tests fail.
>
> Thoughts?
I can say from experience that there are a great heaping load of
packages currently that fail their test suite, the sci-team seems to
be the best in this regard for passing the tests successfully. If we
do in fact plan on making this the default, we should be prepared for
a increase in bugs from  users who will now have ebuilds stop on
failed test suites. While I agree that it would be nice to see more
people using test and collision-protect I don't think its something we
should enable at this point in time till we have many packages working
correctly with the feature. If however people feel that by enabling
it, that it'll make us actually fix these issues then I'd like to see
it move forward. I'm just curious as to why this has come up suddenly
Kevin?

p.s I hope all dev's run with test and collision-protect, I know I
know you don't but I can hope.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE0+VtSENan+PfizARAhNtAKCclIg8ikGcaTgQ9E5YPBjPj93IlACeJyNX
ER9yFoXQ3hOKzoMq4OxL3Rg=
=uS4D
-----END PGP SIGNATURE-----

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 1853 bytes --]

On Fri, 04 Aug 2006 17:25:17 -0700
Joshua Jackson <tsunam@gentoo.org> wrote:

> While I agree that it would be nice to see more
> people using test and collision-protect I don't think its something we
> should enable at this point in time till we have many packages working
> correctly with the feature.

It's a bit chicken-and-egg.  While they're not default, they are a low
priority to be fixed.  However that said, I've been running with
FEATURES=test (and collision-protect) pretty much since I joined
(certainly all the time I've been a dev) and it doesn't cause a
crippling amount of trouble - mainly because most packages do not
include a test suite.

> If however people feel that by enabling
> it, that it'll make us actually fix these issues then I'd like to see
> it move forward.

That's my main motivation for suggesting it.  I think packages should
not be unmasked until they pass their test phase for the maintainer.
They shouldn't go stable until they pass their test phase for the arch
testers.

> I'm just curious as to why this has come up suddenly Kevin?

Well, it's not sudden for me - I've been thinking it should be
the default for a long time, usually every time see a test failure that
should have been caught by the maintainer and raise a bug for it.  Have
to raise the issue some time; there's no particular reason for raising
it now, and there's certainly no hurry to change anything.  A good time
to do it would be when portage goes up a revision (to 2.2, or perhaps
even when it goes to 2.1.1 if we think it won't be too disruptive).

> p.s I hope all dev's run with test and collision-protect, I know I
> know you don't but I can hope.

IMO devs should be working with "collision-protect sandbox strict
stricter test userpriv" but let's not get too excited ;)

-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Sascha Geschwandtner]

"Kevin F. Quinn" <kevquinn@gentoo.org> wrote:

I don't know if anyone is interested in my opinion, but I'll dump it on
you anyway. :-)

> IMO devs should be working with "collision-protect sandbox strict
> stricter test userpriv" but let's not get too excited ;)

ACK. I also agree with the general idea to turn on some FEATUREs
by default to help ensuring ebuild and program correctness, etc. Those
that you mentioned are the features I usually use, some of them with few
or no problems, some with frequent problems. My experiences (of anecdotal
value only, of course):

a) strict - default in the selinux profiles, works fine

b) collision-protect - the last problem I had occurred several months ago
and concerned ownership of /usr/X11R6

c) userpriv:
	1) only one ebuild really failed so far
	2) problems if (like me) you use umask=077 as root to compile your
kernel and the program you wish to install wants to access certain files
in /usr/src/linux, which fails and breaks (e.g. nvidia-drivers) or
misinterprets the situation and configure turns on some weird features or
turns off others, and perhaps breaks later (e.g. alsa-driver)

d) sandbox - no issue with it this year, for some reason not default in
the selinux profile

e) stricter - numerous ebuilds fail to install (ffmpeg, openoffice-bin,...)

f) test - numerous ebuild fail AND a (very) few programs take a *long* time
to go through the tests. Not only the former, but especially the latter
makes "test" IMHO unsuitable as a default option.

So right now, I'd like to see "collision-protect sandbox strict" included
in the default FEATUREs.

Perhaps it would be a good idea to announce somewhere (maybe GWN) that
new default FEATUREs are going to be added at some point and widespread
tests are needed beforehand?
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]

On Sat, 5 Aug 2006 13:14:17 +0200
Sascha Geschwandtner <s.geschwandtner@gmx.de> wrote:

> So right now, I'd like to see "collision-protect sandbox strict"
> included in the default FEATUREs.

sandbox and strict are already default for a long time.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Sascha Geschwandtner]

Marius Mauch <genone@gentoo.org> wrote:

> sandbox and strict are already default for a long time.

Not in the selinux profiles (sandbox is missing there). Regarding strict,
I just found out it's in the base profile, so you are of course correct.
But maybe I'm overlooking something else.

Well, I actually wanted to say something like "making these
features the default where they aren't".
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Sascha Geschwandtner]

Sascha Geschwandtner <s.geschwandtner@gmx.de> wrote:

> Not in the selinux profiles (sandbox is missing there).

No, I'm wrong here either. Sorry for the noise.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Stephen P. Becker]

Marius Mauch wrote:
> On Sat, 5 Aug 2006 13:14:17 +0200
> Sascha Geschwandtner <s.geschwandtner@gmx.de> wrote:
> 
>> So right now, I'd like to see "collision-protect sandbox strict"
>> included in the default FEATUREs.
> 
> sandbox and strict are already default for a long time.

Not 100% true.  Sandbox has been broken on mips for the entirety of its 
existence, so we have to disable it in the top level mips profile.  It 
is something that we would like to see fixed for real.  Kumba snagged 
some patch from debian for glibc that quasi-fixes it, where by that I 
mean it goes from being 100% unusable to about 80% unusable.

-Steve

P.S. Note that we have offered various portage devs hardware and/or an 
account on Iluxa's ginormous Origin 2000 machine in the past with the 
intention of getting this fixed, and nobody has taken us up on that...

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

On Sat, 05 Aug 2006 09:29:48 -0400
"Stephen P. Becker" <geoman@gentoo.org> wrote:

> Marius Mauch wrote:
> > On Sat, 5 Aug 2006 13:14:17 +0200
> > Sascha Geschwandtner <s.geschwandtner@gmx.de> wrote:
> > 
> >> So right now, I'd like to see "collision-protect sandbox strict"
> >> included in the default FEATUREs.
> > 
> > sandbox and strict are already default for a long time.
> 
> Not 100% true.  Sandbox has been broken on mips for the entirety of
> its existence, so we have to disable it in the top level mips
> profile.  It is something that we would like to see fixed for real.
> Kumba snagged some patch from debian for glibc that quasi-fixes it,
> where by that I mean it goes from being 100% unusable to about 80%
> unusable.

I know, but mips is a special case here (together with the non-linux
platforms).

> 
> -Steve
> 
> P.S. Note that we have offered various portage devs hardware and/or
> an account on Iluxa's ginormous Origin 2000 machine in the past with
> the intention of getting this fixed, and nobody has taken us up on
> that...

Probably because portage devs don't know much about sandbox internals
either, really I think the only person that could help you there right
now is Az (or maybe the SoC guy working on a BSD port of sandbox).

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

On Saturday 05 August 2006 09:29, Stephen P. Becker wrote:
> P.S. Note that we have offered various portage devs hardware and/or an
> account on Iluxa's ginormous Origin 2000 machine in the past with the
> intention of getting this fixed, and nobody has taken us up on that...

so ?  none of the portage guys work on sandbox

last i poked at it though, this looked less like a bug in sandbox and more a 
bug in glibc
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Stephen P. Becker]

Mike Frysinger wrote:
> On Saturday 05 August 2006 09:29, Stephen P. Becker wrote:
>> P.S. Note that we have offered various portage devs hardware and/or an
>> account on Iluxa's ginormous Origin 2000 machine in the past with the
>> intention of getting this fixed, and nobody has taken us up on that...
> 
> so ?  none of the portage guys work on sandbox
> 
> last i poked at it though, this looked less like a bug in sandbox and more a 
> bug in glibc
> -mike

The metadata for sandbox suggests that it is under the control of the 
portage team, even if they lack a herd:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE pkgmetadata SYSTEM "http://www.gentoo.org/dtd/metadata.dtd">
<pkgmetadata>
<herd>no-herd</herd>

<!-- portage lacks a herd.  correct this when we have one. -->

<maintainer>
   <email>sandbox@gentoo.org</email>
   <description>Sandbox Maintainers</description>
</maintainer>
</pkgmetadata>


-Steve
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

On Saturday 05 August 2006 14:56, Stephen P. Becker wrote:
> The metadata for sandbox suggests that it is under the control of the
> portage team, even if they lack a herd:

... because it is tightly integrated with portage ... there is the aspects of 
portage which require some sandbox env setup/etc..., then there is sandbox 
itself

but seriously, you've been around forever, you know this :p
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Stephen P. Becker]

Mike Frysinger wrote:
> On Saturday 05 August 2006 14:56, Stephen P. Becker wrote:
>> The metadata for sandbox suggests that it is under the control of the
>> portage team, even if they lack a herd:
> 
> ... because it is tightly integrated with portage ... there is the aspects of 
> portage which require some sandbox env setup/etc..., then there is sandbox 
> itself
> 
> but seriously, you've been around forever, you know this :p

Of course I know this, and it sucks.  If sandbox is so tightly 
integrated with portage, then why *isn't* there a portage team member 
who works on sandbox?

-Steve

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

On Saturday 05 August 2006 16:07, Stephen P. Becker wrote:
> Of course I know this, and it sucks.  If sandbox is so tightly
> integrated with portage, then why *isn't* there a portage team member
> who works on sandbox?

because portage requires deep knowledge in python/bash

sandbox requires deep knowledge in C/ELF

the portage guys like hacking on python all day, they dont like hacking on 
C/ELF ;)
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Ned Ludd]

On Sat, 2006-08-05 at 16:07 -0400, Stephen P. Becker wrote:
> Mike Frysinger wrote:
> > On Saturday 05 August 2006 14:56, Stephen P. Becker wrote:
> >> The metadata for sandbox suggests that it is under the control of the
> >> portage team, even if they lack a herd:
> > 
> > ... because it is tightly integrated with portage ... there is the aspects of 
> > portage which require some sandbox env setup/etc..., then there is sandbox 
> > itself
> > 
> > but seriously, you've been around forever, you know this :p
> 
> Of course I know this, and it sucks.  If sandbox is so tightly 
> integrated with portage, then why *isn't* there a portage team member 
> who works on sandbox?

cuz portage is a python beast and azarah wrote sandbox in c as a 
preload module.

And really as Mike already pointed out the problem lies within the mips
dynamic linker/loader..


-- 
Ned Ludd <solar@gentoo.org>
Gentoo Linux

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Matthias Schwarzott]

[-- Attachment #1: Type: text/plain, Size: 465 bytes --]

On Saturday 05 August 2006 11:33, Kevin F. Quinn wrote:
>
> IMO devs should be working with "collision-protect sandbox strict
> stricter test userpriv" but let's not get too excited ;)

Why not discussing what should be default for developers and then having 
FEATURES=developer activating these flags, but being overwritable when 
needed.

like FEATURES="developer -test" 

Matthias

-- 
Matthias Schwarzott
Gentoo Developer
http://www.gentoo.org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Make FEATURES=test the default

[Duncan]

"Kevin F. Quinn" <kevquinn@gentoo.org> posted
20060805113339.3b4a0569@c1358217.kevquinn.com, excerpted below, on  Sat,
05 Aug 2006 11:33:39 +0200:

> IMO devs should be working with "collision-protect sandbox strict
> stricter test userpriv" but let's not get too excited ;)

Don't forget (for the appropriate arch(s)) multilib-strict! =8^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Make FEATURES=test the default

[Duncan]

"Kevin F. Quinn" <kevquinn@gentoo.org> posted
20060805113339.3b4a0569@c1358217.kevquinn.com, excerpted below, on  Sat,
05 Aug 2006 11:33:39 +0200:

> IMO devs should be working with "collision-protect sandbox strict
> stricter test userpriv" but let's not get too excited ;)

Don't forget (for the appropriate arch(s)) multilib-strict! =8^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Jakub Moc]

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

Kevin F. Quinn wrote:
> I'd like to suggest we make FEATURES=test (and therefore USE=test) the
> default behaviour, rather than the opt-in we currently have.  Far too
> many packages fail their test phase.

Sure everyone likes to watch glibc failing? :P </joke>

Well, can't be done until bugs such as
http://bugs.gentoo.org/show_bug.cgi?id=69343 are solved (at least as in
sticking RESTRICT=test there) instead of being ignored.

> At the very least, ebuild maintainers and ATs should be running with
> tests switched on.  If the tests are known to fail then the ebuild can
> either RESTRICT=test, or just return successfully from src_test()
> where the test report is useful even if some tests fail.

See above. And even then, I don't think it's a good idea to force this
upon users. Lots of packages have tests that are very time-consuming,
and there are packages that always fail tests and it's pretty much
expected (PHP is one of them; and while the failure isn't fatal there,
it still takes tons of time to go thru those ~2000 tests). And there are
tons of packages where tests are more or less unmaintained.


-- 
Best regards,

 Jakub Moc
 mailto:jakub@gentoo.org
 GPG signature:
 http://subkeys.pgp.net:11371/pks/lookup?op=get&search=0xCEBA3D9E
 Primary key fingerprint: D2D7 933C 9BA1 C95B 2C95  B30F 8717 D5FD CEBA 3D9E

 ... still no signature   ;)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Tim Yamin]

On Sat, Aug 05, 2006 at 02:26:16AM +0200, Jakub Moc wrote:
> Kevin F. Quinn wrote:
> > I'd like to suggest we make FEATURES=test (and therefore USE=test) the
> > default behaviour, rather than the opt-in we currently have.  Far too
> > many packages fail their test phase.
> 
> Sure everyone likes to watch glibc failing? :P </joke>
> 
> Well, can't be done until bugs such as
> http://bugs.gentoo.org/show_bug.cgi?id=69343 are solved (at least as in
> sticking RESTRICT=test there) instead of being ignored.
> 
> > At the very least, ebuild maintainers and ATs should be running with
> > tests switched on.  If the tests are known to fail then the ebuild can
> > either RESTRICT=test, or just return successfully from src_test()
> > where the test report is useful even if some tests fail.
> 
> See above. And even then, I don't think it's a good idea to force this
> upon users. Lots of packages have tests that are very time-consuming,
> and there are packages that always fail tests and it's pretty much
> expected (PHP is one of them; and while the failure isn't fatal there,
> it still takes tons of time to go thru those ~2000 tests). And there are
> tons of packages where tests are more or less unmaintained.

Agreed. It may be better to instead have a FORCE="test" on certain
ebuilds (mainly sci-* stuff where you want to be sure the numbers are
coming out correctly) -- adding FEATURES="test" to the default set
will cause serious breakage and will take quite some time to be
fully fixed across the whole tree.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

On Saturday 05 August 2006 18:07, Tim Yamin wrote:
> Agreed. It may be better to instead have a FORCE="test" on certain
> ebuilds (mainly sci-* stuff where you want to be sure the numbers are
> coming out correctly) -- adding FEATURES="test" to the default set
> will cause serious breakage and will take quite some time to be
> fully fixed across the whole tree.

Comming to it. Packages that I maintain such as sys-libs/db and subversion 
have test suites that only run correctly when all kinds of bindings are 
compiled. They do not work in most useflag configurations because they 
unconditionally test the bindings.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 200 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Danny van Dyk]

Am Samstag, 5. August 2006 02:11 schrieb Kevin F. Quinn:
> At the very least, ebuild maintainers and ATs should be running with
> tests switched on.  If the tests are known to fail then the ebuild
> can either RESTRICT=test, or just return successfully from src_test()
> where the test report is useful even if some tests fail.
>
> Thoughts?
* autoconf takes ages (longer than compiling glibc here).
* glibc tests fail on amd64 since at least a year.
* automake|e2fsprogs|neon|gettext|tar have failed tests for me more than
  once.

As soon as these are fixed, i wouldn't mind making FEATURES="test" a 
default.

Danny
-- 
Danny van Dyk <kugelfang@gentoo.org>
Gentoo/AMD64 Project, Gentoo Scientific Project
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]

On Sat, 5 Aug 2006 02:39:16 +0200
Danny van Dyk <kugelfang@gentoo.org> wrote:

> Am Samstag, 5. August 2006 02:11 schrieb Kevin F. Quinn:
> > At the very least, ebuild maintainers and ATs should be running with
> > tests switched on.  If the tests are known to fail then the ebuild
> > can either RESTRICT=test, or just return successfully from
> > src_test() where the test report is useful even if some tests fail.
> >
> > Thoughts?
> * autoconf takes ages (longer than compiling glibc here).
> * glibc tests fail on amd64 since at least a year.
> * automake|e2fsprogs|neon|gettext|tar have failed tests for me more
> than once.
> 
> As soon as these are fixed, i wouldn't mind making FEATURES="test" a 
> default.

Well, if something fails its tests but you still want it regardless or
you want to skip the test phase for some other reason, you can always
do FEATURES="-test" emerge <foo>.

Changing the default doesn't prevent people from skipping tests,
however in the long term it will reduce the amount of stuff committed
to the tree that doesn't pass tests.  It will increase the amount of
times a system or world update falls over, but changing the default
will raise the priority for getting these things fixed.

There are many packages in the tree for which it is clear the
maintainer did not even attempt to run the tests - e.g.
https://bugs.gentoo.org/show_bug.cgi?id=139414  To my mind committing
packages without even bothering to try the test phase is inexcusable.

-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Danny van Dyk]

Am Samstag, 5. August 2006 11:19 schrieb Kevin F. Quinn:
> On Sat, 5 Aug 2006 02:39:16 +0200
>
> Danny van Dyk <kugelfang@gentoo.org> wrote:
> > Am Samstag, 5. August 2006 02:11 schrieb Kevin F. Quinn:
> > > At the very least, ebuild maintainers and ATs should be running
> > > with tests switched on.  If the tests are known to fail then the
> > > ebuild can either RESTRICT=test, or just return successfully from
> > > src_test() where the test report is useful even if some tests
> > > fail.
> > >
> > > Thoughts?
> >
> > * autoconf takes ages (longer than compiling glibc here).
> > * glibc tests fail on amd64 since at least a year.
> > * automake|e2fsprogs|neon|gettext|tar have failed tests for me more
> > than once.
> >
> > As soon as these are fixed, i wouldn't mind making FEATURES="test"
> > a default.
>
> Well, if something fails its tests but you still want it regardless
> or you want to skip the test phase for some other reason, you can
> always do FEATURES="-test" emerge <foo>.
>
> Changing the default doesn't prevent people from skipping tests,
> however in the long term it will reduce the amount of stuff committed
> to the tree that doesn't pass tests.  It will increase the amount of
> times a system or world update falls over, but changing the default
> will raise the priority for getting these things fixed.
>
> There are many packages in the tree for which it is clear the
> maintainer did not even attempt to run the tests - e.g.
> https://bugs.gentoo.org/show_bug.cgi?id=139414  To my mind committing
> packages without even bothering to try the test phase is inexcusable.

Something?

Please re-read the list of packages that fail tests:
 * glibc
 * autoconf
 * gettext
 * tar
That makes _4_ system packages. Before I would consider making 
FEATURES=test a default, I would add least want the system set to 
actually merge with it.

Danny
-- 
Danny van Dyk <kugelfang@gentoo.org>
Gentoo/AMD64 Project, Gentoo Scientific Project
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

On Sat, 5 Aug 2006 11:49:53 +0200
Danny van Dyk <kugelfang@gentoo.org> wrote:

> Please re-read the list of packages that fail tests:
>  * glibc
>  * autoconf
>  * gettext
>  * tar
> That makes _4_ system packages. Before I would consider making 
> FEATURES=test a default, I would add least want the system set to 
> actually merge with it.

So you're happy to let users install these packages without them
knowing the tests would fail?

I certainly agree they should pass their tests.  autoconf-2.60,
gettext-0.15 and tar-1.15.1-r1, which are the latest versions I
have installed here, all pass on my system. If they fail on your
platform, then you should make sure bugs are open and the relevant
maintainers are doing something about it, and IMO they should not go to
arch (i.e. should remain ~arch) until the test issues are resolved.

Thing is, at the moment you have a bunch of packages installed that
fail their tests.  This may mean the tests are broken, however it may
also mean the packages are not working correctly on your system, and
I'd be concerned if I were you.  Avoiding the test phase doesn't make
the packages work, obviously.

glibc is somewhat of a special case; it is especially sensitive to
the environment - many of the tests assume a vanilla RedHat
environment, and often the test failures in glibc are not actual
problems with glibc but limitations of the test suite.  However we
should not be encouraging people to install glibc versions where the
test failures are not understood.  Clearly if something in glibc is not
behaving properly, the effects can be nasty.

-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Ned Ludd]

On Sat, 2006-08-05 at 12:57 +0200, Kevin F. Quinn wrote:
> On Sat, 5 Aug 2006 11:49:53 +0200
> Danny van Dyk <kugelfang@gentoo.org> wrote:
> 
> > Please re-read the list of packages that fail tests:
> >  * glibc
> >  * autoconf
> >  * gettext
> >  * tar
> > That makes _4_ system packages. Before I would consider making 
> > FEATURES=test a default, I would add least want the system set to 
> > actually merge with it.
> 
> So you're happy to let users install these packages without them
> knowing the tests would fail?
> 
> I certainly agree they should pass their tests.  autoconf-2.60,
> gettext-0.15 and tar-1.15.1-r1, which are the latest versions I
> have installed here, all pass on my system. If they fail on your
> platform, then you should make sure bugs are open and the relevant
> maintainers are doing something about it, and IMO they should not go to
> arch (i.e. should remain ~arch) until the test issues are resolved.
> 
> Thing is, at the moment you have a bunch of packages installed that
> fail their tests.  This may mean the tests are broken, however it may
> also mean the packages are not working correctly on your system, and
> I'd be concerned if I were you.

With some arches this is not really an option. Also system pkgs such
like the toolchain need to have additional deps.

>   Avoiding the test phase doesn't make
> the packages work, obviously.
> 
> glibc is somewhat of a special case; it is especially sensitive to
> the environment - many of the tests assume a vanilla RedHat
> environment, and often the test failures in glibc are not actual
> problems with glibc but limitations of the test suite.  

Sometimes the tests are flat out wrong.
Take for example say we decided to paxtest ran itself in as the test.. 
This would surely fail on amd64 as one or two of the tests assume page 
sizes of 4096.

> However we
> should not be encouraging people to install glibc versions where the
> test failures are not understood.  

The alternative would then become for the end user to use 
another distro with less hassles. We would surely get the rep 
of sucking if nobody could even install libc.

> Clearly if something in glibc is not
> behaving properly, the effects can be nasty.

Which for the most part is why features like 
this should be opt-in vs opt-out or be left up 
to the $ARCH teams.

A lot of people are opting in so most of these will be 
fixed in due time.. The $ARCH teams *should* already be setting 
this feature for the most part before stable markings.

It's a noble idea. I just don't think we are ready for 
FEATURES=test && USE=test either.


-- 
Ned Ludd <solar@gentoo.org>
Gentoo Linux

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

On Saturday 05 August 2006 06:57, Kevin F. Quinn wrote:
> On Sat, 5 Aug 2006 11:49:53 +0200
> Danny van Dyk <kugelfang@gentoo.org> wrote:
> > Please re-read the list of packages that fail tests:
> >  * glibc
> >  * autoconf
> >  * gettext
> >  * tar
> > That makes _4_ system packages. Before I would consider making
> > FEATURES=test a default, I would add least want the system set to
> > actually merge with it.
>
> So you're happy to let users install these packages without them
> knowing the tests would fail?

before i added binutils-2.17, i ran `make check` on it for about 25 
targets ... of those, about 10 failed ...

i checked with upstream and others reproduced it ... i dont know about you, 
but i dont have the skills to go in and fix the failures for all of those 
architectures

while i like the idea of all packages being able to pass FEATURES=test, 
somethings it just aint gonna happen with Gentoo's available skill set
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Harald van Dijk]

On Sat, Aug 05, 2006 at 02:35:49PM -0400, Mike Frysinger wrote:
> On Saturday 05 August 2006 06:57, Kevin F. Quinn wrote:
> > On Sat, 5 Aug 2006 11:49:53 +0200
> > Danny van Dyk <kugelfang@gentoo.org> wrote:
> > > Please re-read the list of packages that fail tests:
> > >  * glibc
> > >  * autoconf
> > >  * gettext
> > >  * tar
> > > That makes _4_ system packages. Before I would consider making
> > > FEATURES=test a default, I would add least want the system set to
> > > actually merge with it.
> >
> > So you're happy to let users install these packages without them
> > knowing the tests would fail?
> 
> before i added binutils-2.17, i ran `make check` on it for about 25 
> targets ... of those, about 10 failed ...
> 
> i checked with upstream and others reproduced it ... i dont know about you, 
> but i dont have the skills to go in and fix the failures for all of those 
> architectures

Then RESTRICT=test, or use a src_test which warns on test failures
rather than aborting, could be used. Or am I missing something?

> while i like the idea of all packages being able to pass FEATURES=test, 
> somethings it just aint gonna happen with Gentoo's available skill set
> -mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

On Saturday 05 August 2006 14:48, Harald van Dijk wrote:
> Then RESTRICT=test, or use a src_test which warns on test failures
> rather than aborting, could be used. Or am I missing something?

some architectures pass fine

my [hidden] point was that globally enabling/disabling FEATURES=test isnt a 
cut and dry issue
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Make FEATURES=test the default

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 1475 bytes --]

On Sat, 5 Aug 2006 14:35:49 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> On Saturday 05 August 2006 06:57, Kevin F. Quinn wrote:
> > On Sat, 5 Aug 2006 11:49:53 +0200
> > Danny van Dyk <kugelfang@gentoo.org> wrote:
> > > Please re-read the list of packages that fail tests:
> > >  * glibc
> > >  * autoconf
> > >  * gettext
> > >  * tar
> > > That makes _4_ system packages. Before I would consider making
> > > FEATURES=test a default, I would add least want the system set to
> > > actually merge with it.
> >
> > So you're happy to let users install these packages without them
> > knowing the tests would fail?
> 
> before i added binutils-2.17, i ran `make check` on it for about 25 
> targets ... of those, about 10 failed ...
> 
> i checked with upstream and others reproduced it ... i dont know
> about you, but i dont have the skills to go in and fix the failures
> for all of those architectures

Agreed, however you could rig src_test() to either skip the tests on
those arches, or run them without die()ing and ewarn about the known
failures on those arches, or just leave them in ~arch (or even
masked) for the arches where they fail, depending on the impact of the
failures. That sort of thing is well within our role of package
management.

> while i like the idea of all packages being able to pass
> FEATURES=test, somethings it just aint gonna happen with Gentoo's
> available skill set -mike

-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Make FEATURES=test the default

[Sven Köhler]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

> I'd like to suggest we make FEATURES=test (and therefore USE=test) the
> default behaviour, rather than the opt-in we currently have.  Far too
> many packages fail their test phase.

I have a related question, but first a comment:

I like the idea, that packages run their self-tests before they get
marked stable. And if it should become default and a user doesn't want
it, he can disable it. So what?


So my question is:
where's the difference between USE="test" and FEATURES="test" ?

So FEATURES="test" means, that portage runs src_test(), right?
So what does USE="test" mean?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: [gentoo-dev] Re: Make FEATURES=test the default

[Markus Rothe]

[-- Attachment #1: Type: text/plain, Size: 416 bytes --]

On Saturday 05 August 2006 15:14, Sven Köhler wrote:
> So my question is:
> where's the difference between USE="test" and FEATURES="test" ?
>
> So FEATURES="test" means, that portage runs src_test(), right?
> So what does USE="test" mean?

sometimes packages require special package dependencies when the tests are 
run. USE="test" triggers this dependencies. (look at gdb for example).

regards, corsair

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: Make FEATURES=test the default

[Kevin F. Quinn]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On Sat, 05 Aug 2006 17:14:23 +0200
Sven Köhler <skoehler@upb.de> wrote:

> So my question is:
> where's the difference between USE="test" and FEATURES="test" ?
> 
> So FEATURES="test" means, that portage runs src_test(), right?

Yes.

> So what does USE="test" mean?

USE="test" is a workaround; portage cannot use FEATUREs in dep
strings.  So for example, if the test phase needs a separate tarball of
data (a common enough occurrence) and the ebuild maintainer wants to
download that tarball only if the tests are being run (perhaps the
test data is huge), this can only be done with USE flags.

Basically, if you set FEATURES="test", add "test" also to your USE
flags.  USE="test" should never be used for anything other than
supporting FEATURES="test".

-- 
Kevin F. Quinn

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: Make FEATURES=test the default

[Ciaran McCreesh]

On Sat, 5 Aug 2006 17:35:32 +0200 "Kevin F. Quinn"
<kevquinn@gentoo.org> wrote:
| USE="test" is a workaround; portage cannot use FEATUREs in dep
| strings.

Actually, it could, if anyone ever got around to adding FEATURES to
USE_EXPAND...

-- 
Ciaran McCreesh
Mail            : ciaran dot mccreesh at blueyonder.co.uk


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Make FEATURES=test the default

[Alec Warner]

Ciaran McCreesh wrote:
> On Sat, 5 Aug 2006 17:35:32 +0200 "Kevin F. Quinn"
> <kevquinn@gentoo.org> wrote:
> | USE="test" is a workaround; portage cannot use FEATUREs in dep
> | strings.
> 
> Actually, it could, if anyone ever got around to adding FEATURES to
> USE_EXPAND...
> 

We would do lots of things; that doesn't mean we should ;)
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Make FEATURES=test the default

[Harald van Dijk]

On Sat, Aug 05, 2006 at 12:04:01PM -0400, Alec Warner wrote:
> Ciaran McCreesh wrote:
> >On Sat, 5 Aug 2006 17:35:32 +0200 "Kevin F. Quinn"
> ><kevquinn@gentoo.org> wrote:
> >| USE="test" is a workaround; portage cannot use FEATUREs in 
> >dep
> >| strings.
> >
> >Actually, it could, if anyone ever got around to adding 
> >FEATURES to
> >USE_EXPAND...
> >
> 
> We would do lots of things; that doesn't mean we should ;)

s/would/could/ ? Anyway, sure, there's plenty of things we could do
that we shouldn't. Forcing a hack by users (USE=test) when a hack by
devs is available is one such thing, in my opinion. Yes, I'm aware
that relevant people think FEATURES in USE_EXPAND is a bad idea, and
I'm not sure I disagree. I think as a temporary workaround until a
clean fix is available, it beats the current situation, though.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Make FEATURES=test the default

[Hanno Böck]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

Am Samstag, 5. August 2006 02:11 schrieb Kevin F. Quinn:
> I'd like to suggest we make FEATURES=test (and therefore USE=test) the
> default behaviour, rather than the opt-in we currently have.  Far too
> many packages fail their test phase.

I'm all for making more use of features like test and collision-protect, 
though in the past I noticed that many devs don't seem to care much.
I even think to remember of bugs getting closed invalid with a "we don't care 
about"-comment. But if FEATURES="test" is considered more importand in the 
future, I'll continue bugging you with related bugs.

However I think we have a long way to go till we can even think of enabling it 
by default.

-- 
Hanno Böck		Blog:   http://www.hboeck.de/
GPG: 3DBD3B20		Jabber: jabber@hboeck.de

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Make FEATURES=test the default

[Christian 'Opfer' Faulhammer]

Tach Hanno,                                  0x2B859DE3 (PGP-PK-ID)

Hanno Böck schrieb:
> I'm all for making more use of features like test and collision-protect,
> though in the past I noticed that many devs don't seem to care much. I
> even think to remember of bugs getting closed invalid with a "we don't
> care about"-comment. But if FEATURES="test" is considered more importand
> in the future, I'll continue bugging you with related bugs.

 I would also recommend all ATs not only FEATURES="collision-protect" but  
also to activated the test suites.  I normally report failing tests on all  
bugs I test (which are a few :), but these tests are sometimes to  
dependent on specific versions of other packages or the environment they  
nearly have no use.

> However I think we have a long way to go till we can even think of
> enabling it by default.

 Right.



V-Li

-- 
Fingerprint: 68C5 D381 B69A A777 6A91 E999 350A AD7C 2B85 9DE3
http://www.gnupg.org/

-- 
gentoo-dev@gentoo.org mailing list