[gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 360 bytes --]

What do you think about doing the following change in
/usr/portage/profiles/targets/developer/make.defaults:

replace "test" with "test-fail-continue" to make it just less
frustrating (we still have a lot of test failures)

Hopefully that will also make more of us use the developer profile, and
detect test failures.

What do you think?

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Jesus Rivero (Neurogeek)]

Hi,

On Fri, Jun 4, 2010 at 10:41 AM, "Paweł Hajdan, Jr."
<phajdan.jr@gentoo.org> wrote:
> What do you think about doing the following change in
> /usr/portage/profiles/targets/developer/make.defaults:
>
> replace "test" with "test-fail-continue" to make it just less
> frustrating (we still have a lot of test failures)

I've been thinking about this for a while. Some packages have tests
that are meant only for upstream in certain conditions
and are not meant to be ran during installing. As we have ARCH teams,
couldn't we think a way in which TEST teams can
be created? I mean, a bunch of devs only focused on making tests work
or just restrict them?

This team (or a Gentoo project) can work hand by hand with other teams
and ARCH members.

Is it even possible?

>
> Hopefully that will also make more of us use the developer profile, and
> detect test failures.
>
> What do you think?
>
> Paweł

Best regards,

-- 
Jesus Rivero (Neurogeek)

Re: [gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

On 6/4/10 5:35 PM, Jesus Rivero (Neurogeek) wrote:
> I've been thinking about this for a while. Some packages have tests
> that are meant only for upstream in certain conditions
> and are not meant to be ran during installing.

I think that in extreme cases src_test should not call such tests.

> As we have ARCH teams,
> couldn't we think a way in which TEST teams can
> be created? I mean, a bunch of devs only focused on making tests work
> or just restrict them?

I don't think that would be effective. Making the tests work is hard,
especially for packages like gcc, or python. Having FEATURES="test" is
intended to make developers catch these failures before checking in.

However, with many packages failing tests, people started running
FEATURES="-test" or just stopped (or never used) the developer profile.
With FEATURES="test test-fail-continue" we should get best of both
worlds: run tests always, but don't frustrate people by making build
fail "in the middle of long emerge".

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Jeroen Roovers]

On Fri, 04 Jun 2010 17:11:45 +0200
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:

> What do you think about doing the following change in
> /usr/portage/profiles/targets/developer/make.defaults:

[..]

> What do you think?

I've never felt any need or obligation to use a developer profile. I
don't think I ever saw any announcement to that effect either. What is
the use of a developer profile?[1]

Someone in the know, please sell it to me. :)


Regards,
     jer


[1] I've seen developers complain more and more about failing test
suites. Maybe that's a related issue? Developers now use the
FEATURES set out in a developer profile and can then extract some
kind of validity claim from the fact that I obviously didn't do my QA?
That would explain a lot.

Re: [gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Jeroen Roovers]

On Fri, 4 Jun 2010 18:48:38 +0200
Jeroen Roovers <jer@gentoo.org> wrote:

> [1] I've seen developers complain more and more about failing test
> suites. Maybe that's a related issue? Developers now use the
> FEATURES set out in a developer profile and can then extract some
> kind of validity claim from the fact that I obviously didn't do my QA?
> That would explain a lot.

That came out wrong.

s|from the fact|to the effect|

[gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1897 bytes --]

On Fri, 04 Jun 2010 17:11:45 +0200
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:

> What do you think about doing the following change in
> /usr/portage/profiles/targets/developer/make.defaults:
> 
> replace "test" with "test-fail-continue" to make it just less
> frustrating (we still have a lot of test failures)
> 
> Hopefully that will also make more of us use the developer profile, and
> detect test failures.
> 
> What do you think?

I would say it's an improvement only because it might prevent one or two
people from completely disabling it first chance they get. :)

IMO, test failures should be given the same status as build failures.
Packages shouldn't be commited until they're fixed or bypassed.  Following
that reasoning, FEATURES="test" is the correct setting for the dev profile.
It _should_ be annoying when you hit it, that's the point.  Fix it!  What's
the point of even having a test suite if it always fails?  You'd be better off
to RESTRICT it and save yourself some bug reports from me and all the
other users you're foisting build errors on.

But in the real world it seems it's just never going to happen.  I've been
arguing this for years but people simply don't care.  It doesn't help that we
don't have any finer control than "on" or "off".  I'd like to be able to say
things like "these tests should only be run by developers" or "some failures
are normal" or "hope you have 10 hours to run this" or "don't run these as
root" or "don't run tests on arm" etc etc.  I'd like a pony while I'm at it.

Sorry about the rant.  This is one of my biggest long-standing annoyances.

Um, so yeah.  For it!


-- 
fonts,                               there's a hole in my neighbourhood
gcc-porting,                  down which of late i cannot help but fall
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Thilo Bangert]

[-- Attachment #1: Type: Text/Plain, Size: 2119 bytes --]

Ryan Hill <dirtyepic@gentoo.org> said:
> On Fri, 04 Jun 2010 17:11:45 +0200
> 
> "Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:
> > What do you think about doing the following change in
> > /usr/portage/profiles/targets/developer/make.defaults:
> > 
> > replace "test" with "test-fail-continue" to make it just less
> > frustrating (we still have a lot of test failures)
> > 
> > Hopefully that will also make more of us use the developer profile,
> > and detect test failures.
> > 
> > What do you think?
> 
> I would say it's an improvement only because it might prevent one or
> two people from completely disabling it first chance they get. :)
> 
> IMO, test failures should be given the same status as build failures.
> Packages shouldn't be commited until they're fixed or bypassed. 
> Following that reasoning, FEATURES="test" is the correct setting for
> the dev profile. It _should_ be annoying when you hit it, that's the
> point.  Fix it!  What's the point of even having a test suite if it
> always fails?  You'd be better off to RESTRICT it and save yourself
> some bug reports from me and all the other users you're foisting build
> errors on.
> 
> But in the real world it seems it's just never going to happen.  I've
> been arguing this for years but people simply don't care.  It doesn't
> help that we don't have any finer control than "on" or "off".  I'd
> like to be able to say things like "these tests should only be run by
> developers" or "some failures are normal" or "hope you have 10 hours
> to run this" or "don't run these as root" or "don't run tests on arm"
> etc etc.  I'd like a pony while I'm at it.
> 
> Sorry about the rant.  This is one of my biggest long-standing
> annoyances.
> 
> Um, so yeah.  For it!

as it seems, there is disagreement about the issue among developers. 
Perhaps the council would like to settle this, so that we can go on with 
our lives.

i do agree, that all packages should build successfully including the test 
phase. RESTRICTing the test and an open bug when this is not the case.

thanks
Thilo



[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Jeroen Roovers]

On Mon, 7 Jun 2010 12:10:04 +0200
Thilo Bangert <bangert@gentoo.org> wrote:

> i do agree, that all packages should build successfully including the
> test phase. RESTRICTing the test and an open bug when this is not the
> case.

I see more and more calls for either 1) "fixing the test suite", as if
that is suddenly not an UPSTREAM issue but the ebuilds' maintainers' or
2) setting RESTRICT=test, which would have everyone ignore the
useful aspects of a package's test suite.

The main problem with RESTRICT=test is that it's too restrictive - it
prevents a normal emerge(1) or ebuild(1) run from entering the test
phase at all, with no way to work around it, except by editing the
ebuild to remove the restriction.

======
marga /keeps/gentoo/local/app-portage/foobar # ebuild foobar-1.ebuild
test Forcing test.
 * checking ebuild
checksums ;-) ...                                            [ ok ]
 * checking auxfile
checksums ;-) ...                                           [ ok ]
 * checking miscfile
checksums ;-) ...                                          [ ok ]
 * CPV:  app-portage/foobar-1
 * REPO: JeR
 * USE:  elibc_glibc kernel_linux ppc test userland_GNU
>>> Unpacking source...
>>> Source unpacked in /var/tmp/portage/app-portage/foobar-1/work
>>> Compiling source in /var/tmp/portage/app-portage/foobar-1/work ...
>>> Source compiled.
 * Skipping make test/check due to ebuild restriction.
>>> Test phase [explicitly disabled]: app-portage/foobar-1
marga /keeps/gentoo/local/app-portage/foobar # cat foobar-1.ebuild 
# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

SLOT="0"
RESTRICT="test"
======

If you believe that test suites are a Good Thing, then you should view
RESTRICT=test as the ultimate solution to fix the problem of a broken
test suite that will never be fixed. Now, in case a test suite could
actually be destructive, dangerous to run on an unprepared system, then
there RESTRICT=test is a fine solution.

When instead a test suite should do a SKIP but erroneously does a FAIL,
then RESTRICT=test is not the solution. Fixing the test suite is, but
then that's not the maintainer's problem, but upstream's. Oddly enough
we have QA checks in place (for ICEs, for instance) that direct users
directly to upstream (through the HOMEPAGE variable), when it's
suddenly the maintainer's problem if a package fails its test suite
(because of FEATURES=userpriv or a missing kernel feature, perhaps -
nothing the maintainer can prepare the user's system for). There's
currently no way to describe the test suite's requirements to the user
except by building in many checks or perhaps by simply listing them in
an ewarn.

Since the variable will be passed on to every next ebuild you're stuck
with it, unless you are prepared to edit out RESTRICT=test, see if the
new version still fails, then edit it in again when you find nothing
has been fixed. This in turn doesn't make for easy ebuild maintenance.

There's something wrong with the way we do test phases, the way some
of us rely on them to foretell the stability or quality of a package
version, and the way we stop test suites from failing (by
only having a binary RESTRICT=test).

We could easily extend metadata.xml to describe the test phase to the
developer and user right now, for one thing, and aim for a more
automated approach to fixing the binary problem in the future.

Another solution is to change how emerge(1) and ebuild(1) deal with
RESTRICT=test. On the one hand FEATURES=test should be enabled
by testers and users with caution, as many test suites simply don't do
what you might think they do - there is no guarantee that a program
will run well in the Real World by merely satisfying its (limited?) test
conditions. With that in mind, RESTRICT=test should perhaps not block
testing the way that it does now.


     jer

Re: [gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

On 6/7/10 12:10 PM, Thilo Bangert wrote:
> as it seems, there is disagreement about the issue among developers. 
> Perhaps the council would like to settle this, so that we can go on with 
> our lives.
> 
> i do agree, that all packages should build successfully including the test 
> phase. RESTRICTing the test and an open bug when this is not the case.

Thilo, I'm trying not to touch the issue of test failures at all. Also,
my feeling is that although not too many people commented here, we have
a consensus in favor of my suggestion.

And my suggestion is to replace FEATURES="test" in the developer profile
with FEATURES="test test-fail-continue" to make us developers actually
use it.

I've been running my dev box with FEATURES="test test-fail-continue" and
I'm happy. I'm filing bugs for the test failures, but the packages get
installed regardless of that (i.e. I get the work done).

If it does answer your doubts, then great. If it doesn't, please let me
know what I'm missing.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Thilo Bangert]

[-- Attachment #1: Type: Text/Plain, Size: 611 bytes --]

you make valid points regarding the overall improvement of the handling of 
test suites. I am not opposed to something like that being done...

it still seems like there is agreement around the fact that something 
needs to be done about src_test. currently you cant run a system which 
generally enables this phase.

however, the fact that different people see different problems, should not 
stop us of from solving any problem. so as a small incremental step, the 
method of RESTRICTing failing tests is acceptable despite the negative 
consquences you mention.

thanks

kind regards
Thilo


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]

On 6/4/10 5:11 PM, "Paweł Hajdan, Jr." wrote:
> What do you think about doing the following change in
> /usr/portage/profiles/targets/developer/make.defaults:

The following change has now landed in CVS:

Index: targets/developer/make.defaults
===================================================================
RCS file: /var/cvsroot/gentoo-x86/profiles/targets/developer/make.defaults,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -B -r1.3 -r1.4
--- targets/developer/make.defaults	4 Oct 2009 09:44:27 -0000	1.3
+++ targets/developer/make.defaults	9 Jun 2010 17:03:37 -0000	1.4
@@ -1,8 +1,8 @@
 # Copyright 1999-2008 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
-# $Header:
/var/cvsroot/gentoo-x86/profiles/targets/developer/make.defaults,v 1.3
2009/10/04 09:44:27 ssuominen Exp $
+# $Header:
/var/cvsroot/gentoo-x86/profiles/targets/developer/make.defaults,v 1.4
2010/06/09 17:03:37 phajdan.jr Exp $

-FEATURES="collision-protect cvs digest multilib-strict sign splitdebug
stricter test userpriv usersandbox"
+FEATURES="collision-protect cvs digest multilib-strict sign splitdebug
stricter test test-fail-continue userpriv usersandbox"

 # Disable branding (from desktop)
 USE="-branding"


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

On Wed, Jun 09, 2010 at 07:08:44PM +0200, "Paweee Hajdan, Jr." wrote:
> On 6/4/10 5:11 PM, "Paweł Hajdan, Jr." wrote:
> > What do you think about doing the following change in
> > /usr/portage/profiles/targets/developer/make.defaults:
> 
> The following change has now landed in CVS:

I'd suggest a dev-announce in the future on that one w/ some lead 
time... dev profile is admittedly 'dev', but changes that can induce 
an hour build taking a day due to tests being ran is usually good to 
give a heads up on.

Aside from that, change makes sense.
~harring


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Samuli Suominen]

On 06/10/2010 07:28 PM, Brian Harring wrote:
> On Wed, Jun 09, 2010 at 07:08:44PM +0200, "Paweee Hajdan, Jr." wrote:
>> On 6/4/10 5:11 PM, "Paweł Hajdan, Jr." wrote:
>>> What do you think about doing the following change in
>>> /usr/portage/profiles/targets/developer/make.defaults:
>>
>> The following change has now landed in CVS:
> 
> I'd suggest a dev-announce in the future on that one w/ some lead 
> time... dev profile is admittedly 'dev', but changes that can induce 
> an hour build taking a day due to tests being ran is usually good to 
> give a heads up on.
> 
> Aside from that, change makes sense.
> ~harring
> 

not really, since FEATURES="test" was already there...

http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/profiles/targets/developer/make.defaults?r1=1.3&r2=1.4

Re: [gentoo-dev] RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Thu, Jun 10, 2010 at 07:36:51PM +0300, Samuli Suominen wrote:
> On 06/10/2010 07:28 PM, Brian Harring wrote:
> > On Wed, Jun 09, 2010 at 07:08:44PM +0200, "Paweee Hajdan, Jr." wrote:
> >> On 6/4/10 5:11 PM, "Paweł Hajdan, Jr." wrote:
> >>> What do you think about doing the following change in
> >>> /usr/portage/profiles/targets/developer/make.defaults:
> >>
> >> The following change has now landed in CVS:
> > 
> > I'd suggest a dev-announce in the future on that one w/ some lead 
> > time... dev profile is admittedly 'dev', but changes that can induce 
> > an hour build taking a day due to tests being ran is usually good to 
> > give a heads up on.
> > 
> > Aside from that, change makes sense.
> > ~harring
> > 
> 
> not really, since FEATURES="test" was already there...
> 
> http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/profiles/targets/developer/make.defaults?r1=1.3&r2=1.4

Nard, good catch.

Nevermind ;)
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: RFC: changing the developer profile: FEATURES="test" -> FEATURES="test-fail-continue"

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

On Mon, 7 Jun 2010 14:02:50 +0200
Jeroen Roovers <jer@gentoo.org> wrote:

> I see more and more calls for either 1) "fixing the test suite", as if
> that is suddenly not an UPSTREAM issue but the ebuilds' maintainers'

> When instead a test suite should do a SKIP but erroneously does a FAIL,
> then RESTRICT=test is not the solution. Fixing the test suite is, but
> then that's not the maintainer's problem, but upstream's. Oddly enough
> we have QA checks in place (for ICEs, for instance) that direct users
> directly to upstream (through the HOMEPAGE variable), when it's
> suddenly the maintainer's problem if a package fails its test suite
> (because of FEATURES=userpriv or a missing kernel feature, perhaps -
> nothing the maintainer can prepare the user's system for).

I'm having trouble understanding how you can say fixing build failures is
part of a maintainer's duties while fixing test failures is upstream's
problem.  A test failure _is_ a build failure.  Yes, we should get it fixed
upstream, just like any other bug.  Packages can fail to compile with
userpriv or missing kernel features too.  Should we also send users directly
to upstream in these cases?  Can you explain the difference?

I agree fully with all your other points.  


-- 
fonts,                               there's a hole in my neighbourhood
gcc-porting,                  down which of late i cannot help but fall
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]