[gentoo-dev] FEATURES=test, sys-devel/gcc ignored test failures

[gentoo-dev] FEATURES=test, sys-devel/gcc ignored test failures

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

sys-devel/gcc runs tests, but the results are ignored and I remember the
tests fail most of the time.

Because the tests take long time to run and fail anyway (I understand
it's non-trivial to fix those on Gentoo side), I wonder whether it makes
sense to run them at all:

toolchain.eclass:

gcc_src_test() {
    cd "${WORKDIR}"/build
    emake -j1 -k check || ewarn "check failed and that sucks :("
}

My suggestion is to make the src_test empty (I think the default one
still calls make). I can produce a patch if needed.

What do you think?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] FEATURES=test, sys-devel/gcc ignored test failures

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 376 bytes --]

Excerpts from Paweł Hajdan, Jr.'s message of Mon Mar 21 13:07:33 +0100 2011:
> My suggestion is to make the src_test empty (I think the default one
> still calls make). I can produce a patch if needed.
> 
> What do you think?

Maybe restrict? https://bugs.gentoo.org/298014
-- 
Amadeusz Żołnowski

PGP key fpr: C700 CEDE 0C18 212E 49DA  4653 F013 4531 E1DB FAB5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: FEATURES=test, sys-devel/gcc ignored test failures

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1596 bytes --]

On Mon, 21 Mar 2011 13:07:33 +0100
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:

> sys-devel/gcc runs tests, but the results are ignored and I remember the
> tests fail most of the time.

s/most/all
 
> Because the tests take long time to run and fail anyway (I understand
> it's non-trivial to fix those on Gentoo side), I wonder whether it makes
> sense to run them at all:

It does to me, I use them all the time. ;)  The important part is that we
install the test results, which can then be used for regression testing when
rolling patchsets.

> toolchain.eclass:
> 
> gcc_src_test() {
>     cd "${WORKDIR}"/build
>     emake -j1 -k check || ewarn "check failed and that sucks :("
> }
> 
> My suggestion is to make the src_test empty (I think the default one
> still calls make). I can produce a patch if needed.
> 
> What do you think?

I think that glibc and gcc tests and other testsuites that nearly always
fail shouldn't be run for the average user but should still be easily
accessible in a standard way.  I think we need a more finely grained test
setup, where we can say tests are "expensive" or "interesting only to
developers" or "known to fail", and let people opt-in to these on a
per-package basis. Right now you always have to opt-out using
package.use.mask which "works" but is unintuitive.


-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: FEATURES=test, sys-devel/gcc ignored test failures

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

On 3/21/11 11:02 PM, Ryan Hill wrote:
> It does to me, I use them all the time. ;)  The important part is that we
> install the test results, which can then be used for regression testing when
> rolling patchsets.

I see, it makes sense. I guess you're comparing the test results before
and after rolling patchsets and look for regressions.

> I think that glibc and gcc tests and other testsuites that nearly always
> fail shouldn't be run for the average user but should still be easily
> accessible in a standard way.  I think we need a more finely grained test
> setup, where we can say tests are "expensive" or "interesting only to
> developers" or "known to fail", and let people opt-in to these on a
> per-package basis. Right now you always have to opt-out using
> package.use.mask which "works" but is unintuitive.

My main point is that the developer profile has FEATURES=test, and also
arch testers and developers run with FEATURES=test. Being able to
quickly rebuild gcc, glibc and others is a win.

I'm trying to understand the problem better - do you know what causes
those test failures? I don't expect a "complete" answer because that'd
probably be a half of actually fixing the failures.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: [gentoo-dev] Re: FEATURES=test, sys-devel/gcc ignored test failures

[Matt Turner]

On Tue, Mar 22, 2011 at 12:09 PM, "Paweł Hajdan, Jr."
<phajdan.jr@gentoo.org> wrote:
> I'm trying to understand the problem better - do you know what causes
> those test failures? I don't expect a "complete" answer because that'd
> probably be a half of actually fixing the failures.

They both have huge test suites covering massive amounts of code. It's
only natural that some of the tests fail from version to version.

[gentoo-dev] Re: FEATURES=test, sys-devel/gcc ignored test failures

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 2706 bytes --]

On Tue, 22 Mar 2011 13:09:46 +0100
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:

> On 3/21/11 11:02 PM, Ryan Hill wrote:
> > It does to me, I use them all the time. ;)  The important part is that we
> > install the test results, which can then be used for regression testing when
> > rolling patchsets.
> 
> I see, it makes sense. I guess you're comparing the test results before
> and after rolling patchsets and look for regressions.
> 
> > I think that glibc and gcc tests and other testsuites that nearly always
> > fail shouldn't be run for the average user but should still be easily
> > accessible in a standard way.  I think we need a more finely grained test
> > setup, where we can say tests are "expensive" or "interesting only to
> > developers" or "known to fail", and let people opt-in to these on a
> > per-package basis. Right now you always have to opt-out using
> > package.use.mask which "works" but is unintuitive.
> 
> My main point is that the developer profile has FEATURES=test, and also
> arch testers and developers run with FEATURES=test. Being able to
> quickly rebuild gcc, glibc and others is a win.

Yes, I'm agreeing with you.  I'd like these off by default too.  We need a
standard way of enabling them however.  USE="test-dev" or something.  I
complain about this about once a year or so. ;)  Maybe I should just do it.

In the meantime:

echo -e 'sys-libs/glibc test\nsys-devel/gcc test' \
  >> /etc/portage/profile/package.use.mask
 
> I'm trying to understand the problem better - do you know what causes
> those test failures? I don't expect a "complete" answer because that'd
> probably be a half of actually fixing the failures.

The GCC testsuite isn't designed to pass.  It's designed to be a
regression test.  Check before and after you apply a patch, make sure you
don't cock things up.  From http://gcc.gnu.org/install/test.html: "It is
normal for some tests to report unexpected failures. At the current time the
testing harness does not allow fine grained control over whether or not a test
is expected to fail."  Look at http://gcc.gnu.org/buildstat.html and you'll
see this in practice.

One thing I know of that causes a bunch of failures is the fact that we
enable -Wformat, -Wformat-security, and -Wtrampolines by default.  Any
additional output during a test = fail.  I patched these recently for 4.5
though so they shouldn't be a problem going forward.


-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]