[gentoo-dev] RFC: Removing src_test from www-client/chromium

[gentoo-dev] RFC: Removing src_test from www-client/chromium

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

It's more of a project-internal decision IMHO, but just wanted to get
feedback from the larger community.

Currently 11 out of 27 bugs assigned to chromium.g.o are related to test
failures.

I don't remember a single case where a test failure would point to a
real bug in our package.

I'm seriously considering just removing src_test to make the package
more maintainable (less code, less bugs filed, can focus on things that
*do* impact our users).

If you decide to comment in favor of keeping src_test, please consider
volunteering to help us with the bugs.

Feel free to suggest solutions that fall somewhere in between - e.g.
having src_test but not excluding any tests there and using
RESTRICT=test, so that someone who really wants to run the tests FYI can
do so.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1242 bytes --]

On Tue, 27 May 2014 09:02:37 +0200
""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:

> I'm seriously considering just removing src_test to make the package
> more maintainable (less code, less bugs filed, can focus on things
> that *do* impact our users).
> 
> If you decide to comment in favor of keeping src_test, please consider
> volunteering to help us with the bugs.
> 
> Feel free to suggest solutions that fall somewhere in between - e.g.
> having src_test but not excluding any tests there and using
> RESTRICT=test, so that someone who really wants to run the tests FYI
> can do so.

From a "test it for proper maintenance" point of view this makes some
sense; however, from a "test it for proper usability" point of view
this might become a bit questionable. Some users like to run these
tests in order to know their browser will work on their system; perhaps
even going further than that, it helps more users report bugs upstream.

You could ask the users to file the tests related bugs upstream.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 27/05/14 04:05 AM, Tom Wijsman wrote:
> On Tue, 27 May 2014 09:02:37 +0200 ""Paweł Hajdan, Jr.""
> <phajdan.jr@gentoo.org> wrote:
> 
>> I'm seriously considering just removing src_test to make the
>> package more maintainable (less code, less bugs filed, can focus
>> on things that *do* impact our users).
>> 
>> If you decide to comment in favor of keeping src_test, please
>> consider volunteering to help us with the bugs.
>> 
>> Feel free to suggest solutions that fall somewhere in between -
>> e.g. having src_test but not excluding any tests there and using 
>> RESTRICT=test, so that someone who really wants to run the tests
>> FYI can do so.
> 
> From a "test it for proper maintenance" point of view this makes
> some sense; however, from a "test it for proper usability" point of
> view this might become a bit questionable. Some users like to run
> these tests in order to know their browser will work on their
> system; perhaps even going further than that, it helps more users
> report bugs upstream.
> 
> You could ask the users to file the tests related bugs upstream.
> 

I don't know how much chromium is built and tested on lesser-used
arches (ie: arm, hppa, ia64, etc), but if there are dev's that try and
maintain these keywords that aren't in the team, it might be a good
idea to leave src_test in place, for them.  However, you could always
wrap the actual contents of src_test with an "if [[ -n
${I_KNOW_WHAT_IM_DOING} ]] " or similar to keep tests from running
when the general userbase is trying to emerge it and happens to have
FEATURES="test" globally enabled..

An ewarn telling users that if they really do want to run these tests,
they can set the variable in make.conf or package.env or similar, and
also specifying that the tests are skipped due to only being useful
for upstream developers, may suffice...  thoughts?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlOEnKkACgkQ2ugaI38ACPCBswD8CAx2cK2UH+/IzysGPiQjOEzF
n8Sxv3ZbtKh9+aBFv9IBAIMhfmgABYnUJlGUg/+mPHOY1d9XDRfM9WQskiJBc2Xk
=nWr/
-----END PGP SIGNATURE-----

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

[Jeroen Roovers]

On Tue, 27 May 2014 10:09:45 -0400
Ian Stakenvicius <axs@gentoo.org> wrote:

> I don't know how much chromium is built and tested on lesser-used
> arches (ie: arm, hppa, ia64, etc)

No version of webkit/blink is known to work on HPPA, particularly
because the JS engine is broken on systems where the stack grows up. I
assume the same is true for IA64. Also, chromium compile time would be
awful, and run time would likely be very slow due to a lack of
optimisation.


     jer

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

[Steev Klimaszewski]

On Tue, 2014-05-27 at 10:09 -0400, Ian Stakenvicius wrote:
> I don't know how much chromium is built and tested on lesser-used
> arches (ie: arm, hppa, ia64, etc), but if there are dev's that try and
> maintain these keywords that aren't in the team, it might be a good
> idea to leave src_test in place, for them. 

Well, Chromium is the only browser I use on my Chromebook - it takes
about 5 hours to compile natively, but I don't run the tests on it.  If
it doesn't work... well, it doesn't work...

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]

On Thu, 29 May 2014 04:09:22 -0500
Steev Klimaszewski <steev@gentoo.org> wrote:

> On Tue, 2014-05-27 at 10:09 -0400, Ian Stakenvicius wrote:
> > I don't know how much chromium is built and tested on lesser-used
> > arches (ie: arm, hppa, ia64, etc), but if there are dev's that try
> > and maintain these keywords that aren't in the team, it might be a
> > good idea to leave src_test in place, for them. 
> 
> Well, Chromium is the only browser I use on my Chromebook - it takes
> about 5 hours to compile natively, but I don't run the tests on it.
> If it doesn't work... well, it doesn't work...

It gets fun when the scrollbar doesn't work on 5% of the sites or so;
happened to me, because a dependency of a dependency of Chromium was
broken. At which point I really hoped I would've enabled tests on them.

Can be glad that the fix didn't involve recompiling Chromium...

In general it has always worked well after a compile; but, there's every
now and then one or another annoying regression, like recent Chromium
had some font issues or some random tabs crash some versions ago and ...

If a test catches one of these, you can immediately report the problem;
if it is left untested, you'll have to do a debugging adventure instead.

While I don't run tests myself; the need for them is clear, for those
that aim for more production ready systems (eg. university network PCs).

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: RFC: Removing src_test from www-client/chromium

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1783 bytes --]

On Tue, 27 May 2014 09:02:37 +0200
""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:

> It's more of a project-internal decision IMHO, but just wanted to get
> feedback from the larger community.
> 
> Currently 11 out of 27 bugs assigned to chromium.g.o are related to test
> failures.
> 
> I don't remember a single case where a test failure would point to a
> real bug in our package.
> 
> I'm seriously considering just removing src_test to make the package
> more maintainable (less code, less bugs filed, can focus on things that
> *do* impact our users).
> 
> If you decide to comment in favor of keeping src_test, please consider
> volunteering to help us with the bugs.
> 
> Feel free to suggest solutions that fall somewhere in between - e.g.
> having src_test but not excluding any tests there and using
> RESTRICT=test, so that someone who really wants to run the tests FYI can
> do so.

I've said it before, but I think that by having packages in the tree that we
know consistently fail their testsuites, we create a situation where we are
worse off than if we simply disabled the tests for that package.

Let's look at what enabling tests gets you right now:

- additional dependencies
- longer compile times
- blockers
- lots of scrolly output
- devs ignore your bug reports
- absolutely no peace of mind because every third package fails for no good
  reason

If I wasn't a dev I would have turned it off long long ago (and I suspect many
already have).

Test coverage is a good thing, so it'd be nice to give people an actual
incentive to do it.

So +1.


-- 
Ryan Hill                        psn: dirtyepic_sk
   gcc-porting/toolchain/wxwidgets @ gentoo.org

47C3 6D62 4864 0E49 8E9E  7F92 ED38 BD49 957A 8463

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

On 5/29/14, 12:46 PM, Tom Wijsman wrote:
> In general it has always worked well after a compile; but, there's every
> now and then one or another annoying regression, like recent Chromium
> had some font issues or some random tabs crash some versions ago and ...
> 
> If a test catches one of these, you can immediately report the problem;
> if it is left untested, you'll have to do a debugging adventure instead.

This is one of my points: I don't remember a single chromium bug filed
in Gentoo that would be caught by a test or that a failing test actually
detected.

By the way, I don't remember seeing many reports about font issues or
tab crashes. Please make sure to file them when they occur, or just
point me to them in case I somehow missed them.

> While I don't run tests myself; the need for them is clear, for those
> that aim for more production ready systems (eg. university network PCs).

This seems too theoretical to me. I'd be fine with someone volunteering
to maintain chromium's src_test in Gentoo. Unless we have such a person
though, it seems to mostly take valuable focus away from bugs that
definitely *do* affect our users, for no provable benefit for Gentoo.

Paweł



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2138 bytes --]

On Sat, 31 May 2014 19:50:20 +0200
""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:

> On 5/29/14, 12:46 PM, Tom Wijsman wrote:
> > In general it has always worked well after a compile; but, there's
> > every now and then one or another annoying regression, like recent
> > Chromium had some font issues or some random tabs crash some
> > versions ago and ...
> > 
> > If a test catches one of these, you can immediately report the
> > problem; if it is left untested, you'll have to do a debugging
> > adventure instead.
> 
> This is one of my points: I don't remember a single chromium bug filed
> in Gentoo that would be caught by a test or that a failing test
> actually detected.

Your point covers the lack of tests, or tests that are non-fatal;
however, it doesn't cover tests that are fatal, what if they fail?

> By the way, I don't remember seeing many reports about font issues or
> tab crashes. Please make sure to file them when they occur, or just
> point me to them in case I somehow missed them.

They usually go straight to upstream, though I've managed to somehow
fix it up; as for Gentoo, some people create forum threads about them.

(One was due to a library compiled with a less common flag, the other
due to fontconfig being a regression magnet; both fun to debug, the
former a test wolud've caught, the latter is due to the lack thereof)

> > While I don't run tests myself; the need for them is clear, for
> > those that aim for more production ready systems (eg. university
> > network PCs).
> 
> This seems too theoretical to me. I'd be fine with someone
> volunteering to maintain chromium's src_test in Gentoo. Unless we
> have such a person though, it seems to mostly take valuable focus
> away from bugs that definitely *do* affect our users, for no provable
> benefit for Gentoo.

What about provable benefit for upstream? Does upstream /dev/null them?

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 2892 bytes --]

On 5/31/14, 8:30 PM, Tom Wijsman wrote:
> On Sat, 31 May 2014 19:50:20 +0200
> ""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:
>> This is one of my points: I don't remember a single chromium bug filed
>> in Gentoo that would be caught by a test or that a failing test
>> actually detected.
> 
> Your point covers the lack of tests, or tests that are non-fatal;
> however, it doesn't cover tests that are fatal, what if they fail?

I'm confused by the distinction of fatal and non-fatal tests. Neither
upstream nor the Gentoo chromium package makes that distinction.

>> By the way, I don't remember seeing many reports about font issues or
>> tab crashes. Please make sure to file them when they occur, or just
>> point me to them in case I somehow missed them.
> 
> They usually go straight to upstream, though I've managed to somehow
> fix it up; as for Gentoo, some people create forum threads about them.

I can't speak for other people, but please consider reporting issues to
Gentoo first. Our bug queue is under 30 bugs, while upstream is several
thousand. Once we can confirm a bug clearly belongs to upstream, we can
tell the reporter to file bug upstream or do that ourselves, but keeping
Gentoo out of the loop seems to increase the time needed to fix a bug.

> (One was due to a library compiled with a less common flag, the other
> due to fontconfig being a regression magnet; both fun to debug, the
> former a test wolud've caught, the latter is due to the lack thereof)

If there's something that could be changed e.g. in chromium's
dependencies, please let me know. There are cases where we require
certain USE flags to be set on dependencies for things to work properly.

About the issue that a test would have caught: was that a chromium test?
If so, which one?

>>> While I don't run tests myself; the need for them is clear, for
>>> those that aim for more production ready systems (eg. university
>>> network PCs).
>>
>> This seems too theoretical to me. I'd be fine with someone
>> volunteering to maintain chromium's src_test in Gentoo. Unless we
>> have such a person though, it seems to mostly take valuable focus
>> away from bugs that definitely *do* affect our users, for no provable
>> benefit for Gentoo.
> 
> What about provable benefit for upstream? Does upstream /dev/null them?

Effectively yes. For an example see
https://bugs.gentoo.org/show_bug.cgi?id=497512 and
https://groups.google.com/a/chromium.org/d/msg/chromium-dev/OdX7ShsOqsM/-R9sexJAEa4J

The failure is not Gentoo-specific, and is not a bug in code but problem
with the test (it makes assumptions about internal glibc
implementation). It actually fails on the latest Ubuntu LTS Trusty Tahr,
which means the test will have to be fixed or disabled upstream. But 6
months of no reaction is not really a good sign IMHO.

Paweł



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 841 bytes --]

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 4685 bytes --]

On Sun, 01 Jun 2014 15:41:35 +0200
""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:

> On 5/31/14, 8:30 PM, Tom Wijsman wrote:
> > On Sat, 31 May 2014 19:50:20 +0200
> > ""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:
> >> This is one of my points: I don't remember a single chromium bug
> >> filed in Gentoo that would be caught by a test or that a failing
> >> test actually detected.
> > 
> > Your point covers the lack of tests, or tests that are non-fatal;
> > however, it doesn't cover tests that are fatal, what if they fail?
> 
> I'm confused by the distinction of fatal and non-fatal tests. Neither
> upstream nor the Gentoo chromium package makes that distinction.

Tests that break parts of your browser; you don't notice such tests in
what was said, until one day such a test does break one or another way.

> >> By the way, I don't remember seeing many reports about font issues
> >> or tab crashes. Please make sure to file them when they occur, or
> >> just point me to them in case I somehow missed them.
> > 
> > They usually go straight to upstream, though I've managed to somehow
> > fix it up; as for Gentoo, some people create forum threads about
> > them.
> 
> I can't speak for other people, but please consider reporting issues
> to Gentoo first. Our bug queue is under 30 bugs, while upstream is
> several thousand. Once we can confirm a bug clearly belongs to
> upstream, we can tell the reporter to file bug upstream or do that
> ourselves, but keeping Gentoo out of the loop seems to increase the
> time needed to fix a bug.

This confuses me; your thread opener seems to suggest you have too much
bugs, whereas this one seems to suggest you don't have enough bugs.

What's the purpose of disabling src_test if bug count isn't a problem?
Iotw, why are you making a project-internal decision here?

(Your last two paragraphs may respond to this; in which case, nvm)

> > (One was due to a library compiled with a less common flag, the
> > other due to fontconfig being a regression magnet; both fun to
> > debug, the former a test wolud've caught, the latter is due to the
> > lack thereof)
> 
> If there's something that could be changed e.g. in chromium's
> dependencies, please let me know. There are cases where we require
> certain USE flags to be set on dependencies for things to work
> properly.

Wasn't the case of a different version or USE flag state; but that is
the case one or another day, I'll let you know.

> About the issue that a test would have caught: was that a chromium
> test? If so, which one?

Unable to tell, since I don't run them.

> >>> While I don't run tests myself; the need for them is clear, for
> >>> those that aim for more production ready systems (eg. university
> >>> network PCs).
> >>
> >> This seems too theoretical to me. I'd be fine with someone
> >> volunteering to maintain chromium's src_test in Gentoo. Unless we
> >> have such a person though, it seems to mostly take valuable focus
> >> away from bugs that definitely *do* affect our users, for no
> >> provable benefit for Gentoo.
> > 
> > What about provable benefit for upstream? Does upstream /dev/null
> > them?
> 
> Effectively yes. For an example see
> https://bugs.gentoo.org/show_bug.cgi?id=497512 and
> https://groups.google.com/a/chromium.org/d/msg/chromium-dev/OdX7ShsOqsM/-R9sexJAEa4J
> 
> The failure is not Gentoo-specific, and is not a bug in code but
> problem with the test (it makes assumptions about internal glibc
> implementation). It actually fails on the latest Ubuntu LTS Trusty
> Tahr, which means the test will have to be fixed or disabled
> upstream. But 6 months of no reaction is not really a good sign IMHO.
> 
> Paweł

Yeah; if failing tests on distributions aren't getting fixed by
upstream, then there's indeed no point to keep them running.

Though; on the other hand, one has to consider that this acts like a
priority queue and therefore tests that fail on most distributions would
get fixed before tests that fail on just one or two distributions.

It's a tricky decision to drop them; but it's not an irreversible
decision, thus a reevaluation in 5 years from now could be possible. If
that reevaluation then shows a responsive upstream, reconsider src_test.

Don't mind me, I've played devil's advocate to explore the reasoning;
go ahead if you want to, given it barely result in fatal test failures.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] RFC: Removing src_test from www-client/chromium

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1792 bytes --]

On 6/1/14, 4:41 PM, Tom Wijsman wrote:
>> I can't speak for other people, but please consider reporting issues
>> to Gentoo first. Our bug queue is under 30 bugs, while upstream is
>> several thousand. Once we can confirm a bug clearly belongs to
>> upstream, we can tell the reporter to file bug upstream or do that
>> ourselves, but keeping Gentoo out of the loop seems to increase the
>> time needed to fix a bug.
> 
> This confuses me; your thread opener seems to suggest you have too much
> bugs, whereas this one seems to suggest you don't have enough bugs.

Encouraging people to report bugs to Gentoo is not the same as saying we
don't have enough bugs. My goal is to make sure the package works well,
and I can't fix problems I don't know about.

> Iotw, why are you making a project-internal decision here?

Please refer to my first post - just checking whether there is something
I may have missed, or some volunteers to help us with the tests.

> Yeah; if failing tests on distributions aren't getting fixed by
> upstream, then there's indeed no point to keep them running.
> 
> Though; on the other hand, one has to consider that this acts like a
> priority queue and therefore tests that fail on most distributions would
> get fixed before tests that fail on just one or two distributions.

I haven't seen that happening for Chromium.

> It's a tricky decision to drop them; but it's not an irreversible
> decision, thus a reevaluation in 5 years from now could be possible. If
> that reevaluation then shows a responsive upstream, reconsider src_test.

Yes, totally agreed.

> Don't mind me, I've played devil's advocate to explore the reasoning;
> go ahead if you want to, given it barely result in fatal test failures.

OK.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 841 bytes --]