[gentoo-dev] ccache causing problems

[gentoo-dev] ccache causing problems

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 441 bytes --]

I've just seen two serious problems with ccache in a short period of
time. In both, webkit seems to be somehow involved (it has a complex
build process).

See <https://bugs.gentoo.org/show_bug.cgi?id=316657> and
<https://forums.gentoo.org/viewtopic.php?p=6262495#6262495>.

I'm afraid we probably can't get enough data from the above to produce a
good bug report for ccache.

What actions would you suggest?

Paweł Hajdan jr


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] ccache causing problems

[Robin H. Johnson]

On Thu, Apr 29, 2010 at 09:06:51AM +0200, "Paweł Hajdan, Jr." wrote:
> What actions would you suggest?
Have your user do a binary search of the ccache dir to find which cache
file is causing the problem, by restoring from his backup then renaming
half the directories each time.

Then tie that to what the input file should have generated it.

Most of the problems I've seen have been due to a disk or OS error
causing the cache output file to be corrupt or empty, instead of the
expected object code.

ccache itself hasn't been the problem, but unreliable hardware has.
Provably by removing the corrupt cache files, then running with ccache a
few more times, and having everything work perfectly.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-dev] ccache causing problems

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

On 4/29/10 9:41 AM, Robin H. Johnson wrote:
> On Thu, Apr 29, 2010 at 09:06:51AM +0200, "Paweł Hajdan, Jr." wrote:
>> What actions would you suggest?
> Have your user do a binary search of the ccache dir to find which cache
> file is causing the problem, by restoring from his backup then renaming
> half the directories each time.

It may be difficult, see
<https://bugs.gentoo.org/show_bug.cgi?id=316657#c8>. Do we have some
docs on the web with detailed instructions how to do that?

> ccache itself hasn't been the problem, but unreliable hardware has.
> Provably by removing the corrupt cache files, then running with ccache a
> few more times, and having everything work perfectly.

I see. However, I'd consider not detecting the corruption a bug.

Paweł Hajdan jr


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] ccache causing problems

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

On Thu, 29 Apr 2010 09:48:28 +0200
"Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:
> > ccache itself hasn't been the problem, but unreliable hardware has.
> > Provably by removing the corrupt cache files, then running with
> > ccache a few more times, and having everything work perfectly.
> 
> I see. However, I'd consider not detecting the corruption a bug.

gcc also doesn't detect most hardware faults. Nor does glibc. Nor does
the kernel. There's very little software out there that's designed to
run correctly even on broken hardware, and the little that there is is
mostly keeping planes flying.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] ccache causing problems

[Robin H. Johnson]

On Thu, Apr 29, 2010 at 09:48:28AM +0200, "Paweł Hajdan, Jr." wrote:
> On 4/29/10 9:41 AM, Robin H. Johnson wrote:
> > On Thu, Apr 29, 2010 at 09:06:51AM +0200, "Paweł Hajdan, Jr." wrote:
> >> What actions would you suggest?
> > Have your user do a binary search of the ccache dir to find which cache
> > file is causing the problem, by restoring from his backup then renaming
> > half the directories each time.
> 
> It may be difficult, see
> <https://bugs.gentoo.org/show_bug.cgi?id=316657#c8>. Do we have some
> docs on the web with detailed instructions how to do that?
It's a depth-2, hex-fanout directory structure.
$CCACHE_DIR/[0-9a-f]/[0-9a-f].

Just start with renaming/moving subsets of half of the directories in
the first level, until you hit the problem.

Alternatively, turn on the ccache debug log, using CCACHE_LOGFILE in
make.conf, AND set MAKEOPTS=-j1, and just rename the filenames it points
out to trace which of them is the problem.

> 
> > ccache itself hasn't been the problem, but unreliable hardware has.
> > Provably by removing the corrupt cache files, then running with ccache a
> > few more times, and having everything work perfectly.
> I see. However, I'd consider not detecting the corruption a bug.
How would it know that the file was corrupted after close? The only
possible way would be changing the format or adding another file with
the expected hash of the result file.

Patches accepted, but I think users just need to take a LOT more care of
their hardware.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-dev] ccache causing problems

[Rémi Cardona]

Le 29/04/2010 09:06, "Paweł Hajdan, Jr." a écrit :
> What actions would you suggest?

Don't use ccache. We (speaking as a former gnome herd member) have had
countless unexplained bugs due to ccache.

Now, the gnome procedure for build failures is to ask users to first
disable distcc and ccache before trying to reproduce the bug, and that
solves nearly all the weird issues that no-one else can reproduce.

Bottom line, unless you're building the same code over and over again,
don't use ccache. And even if you are, don't use it, its cache is just
too easily broken.

Cheers,

Rémi

Re: [gentoo-dev] ccache causing problems

[Jorge Manuel B. S. Vicetto]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 29-04-2010 22:16, Rémi Cardona wrote:
> Le 29/04/2010 09:06, "Paweł Hajdan, Jr." a écrit :
>> What actions would you suggest?
> 
> Don't use ccache. We (speaking as a former gnome herd member) have had
> countless unexplained bugs due to ccache.
> 
> Now, the gnome procedure for build failures is to ask users to first
> disable distcc and ccache before trying to reproduce the bug, and that
> solves nearly all the weird issues that no-one else can reproduce.

As a KDE team member, I have to say our experience is the same. We've
also had countless unexplained bugs that couldn't be reproduced without
ccache.

> Cheers,
> 
> Rémi
> 

- -- 
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJL2glUAAoJEC8ZTXQF1qEPeFcP/2GOd56KkHfLIIBncSrNQxEU
v5wlJGsoXEQOqrc30qw/HTKZCMgYuXowNbats1P3Bv7Gw0VYRuD9RMM3t4806rJI
zI8oh6BL3K6TDZCQKvVATB3y6ZBLiJeoZMyXr7HJHqleZq+Wo6qtSvBRfPq/R3g3
H17zDTrH4MzxWZAdyxXpSnmt7w3OL5Ft5CXvWruC2WYHv9lijVw1K4GxP5I17kwy
1xEg59GI4ePtd+bcpxw23C7NGhKLrOcA7ZFw67wKsl4XN807zukm1PtIcCNBWNY5
IL6aRat7uNH0zFYTsrldSunmyRi9bgjVlG9qXXUc1ekv/q65Rd5UvhXk6/3qWQB4
hixp3ExLJwZPOlFt7/TW8SwY+JQb3/ukPiNbfW63HeTErKcD1URttXRxo9YnBRsc
inrYMYN6iZyxenJGhdCeHOrGcjywNgLMIsLfoLoDdGmzRj3hrY/cJLs6QylweHXK
Wn2iiFFKn/w1NUQY27pgQbwmDbHrvgk7kh84SK42Fdow1d3g55w8x7go0NNFoOKn
3lH/s2zxl0NaIcYe/jNsBo0Q3QSlgkW1CDqABaF3KtKI7fgQoKQzKO1ZcyPMfh7L
nk4NfVbli+AMqSHVSwFDKnXQGrCizmtH2OZn8/VTqMNX3DB2ZkDHtwAaHfzrO8r1
OXD1CYl3Rxh2+KqAO4wT
=TRR4
-----END PGP SIGNATURE-----

Re: [gentoo-dev] ccache causing problems

[Gilles Dartiguelongue]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

Le vendredi 30 avril 2010 à 00:16 +0200, Rémi Cardona a écrit :
> Le 29/04/2010 09:06, "Paweł Hajdan, Jr." a écrit :
> > What actions would you suggest?
> 
> Don't use ccache. We (speaking as a former gnome herd member) have had
> countless unexplained bugs due to ccache.
> 
> Now, the gnome procedure for build failures is to ask users to first
> disable distcc and ccache before trying to reproduce the bug, and that
> solves nearly all the weird issues that no-one else can reproduce.
> 
> Bottom line, unless you're building the same code over and over again,
> don't use ccache. And even if you are, don't use it, its cache is just
> too easily broken.
> 
> Cheers,

Not that countless. Personal experience reports one bug in 8 years of
use... Just as was previously mentioned in this thread:
 * check your hardware
 * know your FEATURES and howto test failures when some are enabled

it's just like any package failing with some gcc/linked/whatever flag,
you just don't stop using that flag where it works just because of one
package, nor do you stop using gcc.

-- 
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo

[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] ccache causing problems

[Robin H. Johnson]

On Fri, Apr 30, 2010 at 01:02:48AM +0200, Gilles Dartiguelongue wrote:
> it's just like any package failing with some gcc/linked/whatever flag,
> you just don't stop using that flag where it works just because of one
> package, nor do you stop using gcc.
And for anybody that thinks they have a bug in ccache triggered by some
package.
1. backup the contents of your ccache.
2. Clear the cache.
3. Build once to populate the cache.
4. Build again to build against the cache (once or twice, should have
   identical results).

If step 4 passes, you know you had corruption in your cache.
If step 4 fails, then I'm interested in where it failed, esp. if you can
run the entire process again and repeat the problem on demand.

Note however, that while I have a high level of trust in ccache, I do
think there are more latent bugs in distcc.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[gentoo-dev] Re: ccache causing problems

[Peter Hjalmarsson]

fre 2010-04-30 klockan 16:34 +0000 skrev Robin H. Johnson:
> Note however, that while I have a high level of trust in ccache, I do
> think there are more latent bugs in distcc.
> 

Heh, yeah. I have had problems with ccache having broken cache (i.e.
stuff breaks, removing the cache and it unbreaks) on different
computers, but since I stopped use distcc together with it I have had no
such problems anymore.
Still since those experiences I always start with trying without ccache
when I hit strange bugs.