Re: [gentoo-hardened] Weird coincidental PAX crashes

[Mark Gomersbach <mark@gomersbach.nl>]

Maybe a bug somewhere else too, which combination kernel/grsec/pax was used?

On 05/09/2014 05:15 PM, Michael Orlitzky wrote:
> Last week, the LMTP daemon on our mail server (HP DL360 G6) crashed.
> People noticed that the mail stopped coming in, so I SSHed in to check
> on it, and there were some weird traces in the dmesg. While trying to
> investigate, I noticed some more badness:
> 
>   # emerge -1 openntpd
>   Calculating dependencies... done!
> 
>   >>> Verifying ebuild manifests
>   Killed
> 
> At that point I'm thinking, "hardware problem, there goes the weekend."
> Most of my tools are committing suicide so I surrender and reboot. The
> thing comes up fine and has been working ever since.
> 
> Today, another one of our web servers (HP DL360 G5?) does the same
> thing. The nightly log report was empty, because there's no syslog
> daemon running. This morning dmesg shows:
> 
>> [Fri May  9 11:00:42 2014] PAX: refcount overflow detected in: syslog-ng:21823, uid/euid: 0/0
>> [Fri May  9 11:00:42 2014] CPU: 2 PID: 21823 Comm: syslog-ng Not tainted 3.11.7-hardened-r1 #1
>> [Fri May  9 11:00:42 2014] task: ffff8802cffca080 ti: ffff8802cffca488 task.ti: ffff8802cffca488
>> [Fri May  9 11:00:42 2014] RIP: 0010:[<ffffffff810e311e>]  [<ffffffff810e311e>] 0xffffffff810e311e
>> [Fri May  9 11:00:42 2014] RSP: 0018:ffff880416f21c78  EFLAGS: 00000a96
>> [Fri May  9 11:00:42 2014] RAX: ffff88041f0048a0 RBX: ffff88041a1edf00 RCX: 0000000040276333
>> [Fri May  9 11:00:42 2014] RDX: 0000000040276332 RSI: 0000000000000000 RDI: ffff88041d858720
>> [Fri May  9 11:00:42 2014] RBP: 0000000000000008 R08: 0000000000010bc0 R09: ffff88042fb10bc0
>> [Fri May  9 11:00:42 2014] R10: 8000000000000000 R11: ffffea000fec3040 R12: ffff88041f0048a0
>> [Fri May  9 11:00:42 2014] R13: ffff88026628ef00 R14: ffff88041d858720 R15: ffff88041a1edf10
>> [Fri May  9 11:00:42 2014] FS:  0000000000000000(0000) GS:ffff88042fb00000(0000) knlGS:0000000000000000
>> [Fri May  9 11:00:42 2014] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [Fri May  9 11:00:42 2014] CR2: 0000035fb5abf850 CR3: 000000000138a000 CR4: 00000000000006b0
>> [Fri May  9 11:00:42 2014] Stack:
>> [Fri May  9 11:00:42 2014]  0000000000000000 ffffffff818dde60 ffff8804140ac100 ffff8802cffca570
>> [Fri May  9 11:00:42 2014]  ffff8802cffca080 ffff880416eb4200 ffff8802cffca080 ffffffff81052750
>> [Fri May  9 11:00:42 2014]  0000000000000000 0000000000000001 ffff88038e6260d8 ffff8802cffca598
>> [Fri May  9 11:00:42 2014] Call Trace:
>> [Fri May  9 11:00:42 2014]  [<ffffffff81052750>] ? 0xffffffff81052750
>> [Fri May  9 11:00:42 2014]  [<ffffffff81036e10>] ? 0xffffffff81036e10
>> [Fri May  9 11:00:42 2014]  [<ffffffff810371e8>] ? 0xffffffff810371e8
>> [Fri May  9 11:00:42 2014]  [<ffffffff810449cc>] ? 0xffffffff810449cc
>> [Fri May  9 11:00:42 2014]  [<ffffffff8100241f>] ? 0xffffffff8100241f
>> [Fri May  9 11:00:42 2014]  [<ffffffff81002a89>] ? 0xffffffff81002a89
>> [Fri May  9 11:00:42 2014]  [<ffffffff8137c212>] ? 0xffffffff8137c212
>> [Fri May  9 11:00:42 2014] Code: e9 68 fd 01 00 0f 1f 84 00 00 00 00 00 48 8b 43 18 48 8b 7b 10 48 8b 40 30 f0 ff 88 30 01 00 00 71 09 f0 ff 80 30 01 00 00 cd 04 <0f> b7 00 89 c2 66 81 e2 00 b0 66 81 fa 00 20 0f 84 53 ff ff ff 
>> [Fri May  9 11:00:42 2014] PAX: refcount overflow detected in: syslog-ng:21823, uid/euid: 0/0
>> [Fri May  9 11:00:42 2014] CPU: 2 PID: 21823 Comm: syslog-ng Not tainted 3.11.7-hardened-r1 #1
>> [Fri May  9 11:00:42 2014] task: ffff8802cffca080 ti: ffff8802cffca488 task.ti: ffff8802cffca488
>> [Fri May  9 11:00:42 2014] RIP: 0010:[<ffffffff810e311e>]  [<ffffffff810e311e>] 0xffffffff810e311e
>> [Fri May  9 11:00:42 2014] RSP: 0018:ffff880416f21c78  EFLAGS: 00000a96
>> [Fri May  9 11:00:42 2014] RAX: ffff88041f0048a0 RBX: ffff88041a1edc00 RCX: 0000000040c384f8
>> [Fri May  9 11:00:42 2014] RDX: 0000000040c384f7 RSI: 0000000000000000 RDI: ffff88041d858720
>> [Fri May  9 11:00:42 2014] RBP: 0000000000000008 R08: 0000000000010b60 R09: ffff88042fb10b60
>> [Fri May  9 11:00:42 2014] R10: 8000000000000000 R11: ffffea000f26a840 R12: ffff88041f0048a0
>> [Fri May  9 11:00:42 2014] R13: ffff88026628e000 R14: ffff88041d858720 R15: ffff88041a1edc10
>> [Fri May  9 11:00:42 2014] FS:  0000000000000000(0000) GS:ffff88042fb00000(0000) knlGS:0000000000000000
>> [Fri May  9 11:00:42 2014] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [Fri May  9 11:00:42 2014] CR2: 0000035fb5abf850 CR3: 000000000138a000 CR4: 00000000000006b0
>> [Fri May  9 11:00:42 2014] Stack:
>> [Fri May  9 11:00:42 2014]  0000000000000000 ffffffff818dde60 ffff88041a1ed400 ffff8802cffca570
>> [Fri May  9 11:00:42 2014]  ffff8802cffca080 ffff880416eb4200 ffff8802cffca080 ffffffff81052750
>> [Fri May  9 11:00:42 2014]  0000000000000000 0000000000000001 ffff88038e6260d8 ffff8802cffca598
>> [Fri May  9 11:00:42 2014] Call Trace:
>> [Fri May  9 11:00:42 2014]  [<ffffffff81052750>] ? 0xffffffff81052750
>> [Fri May  9 11:00:42 2014]  [<ffffffff81036e10>] ? 0xffffffff81036e10
>> [Fri May  9 11:00:42 2014]  [<ffffffff810371e8>] ? 0xffffffff810371e8
>> [Fri May  9 11:00:42 2014]  [<ffffffff810449cc>] ? 0xffffffff810449cc
>> [Fri May  9 11:00:42 2014]  [<ffffffff8100241f>] ? 0xffffffff8100241f
>> [Fri May  9 11:00:42 2014]  [<ffffffff81002a89>] ? 0xffffffff81002a89
>> [Fri May  9 11:00:42 2014]  [<ffffffff8137c212>] ? 0xffffffff8137c212
>> [Fri May  9 11:00:42 2014] Code: e9 68 fd 01 00 0f 1f 84 00 00 00 00 00 48 8b 43 18 48 8b 7b 10 48 8b 40 30 f0 ff 88 30 01 00 00 71 09 f0 ff 80 30 01 00 00 cd 04 <0f> b7 00 89 c2 66 81 e2 00 b0 66 81 fa 00 20 0f 84 53 ff ff ff
> 
> 
> And things are segfaulting randomly. These machines have been running
> 3.11.7-hardened-r1 since 2014-01-03 without issue until now -- all of
> our servers have. So the timing seems a little coincidental.
> 
> If it's not hardware (two different machines...), does this look like a
> kernel bug? Should I upgrade over the weekend and pray?
>