[gentoo-user] System freezes during compiles

[gentoo-user] System freezes during compiles

[Carlos Hendson]

[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]

Hello,

For last few weeks or so, I've been getting intermittent hard lock-ups
during the emerge of various packages.  It appears the more compile
intensive the package, the more likely the lock-up.  These lock-ups have
occurred under kernels 3.4.9 and 3.7.10 with gcc 4.5.4 and 4.6.3.

Once the machine is in a frozen state, the only thing that responds is
the soft power reset button.  Some times the machine lock-ups again
after the button is pressed (this is because the compile resumes once
the system comes out of it's frozen state).

If the system subsequently lock-ups because I wasn't able to cancel the
compile fast enough only a only option left is a hard power reset (10sec
+ hold power button).  If I cancel the compile, the system is perfectly
responsive and functions normally.

There are kernel stack traces in /var/log/messages which I'm unable to
decipher and diagnose as to what caused the lock-up.

If I had to guess, I'd blame an incorrect setting in the .config, but
since I'm stuck in the diagnostic of what part of the kernel might be
experiencing the problem, I need a bit of help to pin point the issue.  

I believe it to be a kernel configuration issue because when I booted
the machine using a system rescue Live CD, I was able to chroot into the
system and emerge packages like gcc without the lock-up problem
occurring.  

That's by no means conclusive, however, I've also run a complete pass of
memcheck for over an hour without any issues reported.

I'd like to completely rule out hardware failure, what diagnostic tools
tools are recommend to try identify potential hardware issue of this
type?

The various kernel stack traces are attached in case someone wants to
take a look.  I can provide more information should it be needed.

Any help or advice would be appreciated.

Regards,
Carlos 

[-- Attachment #2: kernel_cpu_stall_msg.txt --]
[-- Type: text/plain, Size: 49254 bytes --]

Mar 12 23:42:03 hydra kernel: [58066.564110] ------------[ cut here ]------------
Mar 12 23:42:03 hydra kernel: [58068.663176] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Mar 12 23:42:03 hydra kernel: [58068.673235] Hardware name: GA-990FXA-D3
Mar 12 23:42:03 hydra kernel: [58068.673303] Watchdog detected hard LOCKUP on cpu 2
Mar 12 23:42:03 hydra kernel: [58068.751056] Modules linked in: usb_storage uas ipv6 it87 hwmon_vid fglrx(PO) uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core joydev radeon i2c_al
go_bit ttm drm_kms_helper drm r8169 xhci_hcd ata_generic pata_acpi i2c_piix4 mii i2c_core pata_atiixp wmi serio_raw k10temp powernow_k8 pcspkr mperf freq_table
Mar 12 23:42:03 hydra kernel: [58068.945979] Pid: 720, comm: cc1 Tainted: P           O 3.4.9-gentoo #2
Mar 12 23:42:03 hydra kernel: [58068.946053] Call Trace:
Mar 12 23:42:03 hydra kernel: [58069.054704]  <NMI>  [<ffffffff81030050>] ? warn_slowpath_common+0x78/0x8c
Mar 12 23:42:03 hydra kernel: [58069.231277]  [<ffffffff810300fc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 12 23:42:03 hydra kernel: [58069.271020]  [<ffffffff8107bf89>] ? watchdog_overflow_callback+0x93/0x9e
Mar 12 23:42:03 hydra kernel: [58069.271135]  [<ffffffff8107bef6>] ? touch_nmi_watchdog+0x62/0x62
Mar 12 23:42:03 hydra kernel: [58069.293566]  [<ffffffff8108c002>] ? __perf_event_overflow+0x12c/0x1ae
Mar 12 23:42:03 hydra kernel: [58069.293689]  [<ffffffff8108a0a1>] ? perf_event_update_userpage+0x13/0xbf
Mar 12 23:42:03 hydra kernel: [58069.293811]  [<ffffffff8100db25>] ? x86_pmu_handle_irq+0xbe/0xf3
Mar 12 23:42:03 hydra kernel: [58069.293939]  [<ffffffff8151ff39>] ? nmi_handle.isra.4+0x3e/0x61
Mar 12 23:42:03 hydra kernel: [58069.294038]  [<ffffffff8151fffb>] ? do_nmi+0x9f/0x287
Mar 12 23:42:03 hydra kernel: [58069.294139]  [<ffffffff8151f7e2>] ? end_repeat_nmi+0x1a/0x1e
Mar 12 23:42:03 hydra kernel: [58069.294253]  [<ffffffff8151f084>] ? _raw_spin_lock_irq+0x6/0x6
Mar 12 23:42:03 hydra kernel: [58069.294357]  [<ffffffff8151f084>] ? _raw_spin_lock_irq+0x6/0x6
Mar 12 23:42:03 hydra kernel: [58069.314699]  [<ffffffff8151f084>] ? _raw_spin_lock_irq+0x6/0x6
Mar 12 23:42:03 hydra kernel: [58069.318869]  <<EOE>>  <IRQ>  [<ffffffff81059da5>] ? ntp_tick_length+0x23/0x28
Mar 12 23:42:03 hydra kernel: [58069.319051]  [<ffffffff8105972a>] ? do_timer+0x89/0x465
Mar 12 23:42:03 hydra kernel: [58069.319185]  [<ffffffff8105e881>] ? tick_do_update_jiffies64+0x74/0x98
Mar 12 23:42:03 hydra kernel: [58069.319300]  [<ffffffff8105e9b1>] ? tick_sched_timer+0x3f/0x8d
Mar 12 23:42:03 hydra kernel: [58069.319424]  [<ffffffff810476b7>] ? __run_hrtimer.isra.27+0x4b/0xa3
Mar 12 23:42:03 hydra kernel: [58069.319547]  [<ffffffff81047ca9>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 12 23:42:03 hydra kernel: [58069.319655]  [<ffffffff81017b71>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 12 23:42:03 hydra kernel: [58069.319750]  [<ffffffff81524907>] ? apic_timer_interrupt+0x67/0x70
Mar 12 23:42:03 hydra kernel: [58069.319810]  <EOI> 
Mar 12 23:42:03 hydra kernel: [58069.324331] ---[ end trace b1a58589d91a0dec ]---


Mar 12 23:58:02 hydra kernel: [59023.803433] ------------[ cut here ]------------
Mar 12 23:58:02 hydra kernel: [59024.963950] ------------[ cut here ]------------
Mar 12 23:58:02 hydra kernel: [59025.152834] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Mar 12 23:58:02 hydra kernel: [59025.152895] Hardware name: GA-990FXA-D3
Mar 12 23:58:02 hydra kernel: [59025.152939] Watchdog detected hard LOCKUP on cpu 4
Mar 12 23:58:02 hydra kernel: [59025.152976] Modules linked in: usb_storage
Mar 12 23:58:02 hydra kernel: [59025.769001] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Mar 12 23:58:02 hydra kernel: [59025.769063] Hardware name: GA-990FXA-D3
Mar 12 23:58:02 hydra kernel: [59025.769103] Watchdog detected hard LOCKUP on cpu 2
Mar 12 23:58:02 hydra kernel: [59025.769140] Modules linked in: usb_storage uas uas ipv6 it87 ipv6 hwmon_vid it87 hwmon_vid videobuf2_vmalloc videobuf2_vmalloc videobuf2_memops videobuf2_co
re joydev videobuf2_memops videobuf2_core joydev i2c_algo_bit ttm drm_kms_helper drm r8169 xhci_hcd ata_generic i2c_algo_bit pata_acpi i2c_piix4 mii ttm drm_kms_helper drm r8169 xhci_hcd at
a_generic pata_acpi i2c_piix4 i2c_core mii pata_atiixp wmi serio_raw k10temp powernow_k8 pcspkr mperf i2c_core freq_table [last unloaded: fglrx]
Mar 12 23:58:02 hydra kernel: [59025.769719]  pata_atiixp wmi serio_rawPid: 6374, comm: cc1 Tainted: P        W  O 3.4.9-gentoo #2
Mar 12 23:58:02 hydra kernel: [59025.769805] Call Trace:
Mar 12 23:58:02 hydra kernel: [59025.769839]  k10temp <NMI>  powernow_k8 pcspkr mperf freq_table [last unloaded: fglrx] [<ffffffff81030050>] ? warn_slowpath_common+0x78/0x8c
Mar 12 23:58:02 hydra kernel: [59025.819917] 
Mar 12 23:58:02 hydra kernel: [59025.820000]  [<ffffffff810300fc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 12 23:58:02 hydra kernel: [59025.820084] Pid: 6378, comm: cc1 Tainted: P        W  O 3.4.9-gentoo #2
Mar 12 23:58:02 hydra kernel: [59025.820137] Call Trace:
Mar 12 23:58:02 hydra kernel: [59025.820202]  [<ffffffff8107bf89>] ? watchdog_overflow_callback+0x93/0x9e
Mar 12 23:58:02 hydra kernel: [59025.820254]  <NMI>  [<ffffffff8107bef6>] ? touch_nmi_watchdog+0x62/0x62
Mar 12 23:58:02 hydra kernel: [59025.873740]  [<ffffffff81030050>] ? warn_slowpath_common+0x78/0x8c
Mar 12 23:58:02 hydra kernel: [59025.873844]  [<ffffffff8108c002>] ? __perf_event_overflow+0x12c/0x1ae
Mar 12 23:58:02 hydra kernel: [59025.873940]  [<ffffffff8108a0a1>] ? perf_event_update_userpage+0x13/0xbf
Mar 12 23:58:02 hydra kernel: [59025.889657]  [<ffffffff8100db25>] ? x86_pmu_handle_irq+0xbe/0xf3
Mar 12 23:58:02 hydra kernel: [59025.889751]  [<ffffffff810300fc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 12 23:58:02 hydra kernel: [59025.889870]  [<ffffffff8107bf89>] ? watchdog_overflow_callback+0x93/0x9e
Mar 12 23:58:02 hydra kernel: [59025.911586]  [<ffffffff8151ff39>] ? nmi_handle.isra.4+0x3e/0x61
Mar 12 23:58:02 hydra kernel: [59025.963434]  [<ffffffff8151fffb>] ? do_nmi+0x9f/0x287
Mar 12 23:58:02 hydra kernel: [59025.963527]  [<ffffffff8107bef6>] ? touch_nmi_watchdog+0x62/0x62
Mar 12 23:58:02 hydra kernel: [59025.963652]  [<ffffffff8108c002>] ? __perf_event_overflow+0x12c/0x1ae
Mar 12 23:58:02 hydra kernel: [59025.964432]  [<ffffffff8151f7e2>] ? end_repeat_nmi+0x1a/0x1e
Mar 12 23:58:02 hydra kernel: [59025.999330]  [<ffffffff81059fbf>] ? second_overflow+0x215/0x228
Mar 12 23:58:02 hydra kernel: [59025.999431]  [<ffffffff8108a0a1>] ? perf_event_update_userpage+0x13/0xbf
Mar 12 23:58:02 hydra kernel: [59025.999539]  [<ffffffff8100db25>] ? x86_pmu_handle_irq+0xbe/0xf3
Mar 12 23:58:02 hydra kernel: [59026.097048]  [<ffffffff81059fbf>] ? second_overflow+0x215/0x228
Mar 12 23:58:02 hydra kernel: [59026.114000]  [<ffffffff81059fbf>] ? second_overflow+0x215/0x228
Mar 12 23:58:02 hydra kernel: [59026.114058]  <<EOE>>  [<ffffffff8151ff39>] ? nmi_handle.isra.4+0x3e/0x61
Mar 12 23:58:02 hydra kernel: [59026.114208]  [<ffffffff8151fffb>] ? do_nmi+0x9f/0x287
Mar 12 23:58:02 hydra kernel: [59026.114253]  <IRQ>  [<ffffffff8105979d>] ? do_timer+0xfc/0x465
Mar 12 23:58:02 hydra kernel: [59026.114396]  [<ffffffff8151f7e2>] ? end_repeat_nmi+0x1a/0x1e
Mar 12 23:58:02 hydra kernel: [59026.114515]  [<ffffffff8105e881>] ? tick_do_update_jiffies64+0x74/0x98
Mar 12 23:58:02 hydra kernel: [59026.114598]  [<ffffffff810590c5>] ? ktime_get+0x29/0x86
Mar 12 23:58:02 hydra kernel: [59026.114680]  [<ffffffff8105e9b1>] ? tick_sched_timer+0x3f/0x8d
Mar 12 23:58:02 hydra kernel: [59026.114769]  [<ffffffff810590c5>] ? ktime_get+0x29/0x86
Mar 12 23:58:02 hydra kernel: [59026.114858]  [<ffffffff810476b7>] ? __run_hrtimer.isra.27+0x4b/0xa3
Mar 12 23:58:02 hydra kernel: [59026.114948]  [<ffffffff81047ca9>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 12 23:58:02 hydra kernel: [59026.134942]  [<ffffffff81017b71>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 12 23:58:02 hydra kernel: [59026.135020]  [<ffffffff810590c5>] ? ktime_get+0x29/0x86
Mar 12 23:58:02 hydra kernel: [59026.135060]  <<EOE>>  <IRQ>  [<ffffffff81524907>] ? apic_timer_interrupt+0x67/0x70
Mar 12 23:58:02 hydra kernel: [59026.135224]  [<ffffffff8105e987>] ? tick_sched_timer+0x15/0x8d
Mar 12 23:58:02 hydra kernel: [59026.136919]  <EOI>  [<ffffffff810476b7>] ? __run_hrtimer.isra.27+0x4b/0xa3
Mar 12 23:58:02 hydra kernel: [59026.137059]  [<ffffffff81047ca9>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 12 23:58:02 hydra kernel: [59026.137097] ---[ end trace b1a58589d91a0ded ]---
Mar 12 23:58:02 hydra kernel: [59026.137153]  [<ffffffff81017b71>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 12 23:58:02 hydra kernel: [59026.137171]  [<ffffffff81524907>] ? apic_timer_interrupt+0x67/0x70
Mar 12 23:58:02 hydra kernel: [59026.137192]  <EOI> 
Mar 12 23:58:02 hydra kernel: [59026.137209] ---[ end trace b1a58589d91a0dee ]---


Mar 13 03:53:55 hydra kernel: [73120.181182] ------------[ cut here ]------------
Mar 13 03:53:55 hydra kernel: [73120.275906] ------------[ cut here ]------------
Mar 13 03:53:55 hydra kernel: [73120.946978] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Mar 13 03:53:55 hydra kernel: [73120.947042] Hardware name: GA-990FXA-D3
Mar 13 03:53:55 hydra kernel: [73120.947137] Watchdog detected hard LOCKUP on cpu 4
Mar 13 03:53:55 hydra kernel: [73120.947176] Modules linked in:
Mar 13 03:53:55 hydra kernel: [73122.054759] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Mar 13 03:53:55 hydra kernel: [73122.054814]  usb_storage uas ipv6 it87
Mar 13 03:53:55 hydra kernel: [73122.054893] Hardware name: GA-990FXA-D3
Mar 13 03:53:55 hydra kernel: [73122.054932] Watchdog detected hard LOCKUP on cpu 2
Mar 13 03:53:55 hydra kernel: [73122.054971] Modules linked in: hwmon_vid usb_storage uas ipv6 it87 videobuf2_vmalloc hwmon_vid videobuf2_memops videobuf2_core joydev i2c_algo_bit ttm drm_kms_helper drm r8169 videobuf2_vmalloc videobuf2_memops videobuf2_core joydev i2c_algo_bit ttm drm_kms_helper drm r8169 xhci_hcd ata_generic pata_acpi i2c_piix4 xhci_hcd ata_generic pata_acpi i2c_piix4 mii i2c_core mii i2c_core pata_atiixp pata_atiixp wmi serio_raw k10temp powernow_k8 pcspkr mperf freq_table [last unloaded: fglrx]
Mar 13 03:53:55 hydra kernel: [73122.055552]  wmiPid: 31713, comm: cc1 Tainted: P        W  O 3.4.9-gentoo #2
Mar 13 03:53:55 hydra kernel: [73122.055637] Call Trace:
Mar 13 03:53:55 hydra kernel: [73122.055686]  <NMI>  serio_raw k10temp powernow_k8 pcspkr mperf freq_table [last unloaded: fglrx]
Mar 13 03:53:55 hydra kernel: [73122.055816] Pid: 31720, comm: cc1 Tainted: P        W  O 3.4.9-gentoo #2
Mar 13 03:53:55 hydra kernel: [73122.185445]  [<ffffffff81030050>] ? warn_slowpath_common+0x78/0x8c
Mar 13 03:53:55 hydra kernel: [73122.185556]  [<ffffffff810300fc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 13 03:53:55 hydra kernel: [73122.185652]  [<ffffffff8107bf89>] ? watchdog_overflow_callback+0x93/0x9e
Mar 13 03:53:55 hydra kernel: [73122.185761]  [<ffffffff8107bef6>] ? touch_nmi_watchdog+0x62/0x62
Mar 13 03:53:55 hydra kernel: [73122.185803] Call Trace:
Mar 13 03:53:55 hydra kernel: [73122.185849]  <NMI>  [<ffffffff8108c002>] ? __perf_event_overflow+0x12c/0x1ae
Mar 13 03:53:55 hydra kernel: [73122.204378]  [<ffffffff81030050>] ? warn_slowpath_common+0x78/0x8c
Mar 13 03:53:55 hydra kernel: [73122.251209]  [<ffffffff810300fc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 13 03:53:55 hydra kernel: [73122.251334]  [<ffffffff8108a0a1>] ? perf_event_update_userpage+0x13/0xbf
Mar 13 03:53:55 hydra kernel: [73122.279130]  [<ffffffff8107bf89>] ? watchdog_overflow_callback+0x93/0x9e
Mar 13 03:53:55 hydra kernel: [73122.279227]  [<ffffffff8107bef6>] ? touch_nmi_watchdog+0x62/0x62
Mar 13 03:53:55 hydra kernel: [73122.279343]  [<ffffffff8108c002>] ? __perf_event_overflow+0x12c/0x1ae
Mar 13 03:53:55 hydra kernel: [73122.279441]  [<ffffffff8100db25>] ? x86_pmu_handle_irq+0xbe/0xf3
Mar 13 03:53:55 hydra kernel: [73122.279542]  [<ffffffff8151ff39>] ? nmi_handle.isra.4+0x3e/0x61
Mar 13 03:53:55 hydra kernel: [73122.279652]  [<ffffffff8108a0a1>] ? perf_event_update_userpage+0x13/0xbf
Mar 13 03:53:55 hydra kernel: [73122.279753]  [<ffffffff8100db25>] ? x86_pmu_handle_irq+0xbe/0xf3
Mar 13 03:53:55 hydra kernel: [73122.279855]  [<ffffffff8151fffb>] ? do_nmi+0x9f/0x287
Mar 13 03:53:55 hydra kernel: [73122.279939]  [<ffffffff8151ff39>] ? nmi_handle.isra.4+0x3e/0x61
Mar 13 03:53:55 hydra kernel: [73122.280032]  [<ffffffff8151fffb>] ? do_nmi+0x9f/0x287
Mar 13 03:53:55 hydra kernel: [73122.280115]  [<ffffffff8151f7e2>] ? end_repeat_nmi+0x1a/0x1e
Mar 13 03:53:55 hydra kernel: [73122.280225]  [<ffffffff81058e56>] ? update_rt_offset+0x24/0x3a
Mar 13 03:53:55 hydra kernel: [73122.280302]  [<ffffffff8151f7e2>] ? end_repeat_nmi+0x1a/0x1e
Mar 13 03:53:55 hydra kernel: [73122.280396]  [<ffffffff8105e972>] ? tick_nohz_handler+0xcd/0xcd
Mar 13 03:53:55 hydra kernel: [73122.287098]  [<ffffffff810590c9>] ? ktime_get+0x2d/0x86
Mar 13 03:53:55 hydra kernel: [73122.287183]  [<ffffffff810590c9>] ? ktime_get+0x2d/0x86
Mar 13 03:53:55 hydra kernel: [73122.287257]  [<ffffffff81058e56>] ? update_rt_offset+0x24/0x3a
Mar 13 03:53:55 hydra kernel: [73122.287343]  [<ffffffff81058e56>] ? update_rt_offset+0x24/0x3a
Mar 13 03:53:55 hydra kernel: [73122.287407]  <<EOE>>  <IRQ>  [<ffffffff810590c9>] ? ktime_get+0x2d/0x86
Mar 13 03:53:55 hydra kernel: [73122.287508]  <<EOE>>  <IRQ>  [<ffffffff81058ecb>] ? timekeeping_update+0x1b/0x3d
Mar 13 03:53:55 hydra kernel: [73122.287660]  [<ffffffff8105e987>] ? tick_sched_timer+0x15/0x8d
Mar 13 03:53:55 hydra kernel: [73122.287769]  [<ffffffff810476b7>] ? __run_hrtimer.isra.27+0x4b/0xa3
Mar 13 03:53:55 hydra kernel: [73122.287861]  [<ffffffff81059ae1>] ? do_timer+0x440/0x465
Mar 13 03:53:55 hydra kernel: [73122.287971]  [<ffffffff8105e881>] ? tick_do_update_jiffies64+0x74/0x98
Mar 13 03:53:55 hydra kernel: [73122.288087]  [<ffffffff81047ca9>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 13 03:53:55 hydra kernel: [73122.288180]  [<ffffffff8105e9b1>] ? tick_sched_timer+0x3f/0x8d
Mar 13 03:53:55 hydra kernel: [73122.288279]  [<ffffffff810476b7>] ? __run_hrtimer.isra.27+0x4b/0xa3
Mar 13 03:53:55 hydra kernel: [73122.288399]  [<ffffffff81047ca9>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 13 03:53:55 hydra kernel: [73122.363879]  [<ffffffff81017b71>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 13 03:53:55 hydra kernel: [73122.363964]  [<ffffffff81524907>] ? apic_timer_interrupt+0x67/0x70
Mar 13 03:53:55 hydra kernel: [73122.364058]  [<ffffffff81017b71>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 13 03:53:55 hydra kernel: [73122.364133]  [<ffffffff81524907>] ? apic_timer_interrupt+0x67/0x70
Mar 13 03:53:55 hydra kernel: [73122.364182]  <EOI>  <EOI> 
Mar 13 03:53:55 hydra kernel: [73122.364249] ---[ end trace b1a58589d91a0def ]---
Mar 13 03:53:55 hydra kernel: [73122.450644] ---[ end trace b1a58589d91a0df0 ]---
Mar 13 03:53:55 hydra kernel: [73137.107343] INFO: rcu_sched detected stalls on CPUs/tasks:
Mar 13 03:53:55 hydra kernel: [73137.107349]    0: (1 ticks this GP) idle=3a9/1/0 drain=0 . timer=-1
Mar 13 03:53:55 hydra kernel: [73137.107353]    (detected by 4, t=13855804 jiffies)
Mar 13 03:53:55 hydra kernel: [73137.107355] INFO: Stall ended before state dump start
Mar 13 03:53:55 hydra kernel: [73137.107403] Clocksource tsc unstable (delta = 604344193081 ns)
Mar 13 03:53:55 hydra kernel: [73137.470589] Switching to clocksource hpet
Mar 13 03:53:55 hydra kernel: [73138.982799] NOHZ: local_softirq_pending 08



Mar 14 19:28:55 hydra kernel: [42815.740460] ------------[ cut here ]------------
Mar 14 19:28:55 hydra kernel: [42816.410513] ------------[ cut here ]------------
Mar 14 19:28:55 hydra kernel: [42817.511328] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Mar 14 19:28:55 hydra kernel: [42817.511388] Hardware name: GA-990FXA-D3
Mar 14 19:28:55 hydra kernel: [42817.511431] Watchdog detected hard LOCKUP on cpu 2
Mar 14 19:28:55 hydra kernel: [42817.511471] Modules linked in:
Mar 14 19:28:55 hydra kernel: [42818.214348] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x93/0x9e()
Mar 14 19:28:55 hydra kernel: [42818.214400]  ipv6 it87 hwmon_vid fglrx(PO) uvcvideo
Mar 14 19:28:55 hydra kernel: [42818.214483] Hardware name: GA-990FXA-D3
Mar 14 19:28:55 hydra kernel: [42818.214533] Watchdog detected hard LOCKUP on cpu 4
Mar 14 19:28:55 hydra kernel: [42818.214573] Modules linked in: ipv6 videobuf2_vmalloc videobuf2_memops videobuf2_core it87 joydev usb_storage uas hwmon_vid fglrx(PO) uvcvideo radeon i2c_al
go_bit ttm videobuf2_vmalloc drm_kms_helper drm ata_generic pata_acpi xhci_hcd pata_atiixp serio_raw powernow_k8 r8169 videobuf2_memops wmi i2c_piix4 videobuf2_core joydev usb_storage uas m
perf pcspkr i2c_core freq_table k10temp mii
Mar 14 19:28:55 hydra kernel: [42818.215092]  radeonPid: 2766, comm: cc1 Tainted: P           O 3.4.9-gentoo #2
Mar 14 19:28:55 hydra kernel: [42818.215179] Call Trace:
Mar 14 19:28:55 hydra kernel: [42818.215220]  <NMI>  i2c_algo_bit ttm drm_kms_helper drm [<ffffffff81030050>] ? warn_slowpath_common+0x78/0x8c
Mar 14 19:28:55 hydra kernel: [42818.349987]  ata_generic [<ffffffff810300fc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 14 19:28:55 hydra kernel: [42818.350164]  [<ffffffff8107bf89>] ? watchdog_overflow_callback+0x93/0x9e
Mar 14 19:28:55 hydra kernel: [42818.350208]  pata_acpi xhci_hcd [<ffffffff8107bef6>] ? touch_nmi_watchdog+0x62/0x62
Mar 14 19:28:55 hydra kernel: [42818.350399]  [<ffffffff8108c002>] ? __perf_event_overflow+0x12c/0x1ae
Mar 14 19:28:55 hydra kernel: [42818.350488]  [<ffffffff8108a0a1>] ? perf_event_update_userpage+0x13/0xbf
Mar 14 19:28:55 hydra kernel: [42818.350589]  [<ffffffff8100db25>] ? x86_pmu_handle_irq+0xbe/0xf3
Mar 14 19:28:55 hydra kernel: [42818.350638]  pata_atiixp [<ffffffff8151ff39>] ? nmi_handle.isra.4+0x3e/0x61
Mar 14 19:28:55 hydra kernel: [42818.350778]  [<ffffffff8151fffb>] ? do_nmi+0x9f/0x287
Mar 14 19:28:55 hydra kernel: [42818.350822]  serio_raw powernow_k8 r8169 wmi i2c_piix4 mperf pcspkr i2c_core freq_table k10temp mii
Mar 14 19:28:55 hydra kernel: [42818.350999]  [<ffffffff8151f7e2>] ? end_repeat_nmi+0x1a/0x1e
Mar 14 19:28:55 hydra kernel: [42818.351075] Pid: 2760, comm: cc1 Tainted: P           O 3.4.9-gentoo #2
Mar 14 19:28:55 hydra kernel: [42818.351127] Call Trace:
Mar 14 19:28:55 hydra kernel: [42818.351176]  <NMI>  [<ffffffff810590c9>] ? ktime_get+0x2d/0x86
Mar 14 19:28:55 hydra kernel: [42818.364923]  [<ffffffff81030050>] ? warn_slowpath_common+0x78/0x8c
Mar 14 19:28:55 hydra kernel: [42818.365045]  [<ffffffff810300fc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 14 19:28:55 hydra kernel: [42818.365128]  [<ffffffff810590c9>] ? ktime_get+0x2d/0x86
Mar 14 19:28:55 hydra kernel: [42818.365232]  [<ffffffff8107bf89>] ? watchdog_overflow_callback+0x93/0x9e
Mar 14 19:28:55 hydra kernel: [42818.365347]  [<ffffffff8107bef6>] ? touch_nmi_watchdog+0x62/0x62
Mar 14 19:28:55 hydra kernel: [42818.365448]  [<ffffffff810590c9>] ? ktime_get+0x2d/0x86
Mar 14 19:28:55 hydra kernel: [42818.365487]  <<EOE>>  <IRQ>  [<ffffffff8108c002>] ? __perf_event_overflow+0x12c/0x1ae
Mar 14 19:28:55 hydra kernel: [42818.365691]  [<ffffffff8105d804>] ? clockevents_program_event+0x73/0xce
Mar 14 19:28:55 hydra kernel: [42818.365784]  [<ffffffff81047cef>] ? hrtimer_interrupt+0x11f/0x1c9
Mar 14 19:28:55 hydra kernel: [42818.365861]  [<ffffffff81017b71>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 14 19:28:55 hydra kernel: [42818.365975]  [<ffffffff8108a0a1>] ? perf_event_update_userpage+0x13/0xbf
Mar 14 19:28:55 hydra kernel: [42818.366060]  [<ffffffff81524907>] ? apic_timer_interrupt+0x67/0x70
Mar 14 19:28:55 hydra kernel: [42818.366105]  <EOI> 
Mar 14 19:28:55 hydra kernel: [42818.366164] ---[ end trace 837a705571690c4c ]---
Mar 14 19:28:55 hydra kernel: [42818.366260]  [<ffffffff8100db25>] ? x86_pmu_handle_irq+0xbe/0xf3
Mar 14 19:28:55 hydra kernel: [42818.366355]  [<ffffffff8151ff39>] ? nmi_handle.isra.4+0x3e/0x61
Mar 14 19:28:55 hydra kernel: [42818.366470]  [<ffffffff8151fffb>] ? do_nmi+0x9f/0x287
Mar 14 19:28:55 hydra kernel: [42818.366596]  [<ffffffff8151f7e2>] ? end_repeat_nmi+0x1a/0x1e
Mar 14 19:28:55 hydra kernel: [42818.366725]  [<ffffffff81059fc6>] ? second_overflow+0x21c/0x228
Mar 14 19:28:55 hydra kernel: [42818.366845]  [<ffffffff81059fc6>] ? second_overflow+0x21c/0x228
Mar 14 19:28:55 hydra kernel: [42818.366960]  [<ffffffff81059fc6>] ? second_overflow+0x21c/0x228
Mar 14 19:28:55 hydra kernel: [42818.367030]  <<EOE>>  <IRQ>  [<ffffffff8105979d>] ? do_timer+0xfc/0x465
Mar 14 19:28:55 hydra kernel: [42818.367268]  [<ffffffff8105e881>] ? tick_do_update_jiffies64+0x74/0x98
Mar 14 19:28:55 hydra kernel: [42818.367404]  [<ffffffff8105e9b1>] ? tick_sched_timer+0x3f/0x8d
Mar 14 19:28:55 hydra kernel: [42818.367529]  [<ffffffff810476b7>] ? __run_hrtimer.isra.27+0x4b/0xa3
Mar 14 19:28:55 hydra kernel: [42818.367669]  [<ffffffff81047ca9>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 14 19:28:55 hydra kernel: [42818.367783]  [<ffffffff81017b71>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 14 19:28:55 hydra kernel: [42818.367882]  [<ffffffff81524907>] ? apic_timer_interrupt+0x67/0x70
Mar 14 19:28:55 hydra kernel: [42818.367951]  <EOI> 
Mar 14 19:28:55 hydra kernel: [42818.373856] ---[ end trace 837a705571690c4d ]---
Mar 14 19:28:55 hydra kernel: [42818.748880] Clocksource tsc unstable (delta = 11059422880 ns)



Mar 15 00:17:26 hydra kernel: [  741.918530] ------------[ cut here ]------------
Mar 15 00:17:26 hydra kernel: [  742.310636] ------------[ cut here ]------------
Mar 15 00:17:26 hydra kernel: [  743.141280] WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x93/0x9e()
Mar 15 00:17:26 hydra kernel: [  743.552043] Hardware name: GA-990FXA-D3
Mar 15 00:17:26 hydra kernel: [  743.969879] Watchdog detected hard LOCKUP on cpu 2
Mar 15 00:17:26 hydra kernel: [  744.555193] WARNING: at kernel/watchdog.c:245 watchdog_overflow_callback+0x93/0x9e()
Mar 15 00:17:26 hydra kernel: [  744.559406] Hardware name: GA-990FXA-D3
Mar 15 00:17:26 hydra kernel: [  744.567109] Modules linked in: ipv6 it87 hwmon_vid joydev
Mar 15 00:17:26 hydra kernel: [  744.879203] Watchdog detected hard LOCKUP on cpu 3
Mar 15 00:17:26 hydra kernel: [  744.879261] Modules linked in: ipv6 it87 hwmon_vid
Mar 15 00:17:26 hydra kernel: [  744.879331]  radeon i2c_algo_bit ata_generic r8169
Mar 15 00:17:26 hydra kernel: [  744.879334]  joydev radeon i2c_algo_bit
Mar 15 00:17:26 hydra kernel: [  744.879397]  pata_acpi i2c_piix4 ttm drm_kms_helper drm xhci_hcd mii k10temp i2c_core pata_atiixp acpi_cpufreq wmi serio_raw mperf pcspkr freq_table
Mar 15 00:17:26 hydra kernel: [  744.879673]  ata_genericPid: 5949, comm: make Not tainted 3.7.10-gentoo #1
Mar 15 00:17:26 hydra kernel: [  744.879730] Call Trace:
Mar 15 00:17:26 hydra kernel: [  744.879756]  <NMI> 
Mar 15 00:17:26 hydra kernel: [  744.879759]  r8169 pata_acpi i2c_piix4 ttm drm_kms_helper
Mar 15 00:17:26 hydra kernel: [  744.879814]  drm [<ffffffff81034a30>] ? warn_slowpath_common+0x78/0x8c
Mar 15 00:17:26 hydra kernel: [  744.879834]  xhci_hcd mii k10temp i2c_core
Mar 15 00:17:26 hydra kernel: [  744.883127]  [<ffffffff81034adc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 15 00:17:26 hydra kernel: [  744.883196]  [<ffffffff8108397f>] ? watchdog_overflow_callback+0x93/0x9e
Mar 15 00:17:26 hydra kernel: [  744.883277]  [<ffffffff810838ec>] ? touch_nmi_watchdog+0x60/0x60
Mar 15 00:17:26 hydra kernel: [  744.883354]  [<ffffffff81095f01>] ? __perf_event_overflow+0x12c/0x1ae
Mar 15 00:17:26 hydra kernel: [  744.883437]  [<ffffffff81093c4a>] ? perf_event_update_userpage+0x13/0xbf
Mar 15 00:17:26 hydra kernel: [  744.883508]  [<ffffffff8100da1f>] ? x86_pmu_handle_irq+0xe1/0x11a
Mar 15 00:17:26 hydra kernel: [  744.883583]  [<ffffffff8152a55b>] ? nmi_handle.isra.0+0x3e/0x61
Mar 15 00:17:26 hydra kernel: [  744.883651]  [<ffffffff8152a61d>] ? do_nmi+0x9f/0x2a9
Mar 15 00:17:26 hydra kernel: [  744.883714]  [<ffffffff81529de6>] ? end_repeat_nmi+0x1e/0x2e
Mar 15 00:17:26 hydra kernel: [  744.883777]  [<ffffffff8105fc8b>] ? ktime_get_boottime+0x1f/0x1f
Mar 15 00:17:26 hydra kernel: [  744.883849]  [<ffffffff8105fc8b>] ? ktime_get_boottime+0x1f/0x1f
Mar 15 00:17:26 hydra kernel: [  744.883907]  [<ffffffff8105fc8b>] ? ktime_get_boottime+0x1f/0x1f
Mar 15 00:17:26 hydra kernel: [  744.883979]  <<EOE>>  <IRQ>  [<ffffffff810607a9>] ? do_timer+0x365/0x38a
Mar 15 00:17:26 hydra kernel: [  744.884076]  [<ffffffff81065641>] ? tick_do_update_jiffies64+0x74/0x98
Mar 15 00:17:26 hydra kernel: [  744.884153]  [<ffffffff81065771>] ? tick_sched_timer+0x3f/0x9f
Mar 15 00:17:26 hydra kernel: [  744.884222]  [<ffffffff8104d9cb>] ? __run_hrtimer.isra.26+0x4b/0xa3
Mar 15 00:17:26 hydra kernel: [  744.884304]  [<ffffffff8104dfb7>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 15 00:17:26 hydra kernel: [  744.884384]  [<ffffffff8101aef3>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 15 00:17:26 hydra kernel: [  744.884467]  [<ffffffff8152f04a>] ? apic_timer_interrupt+0x6a/0x70
Mar 15 00:17:26 hydra kernel: [  744.884548]  <EOI>  [<ffffffff8152e5d2>] ? system_call_fastpath+0x16/0x1b
Mar 15 00:17:26 hydra kernel: [  744.884643] ---[ end trace b01c6c9e63ae3df6 ]---
Mar 15 00:17:26 hydra kernel: [  744.884656]  pata_atiixp acpi_cpufreq wmi serio_raw mperf pcspkr freq_table
Mar 15 00:17:26 hydra kernel: [  744.884752] Pid: 5955, comm: cc1 Tainted: G        W    3.7.10-gentoo #1
Mar 15 00:17:26 hydra kernel: [  744.884765] Call Trace:
Mar 15 00:17:26 hydra kernel: [  744.884775]  <NMI>  [<ffffffff81034a30>] ? warn_slowpath_common+0x78/0x8c
Mar 15 00:17:26 hydra kernel: [  744.884849]  [<ffffffff81034adc>] ? warn_slowpath_fmt+0x45/0x4a
Mar 15 00:17:26 hydra kernel: [  744.884891]  [<ffffffff8108397f>] ? watchdog_overflow_callback+0x93/0x9e
Mar 15 00:17:26 hydra kernel: [  744.884927]  [<ffffffff810838ec>] ? touch_nmi_watchdog+0x60/0x60
Mar 15 00:17:26 hydra kernel: [  744.884952]  [<ffffffff81095f01>] ? __perf_event_overflow+0x12c/0x1ae
Mar 15 00:17:26 hydra kernel: [  744.884957] INFO: rcu_sched self-detected stall on CPU
Mar 15 00:17:26 hydra kernel: [  744.884967]    2: (2 ticks this GP) idle=31d/140000000000001/0 drain=0 . timer not pending
Mar 15 00:17:26 hydra kernel: [  744.884987]    
Mar 15 00:17:26 hydra kernel: [  744.885005]  (t=91823 jiffies)
Mar 15 00:17:26 hydra kernel: [  744.885011] INFO: rcu_sched self-detected stall on CPU
Mar 15 00:17:26 hydra kernel: [  744.885018] sending NMI to all CPUs:
Mar 15 00:17:26 hydra kernel: [  744.885030] NMI backtrace for cpu 2
Mar 15 00:17:26 hydra kernel: [  744.885060]  [<ffffffff81093c4a>] ? perf_event_update_userpage+0x13/0xbf
Mar 15 00:17:26 hydra kernel: [  744.885067] CPU 2 
Mar 15 00:17:26 hydra kernel: [  744.885067] Pid: 5949, comm: make Tainted: G        W    3.7.10-gentoo #1 Gigabyte Technology Co., Ltd. GA-990FXA-D3/GA-990FXA-D3
Mar 15 00:17:26 hydra kernel: [  744.885072]  [<ffffffff8100da1f>] ? x86_pmu_handle_irq+0xe1/0x11a
Mar 15 00:17:26 hydra kernel: [  744.885078]  [<ffffffff8152a55b>] ? nmi_handle.isra.0+0x3e/0x61
Mar 15 00:17:26 hydra kernel: [  744.885083]  [<ffffffff8152a61d>] ? do_nmi+0x9f/0x2a9
Mar 15 00:17:26 hydra kernel: [  744.885087]  [<ffffffff81529de6>] ? end_repeat_nmi+0x1e/0x2e
Mar 15 00:17:26 hydra kernel: [  744.885094] RIP: 0010:[<ffffffff812b99db>]  [<ffffffff812b99db>] __const_udelay+0x9/0x23
Mar 15 00:17:26 hydra kernel: [  744.885100]  [<ffffffff81065732>] ? tick_nohz_handler+0xcd/0xcd
Mar 15 00:17:26 hydra kernel: [  744.885103] RSP: 0000:ffff88023fc83e30  EFLAGS: 00000006
Mar 15 00:17:26 hydra kernel: [  744.885106] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885109] RDX: 0000000000329570 RSI: 0000000000000c00 RDI: 0000000000418958
Mar 15 00:17:26 hydra kernel: [  744.885111] RBP: ffff88023fc8db20 R08: 0000000000000000 R09: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885114] R10: 0000000000000000 R11: 7fffffffffffff00 R12: ffffffff81825700
Mar 15 00:17:26 hydra kernel: [  744.885117] R13: ffff88023fc8d678 R14: ffff8801e9df0000 R15: 0000000000000002
Mar 15 00:17:26 hydra kernel: [  744.885120] FS:  00007fa6bf93f700(0000) GS:ffff88023fc80000(0000) knlGS:0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885123] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 15 00:17:26 hydra kernel: [  744.885126] CR2: 00007fa6bf900000 CR3: 000000022f59c000 CR4: 00000000000007e0
Mar 15 00:17:26 hydra kernel: [  744.885128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885131] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 15 00:17:26 hydra kernel: [  744.885134] Process make (pid: 5949, threadinfo ffff8801e9df0000, task ffff88022f8027c0)
Mar 15 00:17:26 hydra kernel: [  744.885136] Stack:
Mar 15 00:17:26 hydra kernel: [  744.885140]  ffffffff8101b6ba ffffffff81825700
Mar 15 00:17:26 hydra kernel: [  744.885143]  [<ffffffff8105fb0e>] ? ktime_get+0x2b/0x78
Mar 15 00:17:26 hydra kernel: [  744.885148]  [<ffffffff8105fb0e>] ? ktime_get+0x2b/0x78
Mar 15 00:17:26 hydra kernel: [  744.885151]  [<ffffffff8105fb0e>] ? ktime_get+0x2b/0x78
Mar 15 00:17:26 hydra kernel: [  744.885154]  ffffffff810896cb ffff88023fc913b0
Mar 15 00:17:26 hydra kernel: [  744.885157]  <<EOE>>  <IRQ>  [<ffffffff81065747>] ? tick_sched_timer+0x15/0x9f
Mar 15 00:17:26 hydra kernel: [  744.885166]  [<ffffffff8104d9cb>] ? __run_hrtimer.isra.26+0x4b/0xa3
Mar 15 00:17:26 hydra kernel: [  744.885172]  [<ffffffff8104dfb7>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 15 00:17:26 hydra kernel: [  744.885177]  [<ffffffff8101aef3>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 15 00:17:26 hydra kernel: [  744.885182]  [<ffffffff8152f04a>] ? apic_timer_interrupt+0x6a/0x70
Mar 15 00:17:26 hydra kernel: [  744.885188]  <EOI> <4>[  744.885191] ---[ end trace b01c6c9e63ae3df7 ]---
Mar 15 00:17:26 hydra kernel: [  744.885193] 
Mar 15 00:17:26 hydra kernel: [  744.885195]  0000000035b52a80 0000000000000000 ffffffff8105f56e ffffffff81007470
Mar 15 00:17:26 hydra kernel: [  744.885199]  0000000000000002 0000000000000001 0000000000000002 ffff88023fc8d678
Mar 15 00:17:26 hydra kernel: [  744.885203] Call Trace:
Mar 15 00:17:26 hydra kernel: [  744.885204]  <IRQ> 
Mar 15 00:17:26 hydra kernel: [  744.885206]  [<ffffffff8101b6ba>] ? arch_trigger_all_cpu_backtrace+0x59/0x67
Mar 15 00:17:26 hydra kernel: [  744.885214]  [<ffffffff810896cb>] ? rcu_pending+0xfa/0x3dc
Mar 15 00:17:26 hydra kernel: [  744.885217]  [<ffffffff8105f56e>] ? write_seqlock+0xd/0x11
Mar 15 00:17:26 hydra kernel: [  744.885221]  [<ffffffff81007470>] ? read_tsc+0x5/0x14
Mar 15 00:17:26 hydra kernel: [  744.885225]  [<ffffffff8108afae>] ? rcu_check_callbacks+0x76/0x82
Mar 15 00:17:26 hydra kernel: [  744.885229]  [<ffffffff8103eedc>] ? update_process_times+0x31/0x63
Mar 15 00:17:26 hydra kernel: [  744.885233]  [<ffffffff810657ab>] ? tick_sched_timer+0x79/0x9f
Mar 15 00:17:26 hydra kernel: [  744.885237]  [<ffffffff8104d9cb>] ? __run_hrtimer.isra.26+0x4b/0xa3
Mar 15 00:17:26 hydra kernel: [  744.885241]  [<ffffffff8104dfb7>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 15 00:17:26 hydra kernel: [  744.885245]  [<ffffffff8101aef3>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 15 00:17:26 hydra kernel: [  744.885249]  [<ffffffff8152f04a>] ? apic_timer_interrupt+0x6a/0x70
Mar 15 00:17:26 hydra kernel: [  744.885251]  <EOI> 
Mar 15 00:17:26 hydra kernel: [  744.885252]  [<ffffffff8152e5d2>] ? system_call_fastpath+0x16/0x1b
Mar 15 00:17:26 hydra kernel: [  744.885257] Code: eb 0e 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c3 48 8b 05 98 8b 58 00 ff e0 65 48 8b 14 25 98 12 01 00 <48> 69 d2 fa 00 00 00 48 8d 04 bd 00 00 00 00 f7 e2 48 8d 7a 01 
Mar 15 00:17:26 hydra kernel: [  744.885284] NMI backtrace for cpu 5
Mar 15 00:17:26 hydra kernel: [  744.885288] CPU 5 
Mar 15 00:17:26 hydra kernel: [  744.885294] Pid: 0, comm: swapper/5 Tainted: G        W    3.7.10-gentoo #1 Gigabyte Technology Co., Ltd. GA-990FXA-D3/GA-990FXA-D3
Mar 15 00:17:26 hydra kernel: [  744.885297] RIP: 0010:[<ffffffff812b89b0>]  [<ffffffff812b89b0>] vsnprintf+0x126/0x427
Mar 15 00:17:26 hydra kernel: [  744.885305] RSP: 0018:ffff88023fd43ce8  EFLAGS: 00000046
Mar 15 00:17:26 hydra kernel: [  744.885308] RAX: 0000000000000002 RBX: ffff88023fd43d60 RCX: 0000000000000003
Mar 15 00:17:26 hydra kernel: [  744.885310] RDX: ffffffff8173e767 RSI: 0000000000000003 RDI: ffffffff8173e767
Mar 15 00:17:26 hydra kernel: [  744.885313] RBP: ffff88023fd43dd0 R08: 0000000000000002 R09: 00000000fffffffb
Mar 15 00:17:26 hydra kernel: [  744.885315] R10: 0000000000000000 R11: 7fffffffffffff01 R12: ffff8802bfd43dc7
Mar 15 00:17:26 hydra kernel: [  744.885317] R13: ffff88023fd43dc8 R14: ffffffff8173e769 R15: 000000007fffffff
Mar 15 00:17:26 hydra kernel: [  744.885320] FS:  00007f2c58d18700(0000) GS:ffff88023fd40000(0000) knlGS:0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885322] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 15 00:17:26 hydra kernel: [  744.885324] CR2: 00007f2c584880d0 CR3: 000000000180b000 CR4: 00000000000007e0
Mar 15 00:17:26 hydra kernel: [  744.885326] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 15 00:17:26 hydra kernel: [  744.885332] Process swapper/5 (pid: 0, threadinfo ffff8802368be000, task ffff880236886360)
Mar 15 00:17:26 hydra kernel: [  744.885333] Stack:
Mar 15 00:17:26 hydra kernel: [  744.885334]  000000000000ffff 0000000000000000 0000000000000000 ffffffff8173e767
Mar 15 00:17:26 hydra kernel: [  744.885339]  0000000000000000 ffffffffff0a0003 0000000000000002 ffffffff8173e73d
Mar 15 00:17:26 hydra kernel: [  744.885343]  0000000000000009 ffff88023fd4d970 0000000000000005 ffff8802368be000
Mar 15 00:17:26 hydra kernel: [  744.885346] Call Trace:
Mar 15 00:17:26 hydra kernel: [  744.885348]  <IRQ> 
Mar 15 00:17:26 hydra kernel: [  744.885351]  [<ffffffff812b8cf7>] ? sprintf+0x46/0x4b
Mar 15 00:17:26 hydra kernel: [  744.885359]  [<ffffffff81089578>] ? print_cpu_stall_info+0xc0/0x119
Mar 15 00:17:26 hydra kernel: [  744.885363]  [<ffffffff815215bc>] ? printk+0x4f/0x54
Mar 15 00:17:26 hydra kernel: [  744.885367]  [<ffffffff8108969c>] ? rcu_pending+0xcb/0x3dc
Mar 15 00:17:26 hydra kernel: [  744.885372]  [<ffffffff8101aa42>] ? lapic_next_event+0xd/0x11
Mar 15 00:17:26 hydra kernel: [  744.885376]  [<ffffffff81064509>] ? clockevents_program_event+0xaa/0xce
Mar 15 00:17:26 hydra kernel: [  744.885381]  [<ffffffff8108afae>] ? rcu_check_callbacks+0x76/0x82
Mar 15 00:17:26 hydra kernel: [  744.885385]  [<ffffffff8103eedc>] ? update_process_times+0x31/0x63
Mar 15 00:17:26 hydra kernel: [  744.885389]  [<ffffffff810657ab>] ? tick_sched_timer+0x79/0x9f
Mar 15 00:17:26 hydra kernel: [  744.885393]  [<ffffffff8104d9cb>] ? __run_hrtimer.isra.26+0x4b/0xa3
Mar 15 00:17:26 hydra kernel: [  744.885397]  [<ffffffff8104dfb7>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 15 00:17:26 hydra kernel: [  744.885401]  [<ffffffff8101aef3>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 15 00:17:26 hydra kernel: [  744.885406]  [<ffffffff8152f04a>] ? apic_timer_interrupt+0x6a/0x70
Mar 15 00:17:26 hydra kernel: [  744.885408]  <EOI> 
Mar 15 00:17:26 hydra kernel: [  744.885409]  [<ffffffff8108a440>] ? rcu_eqs_exit_common.isra.50+0xbb/0xbb
Mar 15 00:17:26 hydra kernel: [  744.885417]  [<ffffffff8100895d>] ? cpu_idle+0xa8/0xe0
Mar 15 00:17:26 hydra kernel: [  744.885421]  [<ffffffff8151c55b>] ? start_secondary+0x23a/0x23f
Mar 15 00:17:26 hydra kernel: [  744.885423] Code: 0d 89 d0 83 c2 08 48 03 43 10 89 13 eb 0c 48 8b 43 08 48 8d 50 08 48 89 53 08 8b 00 66 89 44 24 2e e9 d0 02 00 00 f6 44 24 29 10 <75> 1e eb 0c 4c 39 e5 73 04 c6 45 00 20 48 ff c5 8b 44 24 2c ff 
Mar 15 00:17:26 hydra kernel: [  744.885452] NMI backtrace for cpu 4
Mar 15 00:17:26 hydra kernel: [  744.885455]    5: (9 GPs behind) idle=06d/1/0 drain=0 . timer not pending
Mar 15 00:17:26 hydra kernel: [  744.885460]     (t=91823 jiffies)
Mar 15 00:17:26 hydra kernel: [  744.885465] CPU 4 
Mar 15 00:17:26 hydra kernel: [  744.885471] Pid: 0, comm: swapper/4 Tainted: G        W    3.7.10-gentoo #1 Gigabyte Technology Co., Ltd. GA-990FXA-D3/GA-990FXA-D3
Mar 15 00:17:26 hydra kernel: [  744.885474] RIP: 0010:[<ffffffff8104665e>]  [<ffffffff8104665e>] __queue_work+0x44/0x203
Mar 15 00:17:26 hydra kernel: [  744.885482] RSP: 0018:ffff88023fd03c58  EFLAGS: 00000046
Mar 15 00:17:26 hydra kernel: [  744.885485] RAX: 0000000000000092 RBX: 0000000000000082 RCX: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885487] RDX: ffffffff81845b90 RSI: ffff8802368633c0 RDI: 0000000000000020
Mar 15 00:17:26 hydra kernel: [  744.885489] RBP: ffffffff81845b90 R08: 0000000000000000 R09: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885492] R10: 0039fc65c2afe9fe R11: 0039fc65c2afe9fe R12: ffff8802368633c0
Mar 15 00:17:26 hydra kernel: [  744.885494] R13: 0000000000000020 R14: 0000000000000000 R15: 0000000081840101
Mar 15 00:17:26 hydra kernel: [  744.885497] FS:  00007fbfd9065700(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885499] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 15 00:17:26 hydra kernel: [  744.885501] CR2: 00007fbfd821292e CR3: 000000021e463000 CR4: 00000000000007e0
Mar 15 00:17:26 hydra kernel: [  744.885504] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885506] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 15 00:17:26 hydra kernel: [  744.885509] Process swapper/4 (pid: 0, threadinfo ffff8802368bc000, task ffff880236885cc0)
Mar 15 00:17:26 hydra kernel: [  744.885510] Stack:
Mar 15 00:17:26 hydra kernel: [  744.885512]  00000000000000ea 0000000000000001 00000000000000ea 0000000000000082
Mar 15 00:17:26 hydra kernel: [  744.885516]  0000000000000020 ffff8802368633c0 ffffffff8184f11c 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885520]  0000000081840101 ffffffff81046873 0000000000000046 ffffffff81845b90
Mar 15 00:17:26 hydra kernel: [  744.885523] Call Trace:
Mar 15 00:17:26 hydra kernel: [  744.885525]  <IRQ> 
Mar 15 00:17:26 hydra kernel: [  744.885528]  [<ffffffff81046873>] ? queue_work_on+0x34/0x43
Mar 15 00:17:26 hydra kernel: [  744.885535]  [<ffffffff812f42b2>] ? kbd_event+0x55d/0x56c
Mar 15 00:17:26 hydra kernel: [  744.885540]  [<ffffffff813cc32e>] ? input_to_handler+0x9a/0xad
Mar 15 00:17:26 hydra kernel: [  744.885544]  [<ffffffff813ce332>] ? input_pass_values+0x62/0x123
Mar 15 00:17:26 hydra kernel: [  744.885548]  [<ffffffff81053147>] ? ttwu_do_wakeup+0xe/0x7e
Mar 15 00:17:26 hydra kernel: [  744.885552]  [<ffffffff813cecd0>] ? input_handle_event+0x45b/0x471
Mar 15 00:17:26 hydra kernel: [  744.885556]  [<ffffffff813cedc7>] ? input_event+0x50/0x75
Mar 15 00:17:26 hydra kernel: [  744.885560]  [<ffffffff81348da6>] ? acpi_button_notify+0x61/0x83
Mar 15 00:17:26 hydra kernel: [  744.885564]  [<ffffffff81326c0a>] ? acpi_device_notify_fixed+0x13/0x17
Mar 15 00:17:26 hydra kernel: [  744.885568]  [<ffffffff81326bf7>] ? acpi_device_notify+0x13/0x13
Mar 15 00:17:26 hydra kernel: [  744.885572]  [<ffffffff81331c00>] ? acpi_ev_fixed_event_detect+0xcc/0xf4
Mar 15 00:17:26 hydra kernel: [  744.885576]  [<ffffffff81333ddf>] ? acpi_ev_sci_xrupt_handler+0xf/0x23
Mar 15 00:17:26 hydra kernel: [  744.885579]  [<ffffffff81324b8d>] ? acpi_irq+0xe/0x29
Mar 15 00:17:26 hydra kernel: [  744.885584]  [<ffffffff81084337>] ? handle_irq_event_percpu+0x24/0x117
Mar 15 00:17:26 hydra kernel: [  744.885587]  [<ffffffff81084458>] ? handle_irq_event+0x2e/0x4e
Mar 15 00:17:26 hydra kernel: [  744.885592]  [<ffffffff810867cd>] ? handle_fasteoi_irq+0x6d/0x9d
Mar 15 00:17:26 hydra kernel: [  744.885595]  [<ffffffff8100398e>] ? handle_irq+0x1a/0x1e
Mar 15 00:17:26 hydra kernel: [  744.885598]  [<ffffffff810036dd>] ? do_IRQ+0x42/0x98
Mar 15 00:17:26 hydra kernel: [  744.885602]  [<ffffffff815298ea>] ? common_interrupt+0x6a/0x6a
Mar 15 00:17:26 hydra kernel: [  744.885604]  <EOI> 
Mar 15 00:17:26 hydra kernel: [  744.885606]  [<ffffffff8100821a>] ? default_idle+0x1e/0x32
Mar 15 00:17:26 hydra kernel: [  744.885613]  [<ffffffff8100835c>] ? amd_e400_idle+0xb7/0xd4
Mar 15 00:17:26 hydra kernel: [  744.885617]  [<ffffffff81008958>] ? cpu_idle+0xa3/0xe0
Mar 15 00:17:26 hydra kernel: [  744.885621]  [<ffffffff8151c55b>] ? start_secondary+0x23a/0x23f
Mar 15 00:17:26 hydra kernel: [  744.885624] Code: f6 c4 02 74 21 80 3d 1a c6 82 00 01 74 18 be cb 04 00 00 48 c7 c7 99 9c 73 81 e8 f2 e3 fe ff c6 05 00 c6 82 00 01 41 f6 04 24 40 <0f> 84 d1 00 00 00 4c 8b 35 7d e2 5b 00 ba 03 00 00 00 83 cf ff 
Mar 15 00:17:26 hydra kernel: [  744.885653] NMI backtrace for cpu 0
Mar 15 00:17:26 hydra kernel: [  744.885658] CPU 0 
Mar 15 00:17:26 hydra kernel: [  744.885665] Pid: 0, comm: swapper/0 Tainted: G        W    3.7.10-gentoo #1 Gigabyte Technology Co., Ltd. GA-990FXA-D3/GA-990FXA-D3
Mar 15 00:17:26 hydra kernel: [  744.885668] RIP: 0010:[<ffffffff8100821a>]  [<ffffffff8100821a>] default_idle+0x1e/0x32
Mar 15 00:17:26 hydra kernel: [  744.885677] RSP: 0018:ffffffff81801f60  EFLAGS: 00000246
Mar 15 00:17:26 hydra kernel: [  744.885679] RAX: 0000000000000000 RBX: ffffffff8187f8d0 RCX: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885682] RDX: 000000000000d4d4 RSI: 0000000000000003 RDI: ffffffff81800000
Mar 15 00:17:26 hydra kernel: [  744.885684] RBP: ffffffff818c9000 R08: 0000000000000000 R09: ffff88023fc0d6c0
Mar 15 00:17:26 hydra kernel: [  744.885686] R10: ffffffff8187f8d0 R11: ffffffff8187f8d0 R12: ffffffff818d20a0
Mar 15 00:17:26 hydra kernel: [  744.885688] R13: ffff88023ffce580 R14: 0000000000000000 R15: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885691] FS:  00007f034f52d700(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885693] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 15 00:17:26 hydra kernel: [  744.885695] CR2: 00007f034ee48630 CR3: 000000000180b000 CR4: 00000000000007f0
Mar 15 00:17:26 hydra kernel: [  744.885697] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885700] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 15 00:17:26 hydra kernel: [  744.885703] Process swapper/0 (pid: 0, threadinfo ffffffff81800000, task ffffffff818133f0)
Mar 15 00:17:26 hydra kernel: [  744.885704] Stack:
Mar 15 00:17:26 hydra kernel: [  744.885706]  ffffffff8100835c ffffffff8187f8d0 00000000818c9000 ffffffff818d20a0
Mar 15 00:17:26 hydra kernel: [  744.885710]  ffffffff81008958 0000000000000000 50bd6e7527e9ba29 ffffffffffffffff
Mar 15 00:17:26 hydra kernel: [  744.885713]  ffffffff81896a25 ffffffff81896539 0000000000986000 ffffffff818d20a0
Mar 15 00:17:26 hydra kernel: [  744.885717] Call Trace:
Mar 15 00:17:26 hydra kernel: [  744.885722]  [<ffffffff8100835c>] ? amd_e400_idle+0xb7/0xd4
Mar 15 00:17:26 hydra kernel: [  744.885727]  [<ffffffff81008958>] ? cpu_idle+0xa3/0xe0
Mar 15 00:17:26 hydra kernel: [  744.885732]  [<ffffffff81896a25>] ? start_kernel+0x305/0x310
Mar 15 00:17:26 hydra kernel: [  744.885735]  [<ffffffff81896539>] ? repair_env_string+0x54/0x54
Mar 15 00:17:26 hydra kernel: [  744.885737] Code: 02 f3 90 e8 d3 ff ff ff 85 c0 74 f5 c3 65 48 8b 04 25 20 b8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 e8 b6 ff ff ff 85 c0 75 04 fb f4 <eb> 01 fb 65 48 8b 04 25 20 b8 00 00 83 88 3c e0 ff ff 04 c3 48 
Mar 15 00:17:26 hydra kernel: [  744.885766] NMI backtrace for cpu 1
Mar 15 00:17:26 hydra kernel: [  744.885770] CPU 1 
Mar 15 00:17:26 hydra kernel: [  744.885776] Pid: 0, comm: swapper/1 Tainted: G        W    3.7.10-gentoo #1 Gigabyte Technology Co., Ltd. GA-990FXA-D3/GA-990FXA-D3
Mar 15 00:17:26 hydra kernel: [  744.885780] RIP: 0010:[<ffffffff8100821a>]  [<ffffffff8100821a>] default_idle+0x1e/0x32
Mar 15 00:17:26 hydra kernel: [  744.885789] RSP: 0018:ffff8802368b7ef0  EFLAGS: 00000246
Mar 15 00:17:26 hydra kernel: [  744.885792] RAX: 0000000000000000 RBX: ffffffff8187f8d0 RCX: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885794] RDX: 000000000000f4f4 RSI: 0000000000000003 RDI: ffff8802368b6000
Mar 15 00:17:26 hydra kernel: [  744.885796] RBP: 0000000000000000 R08: 0000000000000000 R09: ffff88023fc4d6c0
Mar 15 00:17:26 hydra kernel: [  744.885798] R10: ffffffff8187f8d0 R11: ffffffff8187f8d0 R12: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885800] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885803] FS:  00007f2c58d18700(0000) GS:ffff88023fc40000(0000) knlGS:0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885805] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 15 00:17:26 hydra kernel: [  744.885807] CR2: 00007f2c584880d0 CR3: 000000000180b000 CR4: 00000000000007e0
Mar 15 00:17:26 hydra kernel: [  744.885810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885812] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 15 00:17:26 hydra kernel: [  744.885815] Process swapper/1 (pid: 0, threadinfo ffff8802368b6000, task ffff8802368848e0)
Mar 15 00:17:26 hydra kernel: [  744.885816] Stack:
Mar 15 00:17:26 hydra kernel: [  744.885818]  ffffffff8100835c ffffffff8187f8d0 0000000100000000 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885822]  ffffffff81008958 000000007fe0f559 f342466239f1ec55 00000000100000a1
Mar 15 00:17:26 hydra kernel: [  744.885826]  ffffffff8151c55b 0000000000000000 5e0ecbf35dbcac94 0000000000000010
Mar 15 00:17:26 hydra kernel: [  744.885829] Call Trace:
Mar 15 00:17:26 hydra kernel: [  744.885834]  [<ffffffff8100835c>] ? amd_e400_idle+0xb7/0xd4
Mar 15 00:17:26 hydra kernel: [  744.885839]  [<ffffffff81008958>] ? cpu_idle+0xa3/0xe0
Mar 15 00:17:26 hydra kernel: [  744.885843]  [<ffffffff8151c55b>] ? start_secondary+0x23a/0x23f
Mar 15 00:17:26 hydra kernel: [  744.885846] Code: 02 f3 90 e8 d3 ff ff ff 85 c0 74 f5 c3 65 48 8b 04 25 20 b8 00 00 83 a0 3c e0 ff ff fb 0f ae f0 e8 b6 ff ff ff 85 c0 75 04 fb f4 <eb> 01 fb 65 48 8b 04 25 20 b8 00 00 83 88 3c e0 ff ff 04 c3 48 
Mar 15 00:17:26 hydra kernel: [  744.885872] NMI backtrace for cpu 3
Mar 15 00:17:26 hydra kernel: [  744.885874] CPU 3 
Mar 15 00:17:26 hydra kernel: [  744.885878] Pid: 5955, comm: cc1 Tainted: G        W    3.7.10-gentoo #1 Gigabyte Technology Co., Ltd. GA-990FXA-D3/GA-990FXA-D3
Mar 15 00:17:26 hydra kernel: [  744.885881] RIP: 0010:[<ffffffff8105fb0e>]  [<ffffffff8105fb0e>] ktime_get+0x2b/0x78
Mar 15 00:17:26 hydra kernel: [  744.885885] RSP: 0000:ffff88023fcc3eb8  EFLAGS: 00000002
Mar 15 00:17:26 hydra kernel: [  744.885887] RAX: ffffffff81065732 RBX: ffff88023fccd7c0 RCX: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885889] RDX: ffff880234dd2000 RSI: ffff88023fccd610 RDI: ffff88023fccd7c0
Mar 15 00:17:26 hydra kernel: [  744.885892] RBP: 0000000000093cdf R08: 00000098537253d8 R09: 00007ff3f6734670
Mar 15 00:17:26 hydra kernel: [  744.885894] R10: 00007ff3f6734670 R11: 7fffffffffffffff R12: ffff8801c9173f58
Mar 15 00:17:26 hydra kernel: [  744.885896] R13: ffff88023fccd678 R14: ffff88023fccd6b8 R15: 0000000000000001
Mar 15 00:17:26 hydra kernel: [  744.885899] FS:  00007ff3f7628800(0000) GS:ffff88023fcc0000(0000) knlGS:0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885901] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 15 00:17:26 hydra kernel: [  744.885903] CR2: 0000000000690005 CR3: 000000021e62d000 CR4: 00000000000007e0
Mar 15 00:17:26 hydra kernel: [  744.885905] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 15 00:17:26 hydra kernel: [  744.885908] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 15 00:17:26 hydra kernel: [  744.885910] Process cc1 (pid: 5955, threadinfo ffff8801c9172000, task ffff88022f4d1a80)
Mar 15 00:17:26 hydra kernel: [  744.885911] Stack:
Mar 15 00:17:26 hydra kernel: [  744.885913]  ffff88023fccd610 ffff88023fccd7c0 ffff88023fccd600 ffffffff81065747
Mar 15 00:17:26 hydra kernel: [  744.885917]  ffff88023fccd7c0 ffff88023fccd600 ffff88023fccd5c0 ffffffff8104d9cb
Mar 15 00:17:26 hydra kernel: [  744.885920]  00000098537253d8 ffff88023fccd5c0 0000000000000003 00000098537253d8
Mar 15 00:17:26 hydra kernel: [  744.885923] Call Trace:
Mar 15 00:17:27 hydra kernel: [  744.885925]  <IRQ> 
Mar 15 00:17:27 hydra kernel: [  744.885926]  [<ffffffff81065747>] ? tick_sched_timer+0x15/0x9f
Mar 15 00:17:27 hydra kernel: [  744.885933]  [<ffffffff8104d9cb>] ? __run_hrtimer.isra.26+0x4b/0xa3
Mar 15 00:17:27 hydra kernel: [  744.885937]  [<ffffffff8104dfb7>] ? hrtimer_interrupt+0xd9/0x1c9
Mar 15 00:17:27 hydra kernel: [  744.885941]  [<ffffffff8101aef3>] ? smp_apic_timer_interrupt+0x6e/0x80
Mar 15 00:17:27 hydra kernel: [  744.885945]  [<ffffffff8152f04a>] ? apic_timer_interrupt+0x6a/0x70
Mar 15 00:17:27 hydra kernel: [  744.885947]  <EOI> 
Mar 15 00:17:27 hydra kernel: [  744.885949] Code: 55 53 56 83 3d 8b fe 81 00 00 74 11 be ff 00 00 00 48 c7 c7 b3 b2 73 81 e8 44 4f fd ff 8b 2d 0a ae 8f 00 40 f6 c5 01 74 04 f3 90 <eb> f0 48 8b 1d b9 ad 8f 00 48 03 1d 92 ad 8f 00 e8 d3 fe ff ff 
Mar 15 00:17:27 hydra shutdown[6010]: shutting down for system halt

Re: [gentoo-user] System freezes during compiles

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 2513 bytes --]

On Wednesday 20 Mar 2013 04:42:28 Carlos Hendson wrote:
> Hello,
> 
> For last few weeks or so, I've been getting intermittent hard lock-ups
> during the emerge of various packages.  It appears the more compile
> intensive the package, the more likely the lock-up.  These lock-ups have
> occurred under kernels 3.4.9 and 3.7.10 with gcc 4.5.4 and 4.6.3.
> 
> Once the machine is in a frozen state, the only thing that responds is
> the soft power reset button.  Some times the machine lock-ups again
> after the button is pressed (this is because the compile resumes once
> the system comes out of it's frozen state).
> 
> If the system subsequently lock-ups because I wasn't able to cancel the
> compile fast enough only a only option left is a hard power reset (10sec
> + hold power button).  If I cancel the compile, the system is perfectly
> responsive and functions normally.
> 
> There are kernel stack traces in /var/log/messages which I'm unable to
> decipher and diagnose as to what caused the lock-up.
> 
> If I had to guess, I'd blame an incorrect setting in the .config, but
> since I'm stuck in the diagnostic of what part of the kernel might be
> experiencing the problem, I need a bit of help to pin point the issue.
> 
> I believe it to be a kernel configuration issue because when I booted
> the machine using a system rescue Live CD, I was able to chroot into the
> system and emerge packages like gcc without the lock-up problem
> occurring.
> 
> That's by no means conclusive, however, I've also run a complete pass of
> memcheck for over an hour without any issues reported.
> 
> I'd like to completely rule out hardware failure, what diagnostic tools
> tools are recommend to try identify potential hardware issue of this
> type?
> 
> The various kernel stack traces are attached in case someone wants to
> take a look.  I can provide more information should it be needed.
> 
> Any help or advice would be appreciated.
> 
> Regards,
> Carlos

Stating the obvious, it seems that the kernel is struggling and indeed you may 
have come across some nasty kernel bug.  However, it could well be that it is 
not related to the kernel you're running, or your kernel config.  It could be 
a problem with the power supply being faulty and causing these lock ups.

Unless someone else comes up with a better idea to troubleshoot it further, I 
would consider replacing the power supply with another of a known good 
condition.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] System freezes during compiles

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

On Wed, 20 Mar 2013 08:17:11 +0000, Mick wrote:

> Stating the obvious, it seems that the kernel is struggling and indeed
> you may have come across some nasty kernel bug.  However, it could well
> be that it is not related to the kernel you're running, or your kernel
> config.  It could be a problem with the power supply being faulty and
> causing these lock ups.

That's certainly possible, it could also be failing memory, and it's
cheaper to run memtest86+ before buying a new power supply ;-)


-- 
Neil Bothwick

Only an idiot actually READS taglines.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: System freezes during compiles

[Grant Edwards]

On 2013-03-20, Carlos Hendson <skyclan@gmx.net> wrote:

> That's by no means conclusive, however, I've also run a complete pass of
> memcheck for over an hour without any issues reported.

FWIW.  I've had flakey memory that ran memcheck fine for several hours
and multiple passes -- but if I let it run long enough, it would fail.
I wouldn't be confident unless memtest ran for at least 12 hours (24
would be even better).

I'd also keep an eye on CPU core temperature. 

A failing hard-drive can also cause some pretty strange behavior.  If
you're drives are smart (AFAICT, all recent ones are), ask them how
they're feeling with 'smartclt' or something like that.

-- 
Grant Edwards               grant.b.edwards        Yow! I'm having BEAUTIFUL
                                  at               THOUGHTS about the INSIPID
                              gmail.com            WIVES of smug and wealthy
                                                   CORPORATE LAWYERS ...

Re: [gentoo-user] Re: System freezes during compiles

[Paul Hartman]

On Wed, Mar 20, 2013 at 11:14 AM, Grant Edwards
<grant.b.edwards@gmail.com> wrote:
>
> On 2013-03-20, Carlos Hendson <skyclan@gmx.net> wrote:
>
> > That's by no means conclusive, however, I've also run a complete pass of
> > memcheck for over an hour without any issues reported.
>
> FWIW.  I've had flakey memory that ran memcheck fine for several hours
> and multiple passes -- but if I let it run long enough, it would fail.
> I wouldn't be confident unless memtest ran for at least 12 hours (24
> would be even better).


When I've had memory problems, it seemed like it was always shown in
tests 5 and 8 from memtest86+. So, now, to expedite the tests, I set
it to only run tests 5 and 8. A few hours of those can find problems
faster than a couple days of running the full battery of tests. I
always run the full set at least once, but my experience and those of
people I've seen on Google seems to indicate that on modern systems 5
and 8 are where errors are exposed. YMMV of course. :)

[gentoo-user] Re: System freezes during compiles

[Grant Edwards]

On 2013-03-20, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> On 2013-03-20, Carlos Hendson <skyclan@gmx.net> wrote:
>
>> That's by no means conclusive, however, I've also run a complete pass of
>> memcheck for over an hour without any issues reported.
>
> FWIW.  I've had flakey memory that ran memcheck fine for several hours
> and multiple passes -- but if I let it run long enough, it would fail.
> I wouldn't be confident unless memtest ran for at least 12 hours (24
> would be even better).

I should probably be more clear: I was talking about memtest86+ when I
wrote "memcheck".

> I'd also keep an eye on CPU core temperature. 
>
> A failing hard-drive can also cause some pretty strange behavior.  If
> you're drives are smart (AFAICT, all recent ones are), ask them how
> they're feeling with 'smartclt' or something like that.

-- 
Grant Edwards               grant.b.edwards        Yow! Mr and Mrs PED, can I
                                  at               borrow 26.7% of the RAYON
                              gmail.com            TEXTILE production of the
                                                   INDONESIAN archipelago?

Re: [gentoo-user] System freezes during compiles

[Daniel Wagener]

On Wed, 20 Mar 2013 05:42:28 +0100
Carlos Hendson <skyclan@gmx.net> wrote:

> Hello,
> 
> For last few weeks or so, I've been getting intermittent hard lock-ups
> during the emerge of various packages.  It appears the more compile
> intensive the package, the more likely the lock-up.  These lock-ups have
> occurred under kernels 3.4.9 and 3.7.10 with gcc 4.5.4 and 4.6.3.
> 
> Once the machine is in a frozen state, the only thing that responds is
> the soft power reset button.  Some times the machine lock-ups again
> after the button is pressed (this is because the compile resumes once
> the system comes out of it's frozen state).
> 
> If the system subsequently lock-ups because I wasn't able to cancel the
> compile fast enough only a only option left is a hard power reset (10sec
> + hold power button).  If I cancel the compile, the system is perfectly
> responsive and functions normally.
> 
> There are kernel stack traces in /var/log/messages which I'm unable to
> decipher and diagnose as to what caused the lock-up.
> 
> If I had to guess, I'd blame an incorrect setting in the .config, but
> since I'm stuck in the diagnostic of what part of the kernel might be
> experiencing the problem, I need a bit of help to pin point the issue.  
> 
> I believe it to be a kernel configuration issue because when I booted
> the machine using a system rescue Live CD, I was able to chroot into the
> system and emerge packages like gcc without the lock-up problem
> occurring.  
> 
> That's by no means conclusive, however, I've also run a complete pass of
> memcheck for over an hour without any issues reported.
> 
> I'd like to completely rule out hardware failure, what diagnostic tools
> tools are recommend to try identify potential hardware issue of this
> type?
> 
> The various kernel stack traces are attached in case someone wants to
> take a look.  I can provide more information should it be needed.
> 
> Any help or advice would be appreciated.
> 
> Regards,
> Carlos 

"Frozen" means there is no Hard Drive Activity going on right?
And there is no other indication, that you are just running out of memory?

-- 

Re: [gentoo-user] System freezes during compiles

[Volker Armin Hemmann]

Am 20.03.2013 05:42, schrieb Carlos Hendson:
> Hello,
>
> For last few weeks or so, I've been getting intermittent hard lock-ups
> during the emerge of various packages.  It appears the more compile
> intensive the package, the more likely the lock-up.  These lock-ups have
> occurred under kernels 3.4.9 and 3.7.10 with gcc 4.5.4 and 4.6.3.
>
> Once the machine is in a frozen state, the only thing that responds is
> the soft power reset button.  Some times the machine lock-ups again
> after the button is pressed (this is because the compile resumes once
> the system comes out of it's frozen state).
>
> If the system subsequently lock-ups because I wasn't able to cancel the
> compile fast enough only a only option left is a hard power reset (10sec
> + hold power button).  If I cancel the compile, the system is perfectly
> responsive and functions normally.
>
> There are kernel stack traces in /var/log/messages which I'm unable to
> decipher and diagnose as to what caused the lock-up.
>
> If I had to guess, I'd blame an incorrect setting in the .config, but
> since I'm stuck in the diagnostic of what part of the kernel might be
> experiencing the problem, I need a bit of help to pin point the issue.  
>
> I believe it to be a kernel configuration issue because when I booted
> the machine using a system rescue Live CD, I was able to chroot into the
> system and emerge packages like gcc without the lock-up problem
> occurring.  
>
> That's by no means conclusive, however, I've also run a complete pass of
> memcheck for over an hour without any issues reported.
>
> I'd like to completely rule out hardware failure, what diagnostic tools
> tools are recommend to try identify potential hardware issue of this
> type?
>
> The various kernel stack traces are attached in case someone wants to
> take a look.  I can provide more information should it be needed.
>
> Any help or advice would be appreciated.
>
> Regards,
> Carlos 
you might just hit a thrashing situation. Linux is very bad when it
comes to abusing swap in case of an emergency.

But it also sounds like overheating or a power problem. Power problems
might be caused by the PSU - but it could also be the power circuitry of
your mobo.

Re: [gentoo-user] System freezes during compiles

[Paul Hartman]

On Tue, Mar 19, 2013 at 11:42 PM, Carlos Hendson <skyclan@gmx.net> wrote:
> For last few weeks or so, I've been getting intermittent hard lock-ups
> during the emerge of various packages.  It appears the more compile
> intensive the package, the more likely the lock-up.  These lock-ups have
> occurred under kernels 3.4.9 and 3.7.10 with gcc 4.5.4 and 4.6.3.


I had a virtual server that kept crashing/rebooting during compiles of
large packages such as php. It ended up being because it was running
out of memory. Added another 1GB of swap space and it has been happy
ever since.

Re: [gentoo-user] System freezes during compiles

[Carlos Hendson]

On Wed, 2013-03-20 at 18:43 +0100, Daniel Wagener wrote:
> "Frozen" means there is no Hard Drive Activity going on right?
> And there is no other indication, that you are just running out of
> memory? 

I can't categorically state if there was drive activity.  I was so
fixated on regaining control of the machine that I failed to pay
attention to the state of the HDD LED.  I'll make a point of checking it
the next time the machine appears to freeze.

I saw no other indications of memory exhaustion after the system came
back from the "soft-power reset" button being pressed.

Regards,
Carlos

Re: [gentoo-user] System freezes during compiles

[Carlos Hendson]

On Wed, 2013-03-20 at 08:17 +0000, Mick wrote:
> Stating the obvious, it seems that the kernel is struggling and indeed
> you may 
> have come across some nasty kernel bug.  However, it could well be
> that it is 
> not related to the kernel you're running, or your kernel config.  It
> could be 
> a problem with the power supply being faulty and causing these lock
> ups.
> 
> Unless someone else comes up with a better idea to troubleshoot it
> further, I 
> would consider replacing the power supply with another of a known
> good 
> condition. 

Thanks for the good advice Mick.  I don't have spare hardware on-tap so
switching psu, memory or processor may prove to be tricky.  It's one of
those catch 22's where I don't want to spend on components that aren't
faulty, however I need to spend on components to test if they're faulty.

I've been given a few other test to perform before I start moving to
hardware replacement.

Regards,
Carlos

Re: [gentoo-user] System freezes during compiles

[Carlos Hendson]

On Wed, 2013-03-20 at 20:57 +0100, Volker Armin Hemmann wrote:
> you might just hit a thrashing situation. Linux is very bad when it
> comes to abusing swap in case of an emergency.
> 
> But it also sounds like overheating or a power problem. Power problems
> might be caused by the PSU - but it could also be the power circuitry
> of
> your mobo. 

It's not a thrashing issue as I don't have any swap.  The 8GB of ram has
been sufficient memory for all tasks thus far.  I have no objection to
allocating some swap space if it could resolve the issue.

Actually, Grant and you both suggested possible heat issues which has
just made me think that I should check for dust build up in the CPU heat
sink.  There so much dust where I live that I have to vacuum dust build
up from the case.

The sensors tool reports 51C, it doesn't appear to be running too hot,
but I don't have a baseline to compare it to.  I see I need to implement
monitoring for this machine once it's stable again.

k10temp-pci-00c3
Adapter: PCI adapter
temp1:        +51.0°C  (high = +70.0°C)
                       (crit = +71.0°C, hyst = +66.0°C)


I'll give the inside a clean this weekend and see if there's any
improvement.

Thanks for the suggestions.

Regards,
Carlos

Re: [gentoo-user] Re: System freezes during compiles

[Carlos Hendson]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

On Wed, 2013-03-20 at 16:14 +0000, Grant Edwards wrote:
> On 2013-03-20, Carlos Hendson <skyclan@gmx.net> wrote:
> 
> > That's by no means conclusive, however, I've also run a complete pass of
> > memcheck for over an hour without any issues reported.
> 
> FWIW.  I've had flakey memory that ran memcheck fine for several hours
> and multiple passes -- but if I let it run long enough, it would fail.
> I wouldn't be confident unless memtest ran for at least 12 hours (24
> would be even better).
> 
> I'd also keep an eye on CPU core temperature. 
> 
> A failing hard-drive can also cause some pretty strange behavior.  If
> you're drives are smart (AFAICT, all recent ones are), ask them how
> they're feeling with 'smartclt' or something like that.
> 

I'll run a 24 hour memtest this weekend.

I started a long test on the hard drive:

 smartctl -t long /dev/sda

smartctl -a /dev/sda also appears to indicate various errors (the output
is attached).  I'll trying to track down some documentation as to what
they're actually reporting.

Looking at the difference between the output of smartctl for before and
during the test, there has been an increase in errors detected for ID
#195

Before test:
195 Hardware_ECC_Recovered  0x001a   001   001   000    Old_age   Always
-       241822

During test:
195 Hardware_ECC_Recovered  0x001a   001   001   000    Old_age   Always
-       243582

Could this be the cause of the stalls during compiles?  If it is the
cause, is it possible for the kernel to detect such failures and report
them?

Thanks to you and everyone else for your ideas and suggestions.

Regards,
Carlos

[-- Attachment #2: smartctl_log.txt --]
[-- Type: text/plain, Size: 10195 bytes --]

smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.7.10-gentoo] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     SAMSUNG SpinPoint M5
Device Model:     SAMSUNG HM250JI
Serial Number:    S0TVJQSQ501163
LU WWN Device Id: 5 0f0000 031501163
Firmware Version: HS100-10
User Capacity:    250,059,350,016 bytes [250 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 0
Local Time is:    Wed Mar 20 20:55:00 2013 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (  32)	The self-test routine was interrupted
					by the host with a hard or soft reset.
Total time to complete Offline 
data collection: 		(  103) seconds.
Offline data collection
capabilities: 			 (0x51) SMART execute Offline immediate.
					No Auto Offline data collection support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 103) minutes.
SCT capabilities: 	       (0x003f)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   100   100   051    Pre-fail  Always       -       40
  3 Spin_Up_Time            0x0007   252   252   025    Pre-fail  Always       -       2000
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       2012
  5 Reallocated_Sector_Ct   0x0033   054   054   010    Pre-fail  Always       -       428
  7 Seek_Error_Rate         0x000f   252   252   051    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0025   252   252   015    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       7095
 10 Spin_Retry_Count        0x0033   252   252   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0032   099   099   000    Old_age   Always       -       1151
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1112
184 End-to-End_Error        0x0033   252   252   070    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       34
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       57
190 Airflow_Temperature_Cel 0x0022   069   040   040    Old_age   Always   In_the_past 31 (Min/Max 8/60)
191 G-Sense_Error_Rate      0x0032   090   090   000    Old_age   Always       -       103348
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       332
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       9937
194 Temperature_Celsius     0x0022   064   040   000    Old_age   Always       -       31 (Min/Max 8/60)
195 Hardware_ECC_Recovered  0x001a   001   001   000    Old_age   Always       -       241822
196 Reallocated_Event_Count 0x0032   252   252   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   252   252   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   252   252   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age   Always       -       27
200 Multi_Zone_Error_Rate   0x000a   100   100   000    Old_age   Always       -       2
201 Soft_Read_Error_Rate    0x0032   252   252   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 26 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 26 occurred at disk power-on lifetime: 6959 hours (289 days + 23 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 08 a8 4a e0  Error: UNC 8 sectors at LBA = 0x004aa808 = 4892680

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 08 a8 4a e0 00      00:02:31.687  READ DMA
  c8 00 10 68 40 6b e0 00      00:02:24.250  READ DMA
  ef 05 fe 00 00 00 40 00      00:02:24.187  SET FEATURES [Enable APM]
  c8 00 10 e0 a7 4a e0 00      00:02:24.187  READ DMA
  c8 00 00 48 99 c7 e0 00      00:02:24.187  READ DMA

Error 25 occurred at disk power-on lifetime: 5683 hours (236 days + 19 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 20 40 9e 04 e1  Error: UNC 32 sectors at LBA = 0x01049e40 = 17079872

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 20 40 9e 04 e1 00      00:15:33.812  READ DMA
  c8 00 10 28 c3 94 e0 00      00:15:26.375  READ DMA
  ef 05 fe 00 00 00 40 00      00:15:26.312  SET FEATURES [Enable APM]
  c8 00 20 20 9e 04 e1 00      00:15:26.312  READ DMA
  c8 00 08 e0 a9 73 e0 00      00:15:26.312  READ DMA

Error 24 occurred at disk power-on lifetime: 5534 hours (230 days + 14 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 a0 c6 54 e0  Error: UNC 8 sectors at LBA = 0x0054c6a0 = 5555872

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 a0 c6 54 e0 00      00:10:28.500  READ DMA
  c8 00 68 d8 2b cc e0 00      00:10:21.062  READ DMA
  ef c2 00 00 00 00 40 00      00:10:21.062  SET FEATURES [Disable AAM]
  c8 00 10 58 c2 54 e0 00      00:10:21.062  READ DMA
  c8 00 20 b8 2b cc e0 00      00:10:21.062  READ DMA

Error 23 occurred at disk power-on lifetime: 5463 hours (227 days + 15 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 78 68 03 e0  Error: UNC at LBA = 0x00036878 = 223352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 00 78 68 03 e0 00      00:50:45.250  READ DMA
  c8 00 00 68 c4 83 e0 00      00:50:37.812  READ DMA
  ef 05 fe 00 00 00 40 00      00:50:37.750  SET FEATURES [Enable APM]
  c8 00 00 78 67 03 e0 00      00:50:37.750  READ DMA
  c8 00 08 70 17 ab e0 00      00:50:37.750  READ DMA

Error 22 occurred at disk power-on lifetime: 5383 hours (224 days + 7 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 e8 4f 46 e0  Error: UNC at LBA = 0x00464fe8 = 4607976

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 00 e8 4f 46 e0 00      00:02:46.250  READ DMA
  c8 00 10 98 0f f5 e0 00      00:02:38.812  READ DMA
  ef 05 fe 00 00 00 40 00      00:02:38.750  SET FEATURES [Enable APM]
  c8 00 00 e8 4e 46 e0 00      00:02:38.750  READ DMA
  c8 00 88 e0 1f 3d e1 00      00:02:38.750  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%        95         -

Note: selective self-test log revision number (0) not 1 implies that no selective self-test has ever been run
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Interrupted [00% left] (0-65535)
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Re: [gentoo-user] System freezes during compiles

[Carlos Hendson]

On Wed, 2013-03-20 at 16:27 -0500, Paul Hartman wrote:
> 
> I had a virtual server that kept crashing/rebooting during compiles of
> large packages such as php. It ended up being because it was running
> out of memory. Added another 1GB of swap space and it has been happy
> ever since. 

Thanks Paul. Volker suggested a possible caused was swap.  I'll allocate
some swap space after the smartctl self-test finishes and try to
recompile gcc a few times.

Regards,
Carlos

Re: [gentoo-user] System freezes during compiles

[Volker Armin Hemmann]

[-- Attachment #1: Type: text/plain, Size: 1470 bytes --]

You got your answer. 8gig and no swap is NOT ENOUGHT.
Am 20.03.2013 22:51 schrieb "Carlos Hendson" <skyclan@gmx.net>:

> On Wed, 2013-03-20 at 20:57 +0100, Volker Armin Hemmann wrote:
> > you might just hit a thrashing situation. Linux is very bad when it
> > comes to abusing swap in case of an emergency.
> >
> > But it also sounds like overheating or a power problem. Power problems
> > might be caused by the PSU - but it could also be the power circuitry
> > of
> > your mobo.
>
> It's not a thrashing issue as I don't have any swap.  The 8GB of ram has
> been sufficient memory for all tasks thus far.  I have no objection to
> allocating some swap space if it could resolve the issue.
>
> Actually, Grant and you both suggested possible heat issues which has
> just made me think that I should check for dust build up in the CPU heat
> sink.  There so much dust where I live that I have to vacuum dust build
> up from the case.
>
> The sensors tool reports 51C, it doesn't appear to be running too hot,
> but I don't have a baseline to compare it to.  I see I need to implement
> monitoring for this machine once it's stable again.
>
> k10temp-pci-00c3
> Adapter: PCI adapter
> temp1:        +51.0°C  (high = +70.0°C)
>                        (crit = +71.0°C, hyst = +66.0°C)
>
>
> I'll give the inside a clean this weekend and see if there's any
> improvement.
>
> Thanks for the suggestions.
>
> Regards,
> Carlos
>
>
>

[-- Attachment #2: Type: text/html, Size: 1830 bytes --]

Re: [gentoo-user] System freezes during compiles

[Carlos Hendson]

On Thu, 2013-03-21 at 06:45 +0100, Volker Armin Hemmann wrote:
> You got your answer. 8gig and no swap is NOT ENOUGHT.

It's a strong indicator, which is going to be corrected.

I am slightly confused by the resulting behaviour however.  I was of the
impression oomkiller would start to kill processes when unallocated
memory is getting scarce?

How would no free memory cause CPU stalls?

Regards,
Carlos

Re: [gentoo-user] System freezes during compiles

[Volker Armin Hemmann]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

Oom Killer is Not instant, can take a long time or get stuck or kills
something vital.
...
Am 21.03.2013 07:52 schrieb "Carlos Hendson" <skyclan@gmx.net>:

> On Thu, 2013-03-21 at 06:45 +0100, Volker Armin Hemmann wrote:
> > You got your answer. 8gig and no swap is NOT ENOUGHT.
>
> It's a strong indicator, which is going to be corrected.
>
> I am slightly confused by the resulting behaviour however.  I was of the
> impression oomkiller would start to kill processes when unallocated
> memory is getting scarce?
>
> How would no free memory cause CPU stalls?
>
> Regards,
> Carlos
>
>
>

[-- Attachment #2: Type: text/html, Size: 883 bytes --]

[gentoo-user] Re: System freezes during compiles

[Grant Edwards]

On 2013-03-20, Carlos Hendson <skyclan@gmx.net> wrote:

> Looking at the difference between the output of smartctl for before
> and during the test, there has been an increase in errors detected
> for ID #195
>
> Before test:
> 195 Hardware_ECC_Recovered  0x001a   001   001   000    Old_age   Always
> -       241822
>
> During test:
> 195 Hardware_ECC_Recovered  0x001a   001   001   000    Old_age   Always
> -       243582
>
> Could this be the cause of the stalls during compiles?  If it is the
> cause, is it possible for the kernel to detect such failures and report
> them?

I think that as long as the errors are "recovered" they shouldn't
cause problems.  The $64 question is whether those errors are
indicating that you're headed for unrecoverable errors (which will
cause problems).  If they seem to be increasing in frequency, I'd
probably be a little worried.

-- 
Grant Edwards               grant.b.edwards        Yow! SHHHH!!  I hear SIX
                                  at               TATTOOED TRUCK-DRIVERS
                              gmail.com            tossing ENGINE BLOCKS into
                                                   empty OIL DRUMS ...

Re: [gentoo-user] System freezes during compiles [SOLVED]

[Carlos Hendson]

On Wed, 2013-03-20 at 20:57 +0100, Volker Armin Hemmann wrote:
> you might just hit a thrashing situation. Linux is very bad when it
> comes to abusing swap in case of an emergency.
> 
> But it also sounds like overheating or a power problem. Power problems
> might be caused by the PSU - but it could also be the power circuitry
> of
> your mobo. 

First of all, thank you to everyone for the superb help and suggestions
regarding this problem.

Yesterday, I enabled some swap space, but the system froze on the first
attempt at compiling glibc.

The next cheapest option was to clean the case of dust.  The CPU heat
sink was clogged with a think layer of dust.  After thoroughly cleaning
the case, the system compiled glibc, the kernel, qtcore and other
packages without freezing.

The only downside is since the fins on the heat sink are exposed
directly to the fan again, the noise level has gone up.

When I checked the RPMs in the BIOS I noticed a setting which states,
decrease CPU voltage and frequency in the event of a temperature
threshold being exceeded.  This would explain the kernel watchdog
messages reporting stalls were detected.


For anyone that's curious, here's the output of sensors and free during
the compile of glibc.  Swap wasn't being touched at all, there's still
4GB of memory free.  The cpu was getting close to the threshold limit
even after the heat sink was cleaned of dust.

k10temp-pci-00c3
Adapter: PCI adapter
temp1:        +58.9°C  (high = +70.0°C)
                       (crit = +71.0°C, hyst = +66.0°C)

it8720-isa-0228
Adapter: ISA adapter
in0:          +1.49 V  (min =  +0.00 V, max =  +4.08 V)
in1:          +1.47 V  (min =  +0.00 V, max =  +4.08 V)
in2:          +3.38 V  (min =  +0.00 V, max =  +4.08 V)
+5V:          +2.96 V  (min =  +0.00 V, max =  +4.08 V)
in4:          +3.07 V  (min =  +0.00 V, max =  +4.08 V)
in5:          +3.25 V  (min =  +0.00 V, max =  +4.08 V)
in6:          +4.08 V  (min =  +0.00 V, max =  +4.08 V)  ALARM
5VSB:         +2.98 V  (min =  +0.00 V, max =  +4.08 V)
Vbat:         +3.28 V  
fan1:        6750 RPM  (min =    0 RPM)
fan2:           0 RPM  (min =    0 RPM)
fan3:           0 RPM  (min =    0 RPM)
fan4:           0 RPM  (min =    0 RPM)
temp1:        +31.0°C  (low  = +127.0°C, high = +127.0°C)  sensor =
thermistor
temp2:        +67.0°C  (low  = +127.0°C, high = +127.0°C)  sensor =
thermal diode
temp3:        +70.0°C  (low  = +127.0°C, high = +127.0°C)  sensor =
thermal diode
cpu0_vid:    +0.525 V
intrusion0:  ALARM


             total       used       free     shared    buffers
cached
Mem:       8167400    3195300    4972100          0      62024
1379256
-/+ buffers/cache:    1754020    6413380
Swap:       511996          0     511996



Once again, a big thanks for everyone's help.

Regards,
Carlos

Re: [gentoo-user] Re: System freezes during compiles

[Helmut Jarausch]

On 03/20/2013 10:50:51 PM, Carlos Hendson wrote:
> I started a long test on the hard drive:
> 
>  smartctl -t long /dev/sda
> 
> smartctl -a /dev/sda also appears to indicate various errors (the  
> output
> is attached).  I'll trying to track down some documentation as to what
> they're actually reporting.
> 
> Looking at the difference between the output of smartctl for before  
> and
> during the test, there has been an increase in errors detected for ID
> #195
> 
> Before test:
> 195 Hardware_ECC_Recovered  0x001a   001   001   000    Old_age    
> Always
> -       241822
> 
> During test:
> 195 Hardware_ECC_Recovered  0x001a   001   001   000    Old_age    
> Always
> -       243582
> 
> Could this be the cause of the stalls during compiles?  If it is the
> cause, is it possible for the kernel to detect such failures and  
> report
> them?
> 

These numbers are misleading. You might have a look at
http://www.users.on.net/~fzabkar/HDD/Seagate_SER_RRER_HEC.html

Helmut.

Re: [gentoo-user] Re: System freezes during compiles

[J. Roeleveld]

On Thu, March 21, 2013 21:03, Grant Edwards wrote:
> On 2013-03-20, Carlos Hendson <skyclan@gmx.net> wrote:
>
>> Could this be the cause of the stalls during compiles?  If it is the
>> cause, is it possible for the kernel to detect such failures and report
>> them?
>
> I think that as long as the errors are "recovered" they shouldn't
> cause problems.  The $64 question is whether those errors are
> indicating that you're headed for unrecoverable errors (which will
> cause problems).  If they seem to be increasing in frequency, I'd
> probably be a little worried.

Consumer harddrives, which I am assuming you are using, will stall
indefinitely untill they have fixed the error they are encountering.

RAID-drives (With TLER = Time Limited Error Recovery) will return sooner
with an error-code which can then be handled by the controller and/or
driver.

In other words, yes, these can explain the stalls during compiles.

--
Joost

Re: [gentoo-user] Re: System freezes during compiles

[Volker Armin Hemmann]

Am 26.03.2013 15:13, schrieb J. Roeleveld:
> On Thu, March 21, 2013 21:03, Grant Edwards wrote:
>> On 2013-03-20, Carlos Hendson <skyclan@gmx.net> wrote:
>>
>>> Could this be the cause of the stalls during compiles?  If it is the
>>> cause, is it possible for the kernel to detect such failures and report
>>> them?
>> I think that as long as the errors are "recovered" they shouldn't
>> cause problems.  The $64 question is whether those errors are
>> indicating that you're headed for unrecoverable errors (which will
>> cause problems).  If they seem to be increasing in frequency, I'd
>> probably be a little worried.
> Consumer harddrives, which I am assuming you are using, will stall
> indefinitely untill they have fixed the error they are encountering.

no, they don't.
PATa drives take 30secs.

SATA not even a tenth.

That is why dmesg can fill up with hdd errors and you don't realize it
until a reboot and the drive not spinning up anymore.

>
> RAID-drives (With TLER = Time Limited Error Recovery) will return sooner
> with an error-code which can then be handled by the controller and/or
> driver.
>
> In other words, yes, these can explain the stalls during compiles.

they would - if it wasn't already established that it was overheating.