[gentoo-dev] Debugging a Linux kernel deadlock when doing swap on ZFS

[gentoo-dev] Debugging a Linux kernel deadlock when doing swap on ZFS

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]

I am running Gentoo on ZFS using the kernel modules from sys-kernel/spl
and sys-fs/zfs. If I put swap on ZFS, the kernel appears to deadlock
when it tries to use it. I am having trouble getting a backtrace.

Does anyone have any suggestions on how I could debug this?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] Debugging a Linux kernel deadlock when doing swap on ZFS

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 657 bytes --]

On Thursday 12 April 2012 19:30:08 Richard Yao wrote:
> I am running Gentoo on ZFS using the kernel modules from sys-kernel/spl
> and sys-fs/zfs. If I put swap on ZFS, the kernel appears to deadlock
> when it tries to use it. I am having trouble getting a backtrace.
> 
> Does anyone have any suggestions on how I could debug this?

networked swap has always been hairy.  your best bet would be to ask the zfs 
guys first.  otherwise, g'luck -- the mm in the kernel is bad enough without 
adding in the network and filesystem stacks.

otherwise, there's quick options like enabling lockdep checking in the kernel.  
or the NMI watchdog.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Debugging a Linux kernel deadlock when doing swap on ZFS

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

On Thu, 12 Apr 2012 19:30:08 -0400
Richard Yao <ryao@cs.stonybrook.edu> wrote:

> I am running Gentoo on ZFS using the kernel modules from sys-kernel/spl
> and sys-fs/zfs. If I put swap on ZFS, the kernel appears to deadlock
> when it tries to use it. I am having trouble getting a backtrace.
> 
> Does anyone have any suggestions on how I could debug this?
> 
If it's that easy to reproduce, I'd suggest to try it in usermode
linux (ARCH=um make). It's a linux kernel as unprivileged userspace
process. Easy to attach with gdb, add printk and etc.
    Does not support SMP.

- swap on FS has some optimizations which are not very well suited
  to copy-on-write files. kernel picks file block map from filesystem
  and tries to use it directly (code might be shared with FIBMAP
  implementation). It's a tricky part to handle.
- triple-check all ZFS memory allocations go at least via GFP_NOFS
  allocations.

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Debugging a Linux kernel deadlock when doing swap on ZFS

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 345 bytes --]

Thanks everyone for your extremely useful tips. I seem to have it
working now. The problem is that ZFS does memory allocations when asked
to write things.

A makeshift solution is to do `echo 524288 >
/proc/sys/vm/min_free_kbytes`. A more permanent fix will take more time
to produce, but at least I have a handle on the situation now.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]