[gentoo-user] hard lockups w/nfs, md raid5

[gentoo-user] hard lockups w/nfs, md raid5

[matthew.garman]

I've got a homebuilt server running gentoo.  I just built a software
(linux md) RAID5 array using four SATA drives (connected via a
Promise PCI SATA card).

In addition to the RAID array, there's a SCSI drive from which the
OS boots and runs; two PATA drives merged together under lvm2; and
one PATA drive mounted "normally" (i.e. no lvm2/md/whatever).

Last night, I was copying about 26GB from the standalone PATA drive
to the RAID array.  At the same time, I was ripping a DVD to the
RAID drive via NFS (i.e. the rip occurred on a different computer,
but the storage was the RAID array exported via nfs).

Twice in a row, under these conditions, the server locked up.  They
were hard lockups---couldn't ping the machine and the keyboard was
totally unresponsive.

I checked the logs, and couldn't find ANYTHING to suggest the cause
of the lockup (nothing at all out of the ordinary).

After being discouraged by the two hard lockups, I performed one
task at a time: first ripped the DVD, then copied files.  No
lockups.

My questions are: (1) does anyone know what might have caused the
lockups?  I have a feeling I could duplicate this again without too
much effort.  Also, (2) is there any mechanism I can use to actually
track down the root cause?  Right now, there are too many variables:
flaky hardware (although this machine has *never* locked up prior to
adding the SATA card+drives), nfs daemon, libata code, sata
controller driver, linux md code...

Thanks in advance,
Matt

-- 
Matt Garman
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] hard lockups w/nfs, md raid5

[Richard Fish]

On 2/27/06, matthew.garman@gmail.com <matthew.garman@gmail.com> wrote:
> My questions are: (1) does anyone know what might have caused the
> lockups?

My experience has been that lockups occuring during heavy IO can be
caused by aggressive memory timings.  If your BIOS supports it, try
increasing the memory timings.

Also try the memory test script from here, but it does pretty much
what you were describing...massive IO and memory bandwidth test:

http://people.redhat.com/dledford/memtest.html

-Richard

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] hard lockups w/nfs, md raid5

[Richard Fish]

On 2/27/06, matthew.garman@gmail.com <matthew.garman@gmail.com> wrote:
> My questions are: (1) does anyone know what might have caused the
> lockups?

My experience has been that lockups occuring during heavy IO can be
caused by aggressive memory timings.  If your BIOS supports it, try
increasing the memory timings.

Also try the memory test script from here, but it does pretty much
what you were describing...massive IO and memory bandwidth test:

http://people.redhat.com/dledford/memtest.html

-Richard

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] hard lockups w/nfs, md raid5

[matthew.garman]

On Mon, Feb 27, 2006 at 12:51:38PM -0700, Richard Fish wrote:
> Also try the memory test script from here, but it does pretty much
> what you were describing...massive IO and memory bandwidth test:
> 
> http://people.redhat.com/dledford/memtest.html

I don't believe it is a memory, IO or hardware problem (at this
point).  I ran the above test simultaneously with "stress --io 10
--hdd 10" for several hours.  No problems.

I believe the problem is NFS locking up the machine.  If I run the
same "stress" program on a client machine, where the working
directory is NFS mounted to the server, then I can cause a lockup
almost instantaneously.

Has anyone else seen hard lockups like this with NFS?

Any fixes?

Thanks,
Matt

-- 
Matt Garman
email at: http://raw-sewage.net/index.php?file=email
-- 
gentoo-user@gentoo.org mailing list