Re: [gentoo-dev] reiserfs

[Jean-Michel Smith <jsmith@kcco.com>]

On Tuesday 14 May 2002 05:44 pm, Bill Kenworthy wrote:

> The question came up on a local lug as well, with reiserfs and ext3
> seeming to be the top choices, and from memory xfs seemed to be bagged
> (cant remember why - something to do with the linux implementation?).

The patch touches a great many files.  Those who want to apply other patches 
typically find the xfs patch means doing some work by hand, which puts most 
people off.  XFS is rock solid in my experience, even in the face of power 
outages, untimely shutdowns, and the like, but I am conservative and only run 
it patched against stock kernels (e.g. xfs-sources, which is 2.4.18+xfs 
patches only).  Those wanting to play with other experimental patches 
generally avoid XFS because of this (as do I on the machines I do that sort 
of thing on) for this reason.

> Reiserfs did get some bad press in the early days, and I think that may
> be a hangover that effects peoples thinking.  I think this is a case of
> YMMV, and as far as I am concerned, gentoo is the odd one out by not
> reccomending reiserfs, and because there seems to be little
> documentation to back it up its point of view, but a fair bit of
> experiance saying reiserfs is reasonably stable. 

YMMV is reason enough to not recommend a filesystem, when your milage is 
varying with respect to spontaneous filesystem corruption!  Gentoo may be the 
odd one out on this, but in my opinion that says a great deal positively 
about the technical expertise and caution of the Gentoo developers, 
particularly in light of my own experiences.

These will be my final comments on the subject.  Looking at my notes and log 
entries, I had a total of 7 machines (out of 9 total deployed) go south with 
Reiserfs on them (unrecoverable filesystem corruption, including lost 
directories, strangely null files, and in one case oddly corrupt 
files/filenames that were undeletable).  None were due to kernel oopses, 
untimely shutdowns, or any other cause that would lead one to expect 
filesystem troubles, they were all apparently spontaneous, and all happened 
within 6 months of being deployed.  The last two machines were migrated off 
of Reiser (onto ext2) before they could screw up, having been in use only 
about three months.

The corruptions happened between April and August of last year (2001).  5 
machines were running Mandrake, one Red Hat, and one Debian. (~3 months 
testing, ~8 months deployed.  I was incorrect in an earlier post when I said 
none lasted more than 6 months ... one machine survived 9 months before 
problems arose, and another didn't suffer filesystem corruption until 7 
months after deployment.  All of these machines are on 24/7)

My friend had his Suse Reiserfs go south (entire directory tree spontaneously 
vanished, but filesystem usage remained the same and even continued to grow) 
six weeks ago (April 2002), so this is by no means an early development 
glitch that is now ancient history we can comfortably dismiss and forget 
about.  (I do not know how long he had had the machine deployed for, but I 
can find out if anyone is really interested).

XFS is annoying because the patch is big, and sometimes one must wait a week 
or two after a kernel is released before a patch for xfs exists.  In the case 
of gentoo, where multiple cool patches are being applied to an experimental, 
pre-release of 2.4.19, we had to wait a week or two before some kind, 
enterprising soul managed to work the patch into the OS (testing on -r5 is 
looking very good, fwiw).  That having been said, unless one is desperate for 
a particular fix one is generally wise to wait a week or two after a kernel 
release before deploying it in a production environment anyway, so this 
(admittedly minor) irritation is mitigated for the most part.  

I've beaten on XFS under just about every condition imaginable (minus LVS, 
which I do not use) and have yet to be able to make it corrupt the 
filesystem.  I've even deliberately caused kernel oopses by trying to compile 
glibc on a kernel with high-mem, or a machine with 1 GB RAM, and been unable 
to cause damage to the filesystem.  It appears to be very solid and does not 
corrupt spontaneously.  (about 3.5 years fairly rigorous testing and very 
rigorous usage, including 2 enterprise NFS servers on large RAID devices and 
several developer workstations. Of all the filesystems I've tested, this one 
has been tested the most thoroughly, except of course for ext2 which I've 
been using a great deal longer)

JFS I've done less testing with.  It appears to be pretty good, but others 
have reported LVS corruption which may have been caused by JFS.  I haven't 
beaten on it nearly as hard as I have XFS, so I cannot say with certainty 
that it is reliable, but thus far I've yet to have it screw up.  Not exactly 
a ringing endorsement, but a cautious "it looks ok so far." (~3 months casual 
testing)

ditto ext3.  It needs more testing.  It seems to do alright thus far, but I 
tend to treat it as I would an ext2 filesystem. (~5 months casual testing).

ext2 is very solid, provided it is treated correctly (no improper shutdowns or 
power-offs), or buffering is turned off.  It does not corrupt spontaneously, 
ever.  BUT, and this is a big BUT, it can and does become corrupted if it is 
not shutdown properly (and this can happen due to system hangs, e.g. X with 
Nvidia drivers on some configurations, power outage, impatient user hitting 
the reset switch, etc.).  Most of the time it will recover through fsck, but 
not always, and I echo others who have lost ext2 filesystems that have been 
unrecoverably corrupted in this way.  It is why I prefer journalled 
filesystems and have gone to deploying XFS where possible and practical 
(often, but not always, the case due to the patch's size and complexity), and 
why I am keeping an eye on JFS and others.

It is important to note that filesystem corruption due to untimely shutdowns, 
which both ext2 and reiser have suffered from, are a completely different 
animal from the apparent spontaneous loss of data that is my major complaint 
with Reiser, and why I am so vocal in defending Gentoo's word of caution 
regarding it.

Kernel oopses:  all bets are off for any filesystem (though I've yet to be 
able to get XFS to corrupt from this, it is theoretically possible AFAICT 
since something might be going on within the kernel's vfs layer when the hang 
happens.  This is the only situation in which I find filesystem corruption in 
a journaled filesystem to be at all forgivable.

The only filesystem I have ever experienced that has corrupted itself during 
normal operations, with no unexpected reboots, kernel oopses, or other 
mitigating circumstances to explain the corruption, is Reiserfs, and these 
experiences are all within the last 14 months.

The only filesystem I've been unable to corrupt has been XFS.  (JFS and ext3 
do not count, I haven' t beaten on them the way I have ext2, XFS, and 
Reiser).

So while people should continue to experiement with Reiser (after all, that is 
how these sorts of bugs will be found and fixed), a word of caution is IMHO 
certainly in order, regardless of whether or not that makes Gentoo "the odd 
man out" or not.

Jean.