From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (unknown [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id EF5A11381FA for ; Wed, 28 May 2014 19:56:52 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 0DF71E093A; Wed, 28 May 2014 19:56:51 +0000 (UTC) Received: from relay.sgi.com (relay2.sgi.com [192.48.180.65]) by pigeon.gentoo.org (Postfix) with ESMTP id 4D548E0930 for ; Wed, 28 May 2014 19:56:50 +0000 (UTC) Received: from conejo.engr.sgi.com (conejo.engr.sgi.com [150.166.130.27]) by relay2.corp.sgi.com (Postfix) with ESMTP id 9ACA5304043 for ; Wed, 28 May 2014 12:56:49 -0700 (PDT) Received: from sgi.com (conejo.engr.sgi.com [150.166.130.27]) by conejo.engr.sgi.com (Postfix) with ESMTP id 498BF6784DE2 for ; Wed, 28 May 2014 12:56:49 -0700 (PDT) Date: Wed, 28 May 2014 12:56:49 -0700 From: Bob Sanders To: gentoo-amd64@lists.gentoo.org Subject: Re: [gentoo-amd64] Soliciting new RAID ideas Message-ID: <20140528195649.GA3108@sgi.com> Mail-Followup-To: gentoo-amd64@lists.gentoo.org References: <20140527223938.GA3701@sgi.com> <20140528015114.3634f6b4@marcec> <20140528152658.GA13493@sgi.com> <20140528212018.04387c61@marcec> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-amd64@lists.gentoo.org Reply-to: gentoo-amd64@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20140528212018.04387c61@marcec> Organization: SGI, Fremont, California, U.S.A. User-Agent: Mutt/1.5.22 (2013-10-16) X-Archives-Salt: e8765e01-c944-4cd5-a699-d133d8c69cdd X-Archives-Hash: c5f978f40ea638c7b6c829de9c68ed1b Marc Joliet, mused, then expounded: > Am Wed, 28 May 2014 08:26:58 -0700 > schrieb Bob Sanders : > > > > > Marc Joliet, mused, then expounded: > > > Am Tue, 27 May 2014 15:39:38 -0700 > > > schrieb Bob Sanders : > > > > > > While I am far from a filesystem/storage expert (I see myself as a mere user), > > > the cited threads lead me to believe that this is most likely an > > > overhyped/misunderstood class of errors (e.g., posts [1] and [2]), so I would > > > suggest reading them in their entirety. > > > > > > [0] http://comments.gmane.org/gmane.comp.file-systems.btrfs/31832 > > > [1] http://permalink.gmane.org/gmane.comp.file-systems.btrfs/31871 > > > [2] http://permalink.gmane.org/gmane.comp.file-systems.btrfs/31877 > > > [3] http://comments.gmane.org/gmane.comp.file-systems.btrfs/31821 > > > > > > > FWIW - here's the FreeNAS ZFS ECC discussion on what happens with a bad > > memory bit and no ECC memory: > > Just to beat this dead horse some more, an analysis of a academic study on drive failures - http://storagemojo.com/2007/02/20/everything-you-know-about-disks-is-wrong/ And it links to the actual study here - https://www.usenix.org/legacy/events/fast07/tech/schroeder.html Which shows that memory has a fairly high failure rate as well, though the focus is on hard drives. > > http://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ > > Thanks for explicitly linking that. I didn't read it the first time around, > but just read through most of it, then reread the threads [0] and [3] above and > *think* that I understand the problem (and how it doesn't apply to BTRFS) > better now. > > IIUC, the claim is: data is written to disk, but it must go through the RAM > first, obviously, where it is corrupted (due to a permanent bit flip caused, > e.g., by deteriorating hardware). At some later point, when the data is read > back from disk, it might happen to load around the damaged location in RAM, > where it is further corrupted. At this point the checksum fails, and ZFS > corrects the data in RAM (using parity information!), where it is immediately > corrupted again (because apparently it is corrected at the same physical > location in RAM? perhaps this is specific to correction via parity?). This > *additionally* corrupted data is then written back to disk (without any further > checks). > > So the point is that, apparently, without ECC RAM, you could get a (long-term) > cascade of errors, especially during a scrub. The likelihood of such permanent > RAM corruption happening in the first place is another question entirely. > > The various posts in [0] then basically say that regardless of whether this > really is true of ZFS, it certainly doesn't apply to BTRFS, for various > reasons. I suppose this quote from [1] (see above) says it most clearly: > > > In hxxp://forums.freenas.org/threads/ecc-vs-non-ecc-ram-and-zfs.15449, they talk about > > reconstructing corrupted data from parity information: > > > > > Ok, no problem. ZFS will check against its parity. Oops, the parity failed since we have a new corrupted > > bit. Remember, the checksum data was calculated after the corruption from the first memory error > > occurred. So now the parity data is used to "repair" the bad data. So the data is "fixed" in RAM. > > > > i.e. that there is parity information stored with every piece of data, and ZFS will "correct" errors > > automatically from the parity information. I start to suspect that there is confusion here between > > checksumming for data integrity and parity information. If this is really how ZFS works, then if memory > > corruption interferes with this process, then I can see how a scrub could be devastating. I don't know if > > ZFS really works like this. It sounds very odd to do this without an additional checksum check. This sounds > > very different to what you say below that btrfs does, which is only to check against redundantly-stored > > copies, which I agree sounds much safer. > > The rest is also relevant, but I think the point that the data is corrected via > parity information, as opposed to using a known-good redundant copy of the data > (which I originally missed, and thus got confused), is the key point in > understanding the (supposed) difference in behaviour between ZFS and BTRFS. > > All this assumes, of course, that the FreeNAS forum post that ignited this > discussion is correct in the first place. > > > Thanks Mark! Interesting discussion on btrfs. > > > > Bob > > You're welcome! I agree, it's an interesting discussion. And regarding the > misspelling of my name: no problem :-) . > > -- > Marc Joliet > -- > "People who think they know everything really annoy those of us who know we > don't" - Bjarne Stroustrup -- -