Re: [gentoo-amd64] Re: Is my RAID performance bad possibly due to starting sector value?

[Rich Freeman <rich0@gentoo.org>]

On Fri, Jun 21, 2013 at 3:31 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> So with 4k block sizes on a 5-device raid6, you'd have 20k stripes, 12k
> in data across three devices, and 8k of parity across the other two
> devices.

With mdadm on a 5-device raid6 with 512K chunks you have 1.5M in a
stripe, not 20k.  If you modify one block it needs to read all 1.5M,
or it needs to read at least the old chunk on the single drive to be
modified and both old parity chunks (which on such a small array is 3
disks either way).

> Forth, back to the parity.  Remember, raid5/6 has all that parity that it
> writes out (but basically never reads in normal mode, only when degraded,
> in ordered to reconstruct the data from the missing device(s)), but
> doesn't actually use it for integrity checking.

I wasn't aware of this - I can't believe it isn't even an option
either.  Note to self - start doing weekly scrubs...

> The single down side to raid1 as opposed to raid5/6 is the loss of the
> extra space made available by the data striping, 3*single-device-space in
> the case of 5-way raid6 (or 4-way raid5) vs. 1*single-device-space in the
> case of raid1.  Otherwise, no contest, hands down, raid1 over raid6.

This is a HUGE downside.  The only downside to raid1 over not having
raid at all is that your disk space cost doubles.  raid5/6 is
considerably cheaper in that regard.  In a 5-disk raid5 the cost of
redundancy is only 25% more, vs a 100% additional cost for raid1.  To
accomplish the same space as a 5-disk raid5 you'd need 8 disks.  Sure,
read performance would be vastly superior, but if you're going to
spend $300 more on hard drives and whatever it takes to get so many
SATA ports on your system you could instead add an extra 32GB of RAM
or put your OS on a mirrored SSD.  I suspect that both of those
options on a typical workload are going to make a far bigger
improvement in performance.

Which is better really depends on your workload.  In my case much of
my raid space is used my mythtv, or for storage of stuff I only
occasionally use.  In these use cases the performance of the raid5 is
more than adequate, and I'd rather be able to keep shows around for an
extra 6 months in HD than have the DVR respond a millisecond faster
when I hit play.  If you really have sustained random access of the
bulk of your data than a raid1 would make much more sense.

> So several points on btrfs:
>
> 1) It's still in heavy development.

That is what is keeping me away.  I won't touch it until I can use it
with raid5, and the first common containing that hit the kernel weeks
ago I think (and it has known gaps).  Until it is stable I'm sticking
with my current setup.

> 2) RAID levels work QUITE a bit differently on btrfs.  In particular,
> what btrfs calls raid1 mode (with the same applying to raid10) is simply
> two-way-mirroring, NO MATTER THE NUMBER OF DEVICES.  There's no multi-way
> mirroring yet available

Odd, for some reason I thought it let you specify arbitrary numbers of
copies, but looking around I think you're right.  It does store two
copies of metadata regardless of the number of drives unless you
override this.

However, if one considered raid1 expensive, having multiple layers of
redundancy is REALLY expensive if you aren't using Reed Solomon and
many data disks.

From my standpoint I don't think raid1 is the best use of money in
most cases, either for performance OR for data security.  If you want
performance the money is probably better spent on other components.
If you want data security the money is probably better spent on
offline backups.  However, this very-much depends on how the disks
will be used - there are certainly cases where raid1 is your best
option.

Rich