Re: [gentoo-user] OT: btrfs raid 5/6

[Rich Freeman <rich0@gentoo.org>]

On Thu, Dec 7, 2017 at 9:53 AM, Frank Steinmetzger <Warp_7@gmx.de> wrote:
>
> I see. I'm always looking for ways to optimise expenses and cut down on
> environmental footprint by keeping stuff around until it really breaks. In
> order to increase capacity, I would have to replace all four drives, whereas
> with a mirror, two would be enough.
>

That is a good point.  Though I would note that you can always replace
the raidz2 drives one at a time - you just get zero benefit until
they're all replaced.  So, if your space use grows at a rate lower
than the typical hard drive turnover rate that is an option.

>
> When I configured my kernel the other day, I discovered network block
> devices as an option. My PC has a hotswap bay[0]. Problem solved. :) Then I
> can do zpool replace with the drive-to-be-replaced still in the pool, which
> improves resilver read distribution and thus lessens the probability of a
> failure cascade.
>

If you want to get into the network storage space I'd keep an eye on
cephfs.  I don't think it is quite to the point where it is a
zfs/btrfs replacement option, but it could get there.  I don't think
the checksums are quite end-to-end, but they're getting better.
Overall stability for cephfs itself (as opposed to ceph object
storage) is not as good from what I hear.  The biggest issue with it
though is RAM use on the storage nodes.  They want 1GB/TB RAM, which
rules out a lot of the cheap ARM-based solutions.  Maybe you can get
by with less, but finding ARM systems with even 4GB of RAM is tough,
and even that means only one hard drive per node, which means a lot of
$40+ nodes to go on top of the cost of the drives themselves.

Right now cephfs mainly seems to appeal to the scalability use case.
If you have 10k servers accessing 150TB of storage and you want that
all in one managed well-performing pool that is something cephfs could
probably deliver that almost any other solution can't (and the ones
that can cost WAY more than just one box running zfs on a couple of
RAIDs).


-- 
Rich