[gentoo-amd64] btrfs Was: Soliciting new RAID ideas

[Duncan <1i5t5.duncan@cox.net>]

thegeezer posted on Wed, 28 May 2014 00:38:03 +0100 as excerpted:

> depending on your budget a pair of large sata drives + mdadm will be
> ideal, if you had lvm already you could simply 'move' then 'enlarge'
> your existing stuff (tm) : i'd like to know how btrfs would do the same
> for anyone who can let me know.
> you have raid6 because you probably know that raid5 is just waiting for
> trouble, so i'd probably start looking at btrfs for your finanical data
> to be checksummed.

Given that I'm a regular on the btrfs list as well as running it myself, 
I'm likely to know more about it than most.  Here's a whirlwind rundown 
with a strong emphasis on practical points a lot of people miss (IOW, I'm 
skipping a lot of the commonly covered and obvious stuff).  Point 6 below 
directly answers your move/enlarge question.  Meanwhile, points 1, 7 and 
8 are critically important, as we see a lot of people on the btrfs list 
getting them wrong.

1) Since there's raid5/6 discussion on the thread... Don't use btrfs 
raid56 modes at this time, except purely for playing around with trashable 
or fully backed up data.  The implementation as introduced isn't code-
complete, and while the operational runtime side works, recovery from 
dropped devices, not so much.  Thus, in terms of data safety you're 
effectively running a slow raid0 with lots of extra overhead that can be 
considered trash if a device drops, with the sole benefit being that when 
the raid56 mode recovery implementation code gets merged (and is tested 
for a kernel cycle or two to work out the initial bugs), you'll then get 
what amounts to a "free" upgrade to the raid5 or raid6 mode you had 
originally configured, since it was doing the operational parity 
calculation and writes to track it all along, it just couldn't yet be 
used for actual recovery as the code simply wasn't there to do so.

2) Btrfs raid0, raid1 and raid10 modes, along with single mode (on a 
single or multiple-devices) and dup mode (on a single device, metadata is 
by default duplicated -- two copies, except on ssd where the default is 
only a single copy since some ssds dedup anyway) are reasonably mature 
and stable, to the same point as btrfs in general, anyway, which is to 
say it's "mostly stable, keep your backups fresh but you're not /too/ 
likely to have to use them."  There are still enough bugs being fixed in 
each kernel release, however, that running latest stable series is 
/strongly/ recommended, as your data is at risk to known-fixed bugs (even 
if at this point they only tend to hit the corner-cases) if you're not 
doing so.

3) It's worth noting that btrfs treats data and metadata separately -- 
when you do a mkfs.btrfs, you can configure redundancy modes separately 
for each, the single-device default being (as above) dup metadata (except 
for ssd), single data, the multi-device default being raid1 metadata, 
single data

4) FWIW, most of my btrfs formatted partitions are dual-device raid1 mode 
for both data and metadata, on ssd.  (Second backup is reiserfs on 
spinning-rust, just in case some Armageddon bug eats all the btrfs at the 
same time, working copy and first backup, tho btrfs is stable enough now 
that's extremely unlikely, but I didn't consider it so back when I set 
things up nearly a year ago now.)

The reason for my raid1 mode choice isn't that of ordinary raid1, it's 
specifically due to btrfs' checksumming and data integrity features -- if 
one copy fails its checksum, btrfs will, IF IT HAS ANOTHER COPY TO TRY, 
check the second copy and if it's good, will use it and rewrite the bad 
copy.  Btrfs scrub allows checking the entire filesystem for checksum 
errors and restoring any errors it finds from good copies where possible.

Obviously, the default single data mode (or raid0) won't have a second 
copy to check and rewrite from, while raid1 (and raid10) modes will (as 
will dup-mode metadata on a single device, but with one exception, dup 
mode isn't allowed for data, only metadata, the exception being the mixed-
blockgroup mode that mixes data and metadata together, that's the default 
on filesystems under 1 GiB but isn't recommended on large filesystems for 
performance reasons).

So I wanted a second copy of both data and metadata to take advantage of 
btrfs' data integrity and scrub features, and with btrfs raid1 mode, I 
get both that and the traditional raid1 device-loss protection as well. 
=:^)

5) It's worth noting that as of now, btrfs raid1 mode is only two-way-
mirrored, no matter how many devices are configured into the filesystem.  
N-way-mirrored is the next feature on the roadmap after the raid56 work 
is completed, but given how nearly every btrfs feature has taken much 
longer to complete than originally planned, I'm not expecting it until 
sometime next year, now.

Which is unfortunate, as my risk vs. cost sweet spot would be 3-way-
mirroring, covering in case *TWO* copies of a block failed checksum.  Oh, 
well, it's coming, even if it seems at this point like the proverbial 
carrot dangling off a stick held in front of the donkey.

6) Btrfs handles moving then enlarging (parallel to LVM) using btrfs
add/delete, to add or delete a device to/from a filesystem (moving the 
content from a to-be-deleted device in the process), plus btrfs balance, 
to restripe/convert/rebalance between devices as well as to free 
allocated but empty data and metadata chunks back to unallocated.  
There's also btrfs resize, but that's more like the conventional 
filesystem resize command, resizing the part of the filesystem on an 
individual device (partitioned/virtual or whole physical device).

So to add a device, you'd btrfs device add, then btrfs balance, with an 
optional conversion to a different redundancy mode if desired, to 
rebalance the existing data and metadata onto that device.  (Without the 
rebalance it would be used for new chunks, but existing data and metadata 
chunks would stay where they were.  I'll omit the "chunk definition" 
discussion in the interest of brevity.)

To delete a device, you'd btrfs device delete, which would move all the 
data on that device onto other existing devices in the filesystem, after 
which it could be removed.

7) Given the thread, I'd be remiss to omit this one.  VM images and other 
large "internal-rewrite-pattern" files (large database files, etc) need 
special treatment on btrfs, at least currently.  As such, btrfs may not 
be the greatest solution for Mark (tho it would work fine with special 
procedures), given the several VMs he runs.  This one unfortunately hits 
a lot of people. =:^(  But here's a heads-up, so it doesn't have to hit 
anyone reading this! =:^)

As a property of the technology, any copy-on-write-based filesystem is 
going to find files where various bits of existing data within the file 
are repeatedly rewritten (as opposed to new data simply being appended, 
think a log file or live-stored audio/video stream) extremely challenging 
to deal with.  The problem is that unlike ordinary filesystems that 
rewrite the data in place such that a file continues to occupy the same 
extents as it did before, copy-on-write filesystems will write a changed 
block to a different location.  While COW does mean atomic updates and 
thus more reliability since either the new data or the old data should 
exist, never an unpredictable mixture of the two, as a result of the 
above rewrite pattern, this type of internally-rewritten file gets 
**HEAVILY** fragmented over time.

We've had filefrag reports of several gig files with over 100K extents!  
Obviously, this isn't going to be the most efficient file in the world to 
access!

For smaller files, up to a couple hundred MiB or perhaps a bit more, 
btrfs has the autodefrag mount option, which can help a lot.  With this 
option enabled, whenever a block of a file is changed and rewritten, thus 
written elsewhere, btrfs queues up a rewrite of the entire file to happen 
in the background.  The rewrite will be done sequentially, thus defragging 
the file.  This works quite well for firefox's sqlite database files, for 
instance, as they're internal-rewrite-pattern, but they're small enough 
that autodefrag handles them reasonably nicely.

But this solution doesn't scale so well as the file size increases toward 
and past a GiB, particularly for files with a continuous stream of 
internal rewrites such as can happen with an operating VM writing to its 
virtual storage device.  At some point, the stream of writes comes in 
faster than the file can be rewritten, and things start to back up!

To deal with this case, there's the NOCOW file attribute, set with chattr 
+C.  However, to be effective, this attribute must be set when the file 
is empty, before it has existing content.  The easiest way to do that is 
to set the attribute on the directory which will contain the files.  
While it doesn't affect the directory itself any, newly created files 
within that directory inherit the NOCOW attribute before they have data, 
thus allowing it to work without having to worry about it that much.  For 
existing files, create a new directory, set its NOCOW attribute, and COPY 
(don't move, and don't use cp --reflink) the existing files into it.

Once you have your large internal-rewrite-pattern files set NOCOW, btrfs 
will rewrite them in-place as an ordinary filesystem would, thus avoiding 
the problem.

Except for one thing.  I haven't mentioned btrfs snapshots yet as that 
feature, but for this caveat, is covered well enough elsewhere.  But 
here's the problem.  A snapshot locks the existing file data in place.  
As a result, the first write to a block within a file after a snapshot 
MUST be COW, even if the file is otherwise set NOCOW.  

If only the occasional one-off snapshot is done it's not /too/ bad, as 
all the internal file writes between snapshots are NOCOW, it's only the 
first one to each file block after a snapshot that must be COW.  But many 
people and distros are script-automating their snapshots in ordered to 
have rollback capacities, and on btrfs, snapshots are (ordinarily) light 
enough that people are sometimes configuring a snapshot a minute!  If 
only a minute's changes can be written to a the existing location, then 
there's a snapshot and changes must be written to a new location, then 
another snapshot and yet another location...  Basically the NOCOW we set 
on that file isn't doing us any good!

8) So making this a separate point as it's important and a lot of people 
get it wrong.  NOCOW and snapshots don't mix!

There is, however, a (partial) workaround.  Because snapshots stop at 
btrfs subvolume boundaries, if you put your large VM images and similar 
large internal-rewrite-pattern files (databases, etc) in subvolumes, 
making that directory I suggested above a full subvolume not just a NOCOW 
directory, snapshots of the parent subvolume will not include the VM 
images subvolume, thus leaving the VM images alone.  This solves the 
snapshot-broken-NOCOW and thus the fragmentation issue, but it DOES mean 
that those VM images must be backed up using more conventional methods 
since snapshotting won't work for them.

9) Some other still partially broken bits of btrfs include:

9a) Quotas:  Just don't use them on btrfs at this point.  Performance 
doesn't scale (altho there's a rewrite in progress), and they are buggy.  
Additionally, the scaling interaction with snapshots is geometrically 
negative, sometimes requiring 64 GiB of RAM or more and coming to a near 
standstill at that, for users with enough quota-groups and enough 
snapshots.  If you need quotas, use a more traditional filesystem with 
stable quota support.  Hopefully by this time next year...

9b) Snapshot-aware-defrag:  This was enabled at one point but simply 
didn't scale, when it turned out people were doing things like per-minute 
snapshots and thus had thousands and thousands of snapshots.  So this has 
been disabled for the time being.  Btrfs defrag will defrag the working 
copy it is run on, but currently doesn't account for snapshots, so data 
that was fragmented at snapshot time gets duplicated as it is 
defragmented.  However, they plan to re-enable the feature ones they have 
rewritten various bits to scale far better than they do at present.

9c) Send and receive.  Btrfs send and receive are a very nice feature 
that can make backups far faster, with far less data transferred.  
They're great when they work.  Unfortunately, there are still various 
corner-cases where they don't.  (As an example, a recent fix was for the 
case where subdir B was nested inside subdir A for the first, full send/
receive, but later, the relationship was reversed, with subdir B made the 
parent of subdir A.  Until the recent fix, send/receive couldn't handle 
that sort of corner-case.)  You can go ahead and use it if it's working 
for you, as if it finishes without error, the copy should be 100% 
reliable.  However, have an alternate plan for backups if you suddenly 
hit one of those corner-cases and send/receive quits working.

Of course it's worth mentioning that b and c deal with features that most 
filesystems don't have at all, so with the exception of quotas, it's not 
like something's broken on btrfs that works on other filesystems.  
Instead, these features are (nearly) unique to btrfs, so even if they 
come with certain limitations, that's still better than not having the 
option of using the feature at all, because it simply doesn't exist on 
the other filesystem!

10) Btrfs in general is headed toward stable now, and a lot of people, 
including me, have used it for a significant amount of time without 
problems, but it's still new enough that you're strongly urged to make 
and test your backups, because by not doing so, you're stating by your 
actions if not your words, that you simply don't care if some as yet 
undiscovered and unfixed bug in the filesystem eats your data.

For similar reasons altho already mentioned above, run the latest stable 
kernel from the latest stable kernel series, at the oldest, and consider 
running rc kernels from at least rc2 or so (by which time any real data 
eating bugs, in btrfs or elsewhere, should be found and fixed, or at 
least published).  Because anything older and you are literally risking 
your data to known and fixed bugs.

As is said, take reasonable care and you're much less likely to be the 
statistic!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman