[gentoo-user] planned btrfs conversion: questions

[gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 4632 bytes --]

Hi all,

I've become increasingly motivated to convert to btrfs.  From what I've seen,
it has become increasingly stable; enough so that it is apparently supposed to
become the default FS on OpenSuse in 13.2.

I am motivated by various reasons:

- The experience of insignificant data loss after a reboot after no
  (visible) problems during normal operations (no similar occurrence since). I
  suspect this would have been discovered sooner by btrfs' data checksumming.

  While the data lost was of a small amount, not to mention insignificant
  (internet pictures) and could be retrieved again easily, I worry that this
  sort of silent corruption might happen to data that I *do* care about. My
  backups would become less than useless if I were to discover such corruption
  too late.

- The outright sexy multiple device support :) .

This migration will occur in conjunction with a migration of / + /usr to a
cheap SSD that I just bought (Crucial M500 120 GB). The overall plan is thus as
follows:

Replace

	/boot on /dev/md1 (EXT3, RAID 1)
	/ (with assorted sub-directories, sans /usr) on /dev/md2 (EXT4, RAID 10)
	the rest on LVM on /dev/md3 (all LVs EXT4, RAID 10)

with

	/ + /boot + /usr + swapfile on the SSD (EXT4)
	the rest (/home, my media partitions) on a btrfs RAID 10

(which replaces an older plan to recreate the RAID 10 + LVM to use the whole
disks with the current 1.2 metadata format)

The goals are, in addition to alleviating my data safety concerns above, to
guarantee that I don't need an initramfs at boot (hence the SSD), and to
greatly simplify my partitioning scheme (I just have too many separate logical
volumes ;-) ). Any added performance is "just" a nice bonus.

The reason why I would choose EXT4 for the SSD is that btrfs still lacks support
for swap files and I worry about creating a swap partition on the SSD. Is that
warranted, or will the wear-levelling of the SSD handle that just fine? Do swap
partitions support SSDs specially? Also, does anyone know whether EXT4 goes
beyond "merely" supporting TRIM? That is, the btrfs wiki advertises the
following:

  "SSD (Flash storage) awareness (TRIM/Discard for reporting free blocks for
  reuse) and optimizations (e.g. avoiding unnecessary seek optimizations,
  sending writes in clusters, even if they are from unrelated files. This
  results in larger write operations and faster write throughput)"

Does EXT4 also implement such optimisations for SSDs?

So I guess I want to know: does anybody have any further suggestions to make?
Is btrfs a good choice for / after all? And should I be using the most recent
kernel versions? (I would go with no, despite the advice from upstream, because
the changes in the last two versions don't seem to be particularly user
visible, at least to me, from reading kernelnewbies.org.)

I also have a more specific question regarding RAID 10: the btrfs wiki says
that you can add devices with different sizes to a multiple device setup, but I
don't think it says to which RAID levels this applies and how. From [0] I would
say it works with RAID 10 (since that's what the example uses), but thought
maybe somebody here knows more details and/or gotchas. From my understanding,
this means that I can iteratively upgrade my RAID 10 to larger drives and have
btrfs use all of the available space (or at least as much as is possible). This
is important to me because I currently have 4 320 GB HDDs + 1 (possibly broken,
must check) spare and wish to be able to upgrade without having to buy four
HDDs at once.

Now to the migration plan: first, partition the SSD and copy all relevant file
systems to it; this will be done from a Live-CD (SystemRescueCD). After I have
configured the mount options appropriately and can boot from that, I should be
able to mount the other file systems (/home and my media partitions) read-only
from the actual system and do the RAID + LVM -> btrfs migration from there.

Note that I will in general follow the advice from [1-3], and if people
recommend btrfs on /, then I will also try to get relevant information from [4].

[0]
https://btrfs.wiki.kernel.org/index.php/SysadminGuide#RAID_and_data_replication
[1] https://wiki.archlinux.org/index.php/Solid_State_Drives
[2] https://wiki.archlinux.org/index.php/Btrfs
[3] http://wiki.gentoo.org/wiki/Btrfs
[4] http://wiki.gentoo.org/wiki/Btrfs_system_root

Greetings and thanks in advance for any help given
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-user] Re: planned btrfs conversion: questions

[James]

Marc Joliet <marcec <at> gmx.de> writes:


> I've become increasingly motivated to convert to btrfs.  From what 
> I've seen, it has become increasingly stable; enough so that it is 
> apparently supposed to > become the default FS on OpenSuse in 13.2.

> I am motivated.
> Greetings and thanks in advance for any help given


Good news Marc,

Last December I was on the BTRFS train. The docs team was totally
in agreement with my (your ?) enthusiasmn for all things BTRFS; including
but not limited to the development of documentation of how to install, from
scratch, a complete Gentoo system, centric around BTRFS. The goal was for a
group to develop documentation on the the gentoo wiki for this adventure.

Unfortunately, as a single parent, entrepreneur, and handyman living in a 
fixer_upper, my free time is rather scant these months of 2014.....
After some seriously needed projects are closed out, I'll have much
more time; hopefully in some weeks rather than months. My goals was to
clearly delineate possible explicit installation syntax for the smoother
integration (mastery?) of grub2, (u)EFI and BTRFS into a new gentoo
installation.


So if you don't mind, I'd like to  parse out good ideas (from your efforts)
 and start this doc on the Gentoo wiki, based in part on your conversion
adventures. I would not even be upset, if some other, younger,
sharper mind wanted to develop a gentoo-wiki doc, based on your adventures.
Also, I'm hoping others are keenly inspired to contribute to your efforts
and  some experimental installation docs on the gentoo wiki. This is the
gentoo community's  chance to influence future gentoo installation docs, in
a non threatening manor. BTRFS is on the fast track in many linux distros,
as well as cloud/super computers architecture and embedded 64bit arches 
as the main file system of choice.


I've been espousing (as have many others) for a long time on the
convergences of embedded linux and full_bloat linix for some time. Here is
one gentoo_distro where the author experiments with many of the issues
related to  convergence. There is even some suggestions on using BTRFS on
gentoo found here. [1] 


sincerely,
James


[1] https://wiki.gentoo.org/wiki/Project:Hardened_uClibc/Lilblue

Re: [gentoo-user] Re: planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 3771 bytes --]

(FYI: your off-list forward of this email arrived at about the same time as
this one (at about 22:20 CET), strangely enough.)

Am Tue, 6 May 2014 18:13:07 +0000 (UTC)
schrieb James <wireless@tampabay.rr.com>:

> Marc Joliet <marcec <at> gmx.de> writes:
> 
> 
> > I've become increasingly motivated to convert to btrfs.  From what 
> > I've seen, it has become increasingly stable; enough so that it is 
> > apparently supposed to > become the default FS on OpenSuse in 13.2.
> 
> > I am motivated.
> > Greetings and thanks in advance for any help given
> 
> 
> Good news Marc,
> 
> Last December I was on the BTRFS train. The docs team was totally
> in agreement with my (your ?) enthusiasmn for all things BTRFS; including
> but not limited to the development of documentation of how to install, from
> scratch, a complete Gentoo system, centric around BTRFS. The goal was for a
> group to develop documentation on the the gentoo wiki for this adventure.

That would be neat :) . I found it really annoying having to piece lots of the
details together myself. I'm kind of in the middle of a Masters thesis here...

Although I admit this sort of activity satisfies my inner Linux geek ;-)
(there's a reason I like Gentoo, after all). And it will hopefully be satiated
for a while now after setting up automated backups, migrating / to an SSD along
with switching to grub2 (which I already use on my borrowed work/uni laptop) in
the process, *and* migrating an mdraid + LVM setup to btrfs -- all within about
one week :) .

> Unfortunately, as a single parent, entrepreneur, and handyman living in a 
> fixer_upper, my free time is rather scant these months of 2014.....
> After some seriously needed projects are closed out, I'll have much
> more time; hopefully in some weeks rather than months. My goals was to
> clearly delineate possible explicit installation syntax for the smoother
> integration (mastery?) of grub2, (u)EFI and BTRFS into a new gentoo
> installation.
> 
> 
> So if you don't mind, I'd like to  parse out good ideas (from your efforts)
>  and start this doc on the Gentoo wiki, based in part on your conversion
> adventures. I would not even be upset, if some other, younger,
> sharper mind wanted to develop a gentoo-wiki doc, based on your adventures.
> Also, I'm hoping others are keenly inspired to contribute to your efforts
> and  some experimental installation docs on the gentoo wiki. This is the
> gentoo community's  chance to influence future gentoo installation docs, in
> a non threatening manor.
    ^^^^^^^^^^^^^^^^^^^^^

This one looks pretty non-threatening to me ;-) :

https://4.bp.blogspot.com/-9usBSoPnUWo/TaMmrclJngI/AAAAAAAAAD4/8Xqx67g6U34/s1600/DSC06646.JPG

> BTRFS is on the fast track in many linux distros,
> as well as cloud/super computers architecture and embedded 64bit arches 
> as the main file system of choice.

That sounds perfectly fine to me (the parsing out good ideas from this thread,
I mean, but btrfs being on the fast track is also good :) ).

> I've been espousing (as have many others) for a long time on the
> convergences of embedded linux and full_bloat linix for some time. Here is
> one gentoo_distro where the author experiments with many of the issues
> related to  convergence. There is even some suggestions on using BTRFS on
> gentoo found here. [1] 
[...]
> [1] https://wiki.gentoo.org/wiki/Project:Hardened_uClibc/Lilblue

That looks interesting, albeit outside my (current) realm of interest. I can
imagine playing with that sort of thing in the future (1+ years), though.

Greetings,
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 6500 bytes --]

OK, I read several articles (the LWN.net series from December/January [5] and a
January article from ars technica [6]) and quite a lot of comments on btrfs
today and can answer some of my questions myself.

For completions sake, I also read a zdnet article series (starting at [7]), but
it wasn't quite as good as the other two IMHO.

Am Tue, 6 May 2014 12:18:32 +0200
schrieb Marc Joliet <marcec@gmx.de>:

[...]
> This migration will occur in conjunction with a migration of / + /usr to a
> cheap SSD that I just bought (Crucial M500 120 GB). The overall plan is thus as
> follows:
> 
> Replace
> 
> 	/boot on /dev/md1 (EXT3, RAID 1)
> 	/ (with assorted sub-directories, sans /usr) on /dev/md2 (EXT4, RAID 10)
> 	the rest on LVM on /dev/md3 (all LVs EXT4, RAID 10)
> 
> with
> 
> 	/ + /boot + /usr + swapfile on the SSD (EXT4)
> 	the rest (/home, my media partitions) on a btrfs RAID 10

This part I think I will stick with.  From what I've read so far, I wouldn't
trust my entire system to btrfs.  Since "the rest" consists of stuff I either
automatically backup (using rsnapshot) or have multiple copies of, I should be
able to recover from a broken btrfs file system fairly easily.

While I am unsure of my choice of RAID level (some comments on LWN.net claim
that the MD RAID 10 is more comparable to btrfs' RAID 1, which I will attempt to
verify myself beforehand). However, due to btrfs' live rebalancing feature, I
worry less about this. By the time I really need more space the RAID 5/6 code
(and maybe N-way mirroring) ought to be stable (or at least finished), or I
can switch to RAID 1 if I need the flexibility.

[...]
> The reason why I would choose EXT4 for the SSD is that btrfs still lacks support
> for swap files and I worry about creating a swap partition on the SSD. Is that
> warranted, or will the wear-levelling of the SSD handle that just fine? Do swap
> partitions support SSDs specially? Also, does anyone know whether EXT4 goes
> beyond "merely" supporting TRIM? That is, the btrfs wiki advertises the
> following:
> 
>   "SSD (Flash storage) awareness (TRIM/Discard for reporting free blocks for
>   reuse) and optimizations (e.g. avoiding unnecessary seek optimizations,
>   sending writes in clusters, even if they are from unrelated files. This
>   results in larger write operations and faster write throughput)"
> 
> Does EXT4 also implement such optimisations for SSDs?

I will also go ahead with this (despite the open questions), although I will
leave swap on the LVM for now. I think tonight (well, today) I "just" want to
get the SSD running. Furthermore, "btrfs convert" should be able to up-convert
it in the future once btrfs is "production ready" (both articles make a
guesstimate of about 1-2 years).

I think I would also prefer running a few days from the SSD before converting
"the rest" to btrfs, which should be fairly simple at that point.

[...]
> Is btrfs a good choice for / after all?

I have decided: not without a full system backup (which I don't really want).

> And should I be using the most recent
> kernel versions? (I would go with no, despite the advice from upstream, because
> the changes in the last two versions don't seem to be particularly user
> visible, at least to me, from reading kernelnewbies.org.)

I changed my mind on this: I checked the change logs from the btrfs wiki and
realised I should really give the notion of having the latest bug fixes more
weight. *Especially* since the focus of the mainline btrfs development is
stability and performance (and finalisation of central features, e.g., RAID5/6
support, but that's less important).

Thus the question arises: are there any show-stopper bugs in gentoo-sources
3.14.x that I should be aware about? They don't have to be directly btrfs
related.

> I also have a more specific question regarding RAID 10: the btrfs wiki says
> that you can add devices with different sizes to a multiple device setup, but I
> don't think it says to which RAID levels this applies and how. From [0] I would
> say it works with RAID 10 (since that's what the example uses), but thought
> maybe somebody here knows more details and/or gotchas. From my understanding,
> this means that I can iteratively upgrade my RAID 10 to larger drives and have
> btrfs use all of the available space (or at least as much as is possible). This
> is important to me because I currently have 4 320 GB HDDs + 1 (possibly broken,
> must check) spare and wish to be able to upgrade without having to buy four
> HDDs at once.

From the [5] I learned that RAID 10 can be extended with *pairs* of drives, so
that answers that question. Since my SATA ports are all occupied, I can't just
hook up two new drives and remove the old ones.

So I have a new question: does "btrfs replace" work if the new drive is larger
than the old one? Again, according to [0] it sounds like it should work, but
it's not clear to me.

As an addendum: btrfs does not support hot spares, but you can easily replace
one drive with another, so I can keep my current setup; I just have to replace
any failed drive manually (for now).

Also (OT): the possibly broken drive (sda) might have simply been a loose SATA
cable. The first time it spontaneously failed (triggering the minor data loss
mentioned above), adjusting the SATA cables got it to work again, albeit
unstably.  Today I adjusted the SATA cables again after a different drive (sdd)
spontaneously gave up yesterday, and sda started passing SMART tests (both short
and long) again (sdd passed, too). Color me confused :-/ . I should see if I
can buy shorter SATA cables so they don't get in each others way so much.

[...]
> [0]
> https://btrfs.wiki.kernel.org/index.php/SysadminGuide#RAID_and_data_replication
> [1] https://wiki.archlinux.org/index.php/Solid_State_Drives
> [2] https://wiki.archlinux.org/index.php/Btrfs
> [3] http://wiki.gentoo.org/wiki/Btrfs
> [4] http://wiki.gentoo.org/wiki/Btrfs_system_root

[5] http://lwn.net/Articles/576276/
[6]
http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/
[7]
http://www.zdnet.com/btrfs-hands-on-my-first-experiments-with-a-new-linux-file-system-7000023681/

> Greetings and thanks in advance for any help given
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 6432 bytes --]

Am Wed, 7 May 2014 00:53:07 +0200
schrieb Marc Joliet <marcec@gmx.de>:

[...]
> > This migration will occur in conjunction with a migration of / + /usr to a
> > cheap SSD that I just bought (Crucial M500 120 GB). The overall plan is thus as
> > follows:
> > 
> > Replace
> > 
> > 	/boot on /dev/md1 (EXT3, RAID 1)
> > 	/ (with assorted sub-directories, sans /usr) on /dev/md2 (EXT4, RAID 10)
> > 	the rest on LVM on /dev/md3 (all LVs EXT4, RAID 10)
> > 
> > with
> > 
> > 	/ + /boot + /usr + swapfile on the SSD (EXT4)
> > 	the rest (/home, my media partitions) on a btrfs RAID 10
> 
> This part I think I will stick with.  From what I've read so far, I wouldn't
> trust my entire system to btrfs.  Since "the rest" consists of stuff I either
> automatically backup (using rsnapshot) or have multiple copies of, I should be
> able to recover from a broken btrfs file system fairly easily.
> 
> While I am unsure of my choice of RAID level (some comments on LWN.net claim
> that the MD RAID 10 is more comparable to btrfs' RAID 1, which I will attempt to
> verify myself beforehand). However, due to btrfs' live rebalancing feature, I
> worry less about this. By the time I really need more space the RAID 5/6 code
> (and maybe N-way mirroring) ought to be stable (or at least finished), or I
> can switch to RAID 1 if I need the flexibility.
> 
> [...]
> > The reason why I would choose EXT4 for the SSD is that btrfs still lacks support
> > for swap files and I worry about creating a swap partition on the SSD. Is that
> > warranted, or will the wear-levelling of the SSD handle that just fine? Do swap
> > partitions support SSDs specially? Also, does anyone know whether EXT4 goes
> > beyond "merely" supporting TRIM? That is, the btrfs wiki advertises the
> > following:
> > 
> >   "SSD (Flash storage) awareness (TRIM/Discard for reporting free blocks for
> >   reuse) and optimizations (e.g. avoiding unnecessary seek optimizations,
> >   sending writes in clusters, even if they are from unrelated files. This
> >   results in larger write operations and faster write throughput)"
> > 
> > Does EXT4 also implement such optimisations for SSDs?
> 
> I will also go ahead with this (despite the open questions), although I will
> leave swap on the LVM for now. I think tonight (well, today) I "just" want to
> get the SSD running. Furthermore, "btrfs convert" should be able to up-convert
> it in the future once btrfs is "production ready" (both articles make a
> guesstimate of about 1-2 years).
> 
> I think I would also prefer running a few days from the SSD before converting
> "the rest" to btrfs, which should be fairly simple at that point.

Weeee, this part is done as of this morning! I am now successfully booting from
the SSD with GRUB 2.

So far I've noticed the following advantages:

- RAID shutdown actually succeeds now (it never did before with / on a RAID 10).

- It feels like my boot time was halved (not counting the BIOS and boot
  loader, though); logging into my desktop also feels much quicker now. After
  that, though, it feels like there are few noticeable advantages in everyday
  usage (I expect portage to become faster, though). Obviously these types of
  speed-ups are to be expected from an SSD, but it still feels worth mentioning.

- GRUB 2 is so much nicer to use than legacy GRUB! Now every time I install a
  new kernel, I just run "grub2-mkconfig -o /boot/grub/grub.cfg" and reboot,
  and everything will just work!

  (I was already used to this on my work/uni laptop, so manually editing legacy
  GRUB's configuration files on my desktop always felt like a chore.)

The whole procedure wasn't that hard, either (ordered slightly more logically
than what I actually did):

- "emerge grub:2" (I have it in parallel to legacy GRUB for now)
- use gparted to create one large EXT4 boot partition on the SSD
- boot to a systemrescuecd
- "cp -a" the / + /usr + ... to the SSD (yes, I should have used rsync)
- copy the kernel images to /boot on the SSD
- chroot into the SSD (following the Gentoo handbook)
- install grub2 as if it were a new install; this was nigh trivial:
  "grub2-mkconfig -o /boot/grub/grub.cfg" followed by "grub-install /dev/sdf"
- edit /etc/fstab to match the layout of the SSD (i.e., remove obsolete mount
  points and update the "/" line)
- reboot (reconfiguring the boot order in the BIOS as necessary)
- ???
- profit! ;-)

(I hope that I didn't forget anything...)

That is, it would have been this straightforward if I hadn't gotten the chroot
wrong the first time around. I had to reboot into the main system, look up the
correct chroot instructions, run "grub2-install /dev/sdf", and then reboot,
which went successfully this time. Oh, and I had also forgotten to set the boot
flag in the first step. Even so, it was pretty straightforward and required
little research beforehand.

Note that I don't know if you even need to chroot, I just did it because... I
remembered needing to do that? To use the version of GRUB 2 that I emerged? I
dunno, I suspected that GRUB's auto-detection features might detect stuff from
the live CD that would be invalid when booting from the SSD.

[ Why did I have to reboot to the main system? Because my dorm network sucks.
  After the Studentenwerk took over the network after the student networking
  team failed to find successors, we went from simple DHCP and a proper network
  to static addresses and PPPoE, if you can believe that; oh, and the network
  performs worse now, too.  Anyway, I didn't want to bother with configuring
  the network from the live CD (which uses NetworkManager), so couldn't look up
  the instructions from there.

  *grumble* stupid over-complicated dorm network *grumble* ]

Anyway, the only steps left are moving /usr/portage and swap to the SSD
(the former is in progress right now).

> > Is btrfs a good choice for / after all?
> 
> I have decided: not without a full system backup (which I don't really want).

After remembering just how small / really is (about 20 GB, even with /usr), I
realised that there really is no reason *not* to include it in my automatic
backups. I'm still not converting the SSD to btrfs, though.

[...]
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 4426 bytes --]

OK, I am in the middle of copying over my data from my backups to the freshly
created btrfs file system :-) . Read more below.

Am Wed, 7 May 2014 00:53:07 +0200
schrieb Marc Joliet <marcec@gmx.de>:

[...]
> While I am unsure of my choice of RAID level (some comments on LWN.net claim
> that the MD RAID 10 is more comparable to btrfs' RAID 1, which I will attempt to
> verify myself beforehand). However, due to btrfs' live rebalancing feature, I
> worry less about this. By the time I really need more space the RAID 5/6 code
> (and maybe N-way mirroring) ought to be stable (or at least finished), or I
> can switch to RAID 1 if I need the flexibility.

I went with RAID 10 for data and RAID 1 for meta-data. I will see how the disk
usage actually turns out and can decide from there whether I want to change
either one if I'm not satisfied.

> [...]
> > The reason why I would choose EXT4 for the SSD is that btrfs still lacks support
> > for swap files and I worry about creating a swap partition on the SSD. Is that
> > warranted, or will the wear-levelling of the SSD handle that just fine? Do swap
> > partitions support SSDs specially? Also, does anyone know whether EXT4 goes
> > beyond "merely" supporting TRIM? That is, the btrfs wiki advertises the
> > following:
> > 
> >   "SSD (Flash storage) awareness (TRIM/Discard for reporting free blocks for
> >   reuse) and optimizations (e.g. avoiding unnecessary seek optimizations,
> >   sending writes in clusters, even if they are from unrelated files. This
> >   results in larger write operations and faster write throughput)"
> > 
> > Does EXT4 also implement such optimisations for SSDs?
> 
> I will also go ahead with this (despite the open questions), although I will
> leave swap on the LVM for now. I think tonight (well, today) I "just" want to
> get the SSD running. Furthermore, "btrfs convert" should be able to up-convert
> it in the future once btrfs is "production ready" (both articles make a
> guesstimate of about 1-2 years).
> 
> I think I would also prefer running a few days from the SSD before converting
> "the rest" to btrfs, which should be fairly simple at that point.

So, as before, the conversion was straightforward, since the RAID + LVM only
contained /home and data, thus I could convert without rebooting (though I will
reboot once the backups are restored, just to see if that works as expected).

Anyway, I performed the following steps:

- remount all affected partitions read-only
- perform one last backup
- unmount the affected partitions
- shut down the logical volumes and the RAID (also, remove mdadm, mdraid and
  lvm from all run-levels)
- run "mkfs.btrfs -f -m raid1 -d raid10 -L MARCEC_STORAGE /dev/sd{a,b,c,d}" (-f
  because the HDDs still had a partition table)
- create subvolumes where I used to have separate partitions (adjusting
  permissions where necessary)
- rsync /home from the backup drive (which was surprisingly fast)

What I am now in the middle of (2/3 of the way through) is syncing my media
partitions. After that, the conversion will be complete, and I will hope and
pray to our noodly overlord that I don't encounter any bugs.

[...]
> Thus the question arises: are there any show-stopper bugs in gentoo-sources
> 3.14.x that I should be aware about? They don't have to be directly btrfs
> related.

This is still an open question. I of course already upgraded prior to the btrfs
conversion and haven't noticed anything out of the ordinary, but I would be
interested in anybody else's experience.

One other thing I noticed: my old RAID had the distinct disadvantage that the
newest drive I had added to it had a different alignment than the old ones.
Since I had to copy the partition table from one of the existing drives, it was
not possible to accommodate this (though I only found out after recently
running "blockdev --getalignoff"). I suspect btrfs could be able to deal with
that better than mdraid, and searching for "alignment" on the btrfs wiki shows
results which heavily imply that it in fact can (quote from the description of
the btrfs_chunk data structure: "Optimal alignment parameters for block I/O").

Anyway, everything seems to be running fine so far :-) .

[...]

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[William Kenworthy]

On 05/06/14 18:18, Marc Joliet wrote:
> Hi all,
> 
> I've become increasingly motivated to convert to btrfs.  From what I've seen,
> it has become increasingly stable; enough so that it is apparently supposed to
> become the default FS on OpenSuse in 13.2.
> 
> I am motivated by various reasons:
....

My btrfs experience:

I have been using btrfs seriously (vs testing) for a while now with
mixed results but the latest kernel/tools seem to be holding up quite well.

~ 2yrs on a Apple/gentoo laptop (I handed it back to work a few months
back) - never a problem! (mounted with discard/trim)

btrfs on a 128MB intel ssd (linux root drive) had to secure reset a few
times as btrfs said the filesystem was full, but there was 60G+ free -
happens after multiple crashes and it seemed the btrfs metadata and the
ssd disagreed on what was actually in use - reset drive and restore from
backups :(  Now running ext4 on that drive with no problems - will move
back to btrfs at some point.

cephfs - rolling disaster but its more to do with not giving the system
adequate resources and using what from ceph's point of view are bad
practises (running ceph on the same machine used for VM's and mounts) -
mostly resulted in gradually corrupted and unrecoverable btrfs
partitions over time.

3 x raid 0+1 (btrfs raid 1 with 3 drives) - working well for about a month

~10+ gentoo VM's, one ubuntu and 3 x Win VM's with kvm/qemu storage on
btrfs - regular scrubs show an occasional VM problem after system crash
(VM server), otherwise problem free since moving to pure btrfs from
ceph.  Gentoo VM's were btrfs in raw qemu containers and are now
converted to qcow2 - no problems since moving from ceph.  Fragmentation
on VM's is a problem but "cp --reflink vm1 vm2" for vm's is really
really cool!

I have a clear impression that btrfs has been incrementally improving
and the current kernel and recovery tools are quite good but its still
possible to end up with an unrecoverable partition (in the sense that
you might be able to get to some of the the data using recovery tools,
but the btrfs mount itself is toast)

Backups using dirvish - was getting an occasional corruption (mainly
checksum) that seemed to coincide with network problems during a backup
sequence - have not seen it for a couple of months now.  Only lost whole
partition once :(  Dirvish really hammers a file system and ext4 usually
dies very quickly so even now btrfs is far better here.

The comments on ceph only hold in my use case i.e., dont do it this way!
 After the experience and problems, I would still choose ceph for its
proper use case (its actually way cool!) - the ceph people do not
recommend btrfs for production use.

I am slowly moving my systems from reiserfs to btrfs as my confidence in
it and its tools builds.  I really dislike ext4 and its ability to lose
valuable data (though that has improved dramaticaly) but it still seems
better than btrfs on solid state and hard use - but after getting burnt
I am avoiding that scenario so need to retest.

BillK

Re: [gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 3992 bytes --]

Am Wed, 07 May 2014 06:56:12 +0800
schrieb William Kenworthy <billk@iinet.net.au>:

> On 05/06/14 18:18, Marc Joliet wrote:
> > Hi all,
> > 
> > I've become increasingly motivated to convert to btrfs.  From what I've seen,
> > it has become increasingly stable; enough so that it is apparently supposed to
> > become the default FS on OpenSuse in 13.2.
> > 
> > I am motivated by various reasons:
> ....
> 
> My btrfs experience:
> 
> I have been using btrfs seriously (vs testing) for a while now with
> mixed results but the latest kernel/tools seem to be holding up quite well.
> 
> ~ 2yrs on a Apple/gentoo laptop (I handed it back to work a few months
> back) - never a problem! (mounted with discard/trim)

That's one HDD, right? From what I've read, that's the most tested and stable
use case for btrfs, so it doesn't surprise me that much that it worked so well.

> btrfs on a 128MB intel ssd (linux root drive) had to secure reset a few
> times as btrfs said the filesystem was full, but there was 60G+ free -
> happens after multiple crashes and it seemed the btrfs metadata and the
> ssd disagreed on what was actually in use - reset drive and restore from
> backups :(  Now running ext4 on that drive with no problems - will move
> back to btrfs at some point.

All the more reason to stick with EXT4 on the SSD for now.

[snip interesting but irrelevant ceph scenario]
> 
> 3 x raid 0+1 (btrfs raid 1 with 3 drives) - working well for about a month

That last one is particularly good to know. I expect RAID 0, 1 and 10 to work
fairly well, since those are the oldest supported RAID levels. 

> ~10+ gentoo VM's, one ubuntu and 3 x Win VM's with kvm/qemu storage on
> btrfs - regular scrubs show an occasional VM problem after system crash
> (VM server), otherwise problem free since moving to pure btrfs from
> ceph.  Gentoo VM's were btrfs in raw qemu containers and are now
> converted to qcow2 - no problems since moving from ceph.  Fragmentation
> on VM's is a problem but "cp --reflink vm1 vm2" for vm's is really
> really cool!

That matches the scenario from the ars technica article; the author is a huge
fan of file cloning in btrfs :) .

And yeah, too bad autodefrag is not yet stable.

> I have a clear impression that btrfs has been incrementally improving
> and the current kernel and recovery tools are quite good but its still
> possible to end up with an unrecoverable partition (in the sense that
> you might be able to get to some of the the data using recovery tools,
> but the btrfs mount itself is toast)
> 
> Backups using dirvish - was getting an occasional corruption (mainly
> checksum) that seemed to coincide with network problems during a backup
> sequence - have not seen it for a couple of months now.  Only lost whole
> partition once :(  Dirvish really hammers a file system and ext4 usually
> dies very quickly so even now btrfs is far better here.

I use rsnapshot here with an external hard drive formatted to EXT4. I'm not
*that* worried about the FS dying, more that it dies at an inopportune moment
where I can't immediately restore it.

[again, snip interesting but irrelevant ceph scenario]
> 
> I am slowly moving my systems from reiserfs to btrfs as my confidence in
> it and its tools builds.  I really dislike ext4 and its ability to lose
> valuable data (though that has improved dramaticaly) but it still seems
> better than btrfs on solid state and hard use - but after getting burnt
> I am avoiding that scenario so need to retest.

Rising confidence: good to hear :) .

Perhaps this will turn out similarly to when I was using the xf86-video-ati
release candidates and bleeding edge gentoo-sources/mesa/libdrm/etc. (for 3D
support in the r600 driver): I start using it shortly before it starts truly
stabilising :) .

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[William Kenworthy]

On 05/07/14 07:51, Marc Joliet wrote:
> Am Wed, 07 May 2014 06:56:12 +0800
> schrieb William Kenworthy <billk@iinet.net.au>:
>
>> On 05/06/14 18:18, Marc Joliet wrote:
>>> Hi all,
>>>
>>> I've become increasingly motivated to convert to btrfs.  From what I've seen,
>>> it has become increasingly stable; enough so that it is apparently supposed to
>>> become the default FS on OpenSuse in 13.2.
>>>
>>> I am motivated by various reasons:
>> ....
>>
>> My btrfs experience:
>>
>> I have been using btrfs seriously (vs testing) for a while now with
>> mixed results but the latest kernel/tools seem to be holding up quite well.
>>
>> ~ 2yrs on a Apple/gentoo laptop (I handed it back to work a few months
>> back) - never a problem! (mounted with discard/trim)
> That's one HDD, right? From what I've read, that's the most tested and stable
> use case for btrfs, so it doesn't surprise me that much that it worked so well.
>
Yes, light duty using the builtin ssd chips on the motherboard.
>> btrfs on a 128MB intel ssd (linux root drive) had to secure reset a few
>> times as btrfs said the filesystem was full, but there was 60G+ free -
>> happens after multiple crashes and it seemed the btrfs metadata and the
>> ssd disagreed on what was actually in use - reset drive and restore from
>> backups :(  Now running ext4 on that drive with no problems - will move
>> back to btrfs at some point.
> All the more reason to stick with EXT4 on the SSD for now.
I have had had very poor luck with ext anything and would hesitate it to
recommend it except for this very specific case where there is little
alternative - reiserfs is far better on platters for instance.
>
> [snip interesting but irrelevant ceph scenario]
Its relevant because it keeps revealing bugs in btrfs by stressing it -
one of those reported by me to ceph was reported upstream by the ceph
team and fixed last year - bugs still exist in btrfs !
>> 3 x raid 0+1 (btrfs raid 1 with 3 drives) - working well for about a month
> That last one is particularly good to know. I expect RAID 0, 1 and 10 to work
> fairly well, since those are the oldest supported RAID levels.
>
>> ~10+ gentoo VM's, one ubuntu and 3 x Win VM's with kvm/qemu storage on
>> btrfs - regular scrubs show an occasional VM problem after system crash
>> (VM server), otherwise problem free since moving to pure btrfs from
>> ceph.  Gentoo VM's were btrfs in raw qemu containers and are now
>> converted to qcow2 - no problems since moving from ceph.  Fragmentation
>> on VM's is a problem but "cp --reflink vm1 vm2" for vm's is really
>> really cool!
> That matches the scenario from the ars technica article; the author is a huge
> fan of file cloning in btrfs :) .
>
> And yeah, too bad autodefrag is not yet stable.
Not that its not stable but that it cant deal with large files that
change randomly on a continual basis like VM virtual disks.
>
>> I have a clear impression that btrfs has been incrementally improving
>> and the current kernel and recovery tools are quite good but its still
>> possible to end up with an unrecoverable partition (in the sense that
>> you might be able to get to some of the the data using recovery tools,
>> but the btrfs mount itself is toast)
>>
>> Backups using dirvish - was getting an occasional corruption (mainly
>> checksum) that seemed to coincide with network problems during a backup
>> sequence - have not seen it for a couple of months now.  Only lost whole
>> partition once :(  Dirvish really hammers a file system and ext4 usually
>> dies very quickly so even now btrfs is far better here.
> I use rsnapshot here with an external hard drive formatted to EXT4. I'm not
> *that* worried about the FS dying, more that it dies at an inopportune moment
> where I can't immediately restore it.
>
> [again, snip interesting but irrelevant ceph scenario]
as I said above - if it fails under ceph, its likely going to fail under
similar stresses using other software - I am not talking ceph bugs (of
which there are many) but actual btrfs corruption.
>> I am slowly moving my systems from reiserfs to btrfs as my confidence in
>> it and its tools builds.  I really dislike ext4 and its ability to lose
>> valuable data (though that has improved dramaticaly) but it still seems
>> better than btrfs on solid state and hard use - but after getting burnt
>> I am avoiding that scenario so need to retest.
> Rising confidence: good to hear :) .
>
> Perhaps this will turn out similarly to when I was using the xf86-video-ati
> release candidates and bleeding edge gentoo-sources/mesa/libdrm/etc. (for 3D
> support in the r600 driver): I start using it shortly before it starts truly
> stabilising :) .
>
More exposure, more bugs will surface and be fixed - its getting there.

BillK

Re: [gentoo-user] planned btrfs conversion: questions

[Stefan G. Weichinger]

Am 08.05.2014 13:57, schrieb William Kenworthy:

> I have had had very poor luck with ext anything and would hesitate it to
> recommend it except for this very specific case where there is little
> alternative - reiserfs is far better on platters for instance.

I would be interested in your experience with ext anything. Poor luck
"only" ? IMO the extX-filesystems are stable and recommended in general.

And I use them in a lot of places if I dont't have specific reasons to
do otherwise.

Stefan

Re: [gentoo-user] planned btrfs conversion: questions

[Stefan G. Weichinger]

Am 08.05.2014 20:14, schrieb Stefan G. Weichinger:
> Am 08.05.2014 13:57, schrieb William Kenworthy:
> 
>> I have had had very poor luck with ext anything and would hesitate it to
>> recommend it except for this very specific case where there is little
>> alternative - reiserfs is far better on platters for instance.
> 
> I would be interested in your experience with ext anything. Poor luck
> "only" ? IMO the extX-filesystems are stable and recommended in general.
> 
> And I use them in a lot of places if I dont't have specific reasons to
> do otherwise.

Aside from that I plugged a non-used older SSD into my SATA/USB-adapter
and put a btrfs-root onto it ...

After some fiddling I currently run my system booted from that ;-)

Yes, sure, slow because of the external SSD right now .. .just to see if
it works.

/home and stuff still on my internal disks, but the / now on a btrfs
subvolume.

Nice.

I will play around with that and see what happens.

Stefan

Re: [gentoo-user] planned btrfs conversion: questions

[Stefan G. Weichinger]

Am 09.05.2014 21:59, schrieb Stefan G. Weichinger:

> Aside from that I plugged a non-used older SSD into my SATA/USB-adapter
> and put a btrfs-root onto it ...
> 
> After some fiddling I currently run my system booted from that ;-)
> 
> Yes, sure, slow because of the external SSD right now .. .just to see if
> it works.
> 
> /home and stuff still on my internal disks, but the / now on a btrfs
> subvolume.
> 
> Nice.
> 
> I will play around with that and see what happens.

The ssd containing the btrfs-root is now inside the box and connected
via SATA ... GRUB2-entries (yes, booting that via UEFI) edited so I have
the choice to select booting from SSD1 where root and /home are on ext4
... and SSD2 where now both root and /home are btrfs-subvolumes.

I adjusted my backup routines as well, sure ;-)

For now I am compiling libreoffice for a start ... just to see if things
work. No multi-device-trickery with btrfs so far ... I want to learn my
way with that one device only.

small note: I compiled and loaded the kernel module crc32c_intel as
mentioned here ->

http://www.funtoo.org/BTRFS_Fun#SSE_4.2_boost

Don't know if that helps much, didn't make specific tests ... right now
the load is high due to the LO-merging.

Looking forward to more learning.
At first now a bit of weekend, afk.

Stefan

Re: [gentoo-user] planned btrfs conversion: questions

[Stefan G. Weichinger]

Am 10.05.2014 13:14, schrieb Stefan G. Weichinger:

> Looking forward to more learning.
> At first now a bit of weekend, afk.

I also had ~180 GB free on one of my thinkpads so I also added a
btrfs-root there just for fun ;-)

Had a bit of fiddling with the boot-options but now I successfully
booted it ... now for some backups and snapshotting.

Stefan

Re: [gentoo-user] planned btrfs conversion: questions

[Stefan G. Weichinger]

Am 11.05.2014 14:35, schrieb Stefan G. Weichinger:
> Am 10.05.2014 13:14, schrieb Stefan G. Weichinger:
> 
>> Looking forward to more learning.
>> At first now a bit of weekend, afk.
> 
> I also had ~180 GB free on one of my thinkpads so I also added a
> btrfs-root there just for fun ;-)
> 
> Had a bit of fiddling with the boot-options but now I successfully
> booted it ... now for some backups and snapshotting.

"mount" does not show me the subvol-id or subvol-name of a mounted
btrfs-subvolume.

Seems like a bug in util-linux to me, in other distros that seems to work.

/proc/mounts and findmnt also don't display that info, does anyone know
how to check that? Would be very handy when dealing with snapshots etc ...

thanks, Stefan

Re: [gentoo-user] planned btrfs conversion: questions

[Peter Humphrey]

On Sunday 11 May 2014 18:17:40 Stefan G. Weichinger wrote:
> Am 11.05.2014 14:35, schrieb Stefan G. Weichinger:
> > Am 10.05.2014 13:14, schrieb Stefan G. Weichinger:
> >> Looking forward to more learning.
> >> At first now a bit of weekend, afk.
> > 
> > I also had ~180 GB free on one of my thinkpads so I also added a
> > btrfs-root there just for fun ;-)
> > 
> > Had a bit of fiddling with the boot-options but now I successfully
> > booted it ... now for some backups and snapshotting.
> 
> "mount" does not show me the subvol-id or subvol-name of a mounted
> btrfs-subvolume.
> 
> Seems like a bug in util-linux to me, in other distros that seems to work.
> 
> /proc/mounts and findmnt also don't display that info, does anyone know
> how to check that? Would be very handy when dealing with snapshots etc ...
> 
> thanks, Stefan

Have you tried the -R option to findmnt? I almost missed it in the man page 
while looking for submounts of /sys and /dev in a chroot.

-- 
Regards
Peter

Re: [gentoo-user] btrfs conversion: first impressions

[Stefan G. Weichinger]

After some initial learning and fiddling I have to say that I really
like the features and possibilities of btrfs so far.

OK, features bring complexity as well ... some technology hides that
more, some less.

... but it is really nice-to-have the option to snapshot your root-fs,
do-something-to-it (emerge unstable stuff, delete the wrong files, you
name it ...), and if you don't like it you simply boot using your
snapshot ... that is actually really helpful and also rather easy once
you get your head wrapped around the concepts and the few steps
necessary (and it's quick: the snapshot is done in a blink ...)

No specific benchmarks done here, the internet is full of ... so far the
performance was not noticeable different from the ext4-fs before. This
might be more visible with hdds, I only used SSDs so far.

As far as I researched btrfs seems to be quite reliable in a not too
complex (read: multi devices) setup ... and backups never hurt anyway.

As I do backups all the time I feel quite confident to test my setups
for the next few days and maybe even completely overhaul my desktop setup.

-> 2x 1TB HDDs plus 1x 256GB SSD (plus the one older 80GB SSD for tests
right now) ... with LVM and stuff (remember my hassles last week with
the LVMs not activated??) ... I could run one btrfs-pool on the 2 HDDs
and one on the SSD and cut all of my various filesystems out of that.

Would mixing hdds and the ssd into one pool make sense? I think, no ... ?

--

I will also test running VMs on btrfs-subvolumes and doing snapshots:

snapshot the underlying subvolume, apply some changes within the VM and
then rollback to the snapshot.

This would remove LVM-snapshotting out of the way ... etc etc

As mentioned before, looking forward ... and curious!

Stefan

Re: [gentoo-user] btrfs conversion: first impressions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 2989 bytes --]

Am Sun, 11 May 2014 23:24:34 +0200
schrieb "Stefan G. Weichinger" <lists@xunil.at>:

[...]
> ... but it is really nice-to-have the option to snapshot your root-fs,
> do-something-to-it (emerge unstable stuff, delete the wrong files, you
> name it ...), and if you don't like it you simply boot using your
> snapshot ... that is actually really helpful and also rather easy once
> you get your head wrapped around the concepts and the few steps
> necessary (and it's quick: the snapshot is done in a blink ...)

In a presentation by Donny Berkholz at Fosdem this year [0], he mentioned the
distro CoreOS, and that they can do atomic updates. I haven't looked it up in
detail, but they're website says that they use a dual-root scheme where the
update is performed in a second root, which is made the real root after
rebooting or after a kexec [1]. It seems to me that this could be made simpler
and easier with btrfs snapshots.

> As far as I researched btrfs seems to be quite reliable in a not too
> complex (read: multi devices) setup ... and backups never hurt anyway.

Of course, for me one of *the* big features was the capability to automatically
fix corrupted data (the self-healing features of btrfs). This is only possible
when you have redundancy across multiple devices. (I'm running a scrub right
now.)

But even with a single device, you can at least *detect* corruption, I just
want to also be able to have it automatically *corrected*.

> As I do backups all the time I feel quite confident to test my setups
> for the next few days and maybe even completely overhaul my desktop setup.

Ditto :) . As risky as it is, this was also a "test" of my backup, in the
sense that, while I knew the backups looked okay by manual inspection, I
hadn't actually restored from backup yet. Obviously, it worked :) .

> -> 2x 1TB HDDs plus 1x 256GB SSD (plus the one older 80GB SSD for tests
> right now) ... with LVM and stuff (remember my hassles last week with
> the LVMs not activated??) ... I could run one btrfs-pool on the 2 HDDs
> and one on the SSD and cut all of my various filesystems out of that.
> 
> Would mixing hdds and the ssd into one pool make sense? I think, no ... ?

I suspect something like bcache would work (except I remember reading that
btrfs does not work with it yet).

> --
> 
> I will also test running VMs on btrfs-subvolumes and doing snapshots:
> 
> snapshot the underlying subvolume, apply some changes within the VM and
> then rollback to the snapshot.
> 
> This would remove LVM-snapshotting out of the way ... etc etc
> 
> As mentioned before, looking forward ... and curious!

I'm glad I motivated some people to try btrfs themselves :) .

[0] https://www.youtube.com/watch?v=egjcVGKwnrw
[1] http://coreos.com/using-coreos/updates/ (section "technical details")
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] btrfs conversion: first impressions

[Stefan G. Weichinger]

Am 12.05.2014 16:30, schrieb Marc Joliet:

> In a presentation by Donny Berkholz at Fosdem this year [0], he
> mentioned the distro CoreOS, and that they can do atomic updates. I
> haven't looked it up in detail, but they're website says that they
> use a dual-root scheme where the update is performed in a second
> root, which is made the real root after rebooting or after a kexec
> [1]. It seems to me that this could be made simpler and easier with
> btrfs snapshots.

Yes. I will maybe look into your link sometimes ...

What I want to research:

systemd is now able to "detect" / and /home (and swap) via the GPT-IDs
... I wonder if I could get rid of fstab or at least the entries for /

Right now when I test my rollbacks I edit (a) the subvolid in the
kernel-line of grub.cfg and (b) in /etc/fstab on the "target" subvol.

Maybe the second part is redundant, I don't know?

I think of generating something like a daily "last known good rootfs"
and a related entry in grub2 :-)

Just playing here so far, but interesting options.

(2nd thought: the subvol would then need to get that GPT-ID?)

>> Would mixing hdds and the ssd into one pool make sense? I think, no
>> ... ?
> 
> I suspect something like bcache would work (except I remember reading
> that btrfs does not work with it yet).

So far I run one btrfs on a partition (/dev/sdc3) of an SSD (containing
my active /, /home and some data I want to have mounted with speedy
performance) and a second btrfs on a 1TB-HDD.

I decided to degrade my mdadm RAID1 and dedicate one of the hdds (sdd)
completely to btrfs ... then migrated the old LVs into subvols today.

Looking good so far (right now I have no more active LVs here ...).

Next steps:

* see if things work :-)

* migrate my other, faster and bigger SSD to the new and shiny root-fs.
I boot from the EFI partition there ... but root and stuff is on the 2nd
SSD.

* decide if to get rid of the old Win7-partition (on sdb) that wasn't
booted for years (?) now ... if I don't do that I don't have a second
hdd with the same size for mirroring the btrfs ...

a) sdd = sdb

b) size of sdd > (size of sdb minus partition-win7)

So more to consider here.

But the btrfs contains mostly pics and music and test-vms ... all backed
up via amanda to tape nearly daily, so no specific need for mirroring.


> I'm glad I motivated some people to try btrfs themselves :) .

Thanks for the reminder .. it fits into my cleaning up here ;-)

Now for some scrubbing, backups and watching TV in the meantime.

Greets, Stefan

Re: [gentoo-user] btrfs conversion: first impressions

[Stefan G. Weichinger]

Am 12.05.2014 20:28, schrieb Stefan G. Weichinger:

> Next steps:
> 
> * see if things work :-)
> 
> * migrate my other, faster and bigger SSD to the new and shiny root-fs.
> I boot from the EFI partition there ... but root and stuff is on the 2nd
> SSD.

done that today ... and removed the older SSD so my main desktop is now
on btrfs-only on one SSD and one HDD ... and currently not actively
mounting any extX-fs or LVM-LVs ...

interesting ;-)

I moved the underlying imagefiles of my VMs into separate
btrfs-subvolumes ... nice to be able to do snapshots here and then ... I
also took a snapshot of my /home before I cleaned up some of my
.thunderbird profile, for example.

Helpful and promising in a way, I hope I won't see big crashes in the
next days or so.

How to transform partitions/directories set up with cryptsetup into this
new world? Set up a btrfs on top of the crypted fs ? I ask because I
look for a clean setup for my 2 thinkpads.

Stefan

Re: [gentoo-user] btrfs conversion: first impressions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

On Wed, 14 May 2014 00:34:12 +0200, Stefan G. Weichinger wrote:

> How to transform partitions/directories set up with cryptsetup into this
> new world? Set up a btrfs on top of the crypted fs ? I ask because I
> look for a clean setup for my 2 thinkpads.

Encrypt the partition(s) with cryptsetup and them use the devices
in /dev/mapper to create the volumes. That's how I have my ZFS pools set
up and I'm looking to do the same when I try BTRFS.


-- 
Neil Bothwick

I'm in shape ... Rounds a shape isn't it?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] btrfs conversion: first impressions

[Stefan G. Weichinger]

Am 14.05.2014 01:02, schrieb Neil Bothwick:
> On Wed, 14 May 2014 00:34:12 +0200, Stefan G. Weichinger wrote:
> 
>> How to transform partitions/directories set up with cryptsetup
>> into this new world? Set up a btrfs on top of the crypted fs ? I
>> ask because I look for a clean setup for my 2 thinkpads.
> 
> Encrypt the partition(s) with cryptsetup and them use the devices 
> in /dev/mapper to create the volumes. That's how I have my ZFS
> pools set up and I'm looking to do the same when I try BTRFS.

Doesn't that screw up the whole idea of checksumming etc ?

In my understanding the FS (=btrfs or zfs) should have the direct
contact to the "metal" (=hdd/sdd) to be fully able to detect bitrot
and stuff.

btrfs inside a crypted volume will work, yes, but it won't be able to
see or correct flipping bits on the hardware, because the underlying
layers of cryptsetup etc might hide them.

Right?

Re: [gentoo-user] btrfs conversion: first impressions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

On Wed, 14 May 2014 01:09:17 +0200, Stefan G. Weichinger wrote:

> >> How to transform partitions/directories set up with cryptsetup
> >> into this new world? Set up a btrfs on top of the crypted fs ? I
> >> ask because I look for a clean setup for my 2 thinkpads.  
> > 
> > Encrypt the partition(s) with cryptsetup and them use the devices 
> > in /dev/mapper to create the volumes. That's how I have my ZFS
> > pools set up and I'm looking to do the same when I try BTRFS.  
> 
> Doesn't that screw up the whole idea of checksumming etc ?

Not to my mind. The bits are recorded and checksummed, that's what
matters. If a bit on a platter is flipped, the decrypted bits will also
change.

> In my understanding the FS (=btrfs or zfs) should have the direct
> contact to the "metal" (=hdd/sdd) to be fully able to detect bitrot
> and stuff.

It is a recommended method of encryption in the BTRFS FAQ.

https://btrfs.wiki.kernel.org/index.php/FAQ#Does_btrfs_support_encryption.3F

As btrfs does not support encryption itself, this or ecryptfs are the
only options. 


-- 
Neil Bothwick

ASSISTANT MANAGER: Feminine form of the word manager (q.v.).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] btrfs conversion: first impressions

[Stefan G. Weichinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 14.05.2014 02:39, schrieb Neil Bothwick:

>> Doesn't that screw up the whole idea of checksumming etc ?
> 
> Not to my mind. The bits are recorded and checksummed, that's what 
> matters. If a bit on a platter is flipped, the decrypted bits will
> also change.

But then the "container" of the btrfs would be corrupted, right?

>> In my understanding the FS (=btrfs or zfs) should have the
>> direct contact to the "metal" (=hdd/sdd) to be fully able to
>> detect bitrot and stuff.
> 
> It is a recommended method of encryption in the BTRFS FAQ.
> 
> https://btrfs.wiki.kernel.org/index.php/FAQ#Does_btrfs_support_encryption.3F
>
>  As btrfs does not support encryption itself, this or ecryptfs are
> the only options.

Yes, I saw that as well. Will give it a try sometimes soon ...

S

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTcyLmAAoJEClcuD1V0PzmV2MP+wWqxS7BhG2RcWgXsDycgstu
yosr450brlmx+NRD4PUq47yj+GTsum/9l2WdxAlDZA90f8CaJefnWK9csNLhGBMA
BqFT69WNSJEmOJ2l8x2FHZJ1gjyMOQp16q7r0d1G7Asdoubhx/uRSeSjJEnk5gqu
UVA77/jbJ1CJSAv4ATnGl67RHWew3KR99VZbbFs7Cr3fjbOZmqWpuSKbf6ZdylHO
UatP/sc+ye09sSpUWW5Q4i8v8gF0XC/qy80iPLy84G0+IzDaNaoTfQY3Ff2+u9Nn
OAhn8lT/HmMJiJUTp4YeAVjzLIx+Nq2NrAqEoaIqER1sfGH5bh2oK5yiJ+mW/8cQ
4OBdcQXoXnDtyQvoT69EELq+vsm5nVi8ibb/msE9r9ImE2FJ+2mHKadFpluBGCnX
xApAaxqkX4lPEphpysdlcq9GF1/eSyGRFM/3tDM/vefy9VokjhXHXAbZ2jhfSoTS
PfWZpxi8sur3z/s/sIVaqMbLG8IdjOfoGKTgtzoYHVqvtlYxOubbvh52iI58CCL7
0LblkAeOSlk2UITYMV+yF3WB0naRjO41LngwwmxcJyRsJDvrzQkSEvhtWaJ9Qlh5
d/k6f2y/hDrpITLxf5C+I1/mg8WDtmgODczUTlSj+u6NCLMUT0onv5XdsEQXuGX6
7yFDEdLFjPEpGoMIBLO/
=dRlH
-----END PGP SIGNATURE-----

Re: [gentoo-user] btrfs conversion: first impressions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On Wed, 14 May 2014 10:01:42 +0200, Stefan G. Weichinger wrote:

> >> Doesn't that screw up the whole idea of checksumming etc ?  
> > 
> > Not to my mind. The bits are recorded and checksummed, that's what 
> > matters. If a bit on a platter is flipped, the decrypted bits will
> > also change.  
> 
> But then the "container" of the btrfs would be corrupted, right?

No, because each element of the RAID is encrypted separately, so btrfs
can still use the good one to repair the bad one. If you were to create
an mdadm array, put dm-crypt on that and then btrfs on top, which is more
efficient because it encrypts each byte only once, then you would lose
the check and repair facilities.


-- 
Neil Bothwick

-Come, come, why they couldn't hit an elephant from this dist-

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] btrfs conversion: first impressions

[Stefan G. Weichinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 14.05.2014 10:42, schrieb Neil Bothwick:
> On Wed, 14 May 2014 10:01:42 +0200, Stefan G. Weichinger wrote:
>> But then the "container" of the btrfs would be corrupted, right?
> 
> No, because each element of the RAID is encrypted separately, so
> btrfs can still use the good one to repair the bad one. If you were
> to create an mdadm array, put dm-crypt on that and then btrfs on
> top, which is more efficient because it encrypts each byte only
> once, then you would lose the check and repair facilities.

What RAID? I think of a laptop with only one SSD inside. You mean the
duplicated metadata in this case?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTcy9LAAoJEClcuD1V0PzmacMP/0kSvPhK5iShKxwAQnJfuSRP
S0Q+d1etq/H6TJoBCwRDuu1OLEU2iiyeOcMgPn7ZsFnFgP3joiNtV2OOVioFFbSY
7DvHujkSuTGrD3KxWGV59dZp1TCEX1UYL/L0UOkSujzDvspB1io7mbbEqlAhARpw
79cHZjMSK4juf6Go21Dos1Fnj5peSLPBASVZl7qmRjORZukZMOh70wVEcVcEVwoW
VGDnme21GuaiMZn862ha1WMfIJVBArPsqial/hSsilS8l9np1IgtgZDUe1uH4VsA
botwaEyr3wrPARBOipqLpCPmK/+R8zihFc/pIbO9c2PsSOKiRQPzVq4VAK4KWyc4
IutpBCO6tOpL6QBtjYmqJk+JF02/Bz1EtmaIfvyUSaQ4ZGP0nEb4qB9K+oWMriBd
FDsOz2QT9+X+h/S7wfw3OPqCBHuqSlqqkyy/TFwCiO9e0GFLz8HONVV1rP+GeK5c
7Y3UBPbRDIbb4lsaySfDKgMtMDQ59ei6mwQL6A9CyA5MrL1R0wydYvOrkHyTRSHK
xy+vuqxL384zHjORWX6uLj1Tu5hbJhG1JgNFrcHMdKwIV6kRJzYMUHMVYmLN+ENX
5Z2IWJRyVlr5yNGE/NoCDJQue2EO9H/gzxpSnwXeFjMpRSuvGN/2Ay2Ml5zSXB6x
qrOj0Zp81jcky7zjXp01
=JG2q
-----END PGP SIGNATURE-----

Re: [gentoo-user] btrfs conversion: first impressions

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

On Wed, 14 May 2014 10:54:35 +0200, Stefan G. Weichinger wrote:

> > No, because each element of the RAID is encrypted separately, so
> > btrfs can still use the good one to repair the bad one. If you were
> > to create an mdadm array, put dm-crypt on that and then btrfs on
> > top, which is more efficient because it encrypts each byte only
> > once, then you would lose the check and repair facilities.  
> 
> What RAID? I think of a laptop with only one SSD inside. You mean the
> duplicated metadata in this case?

Ah right. I've got RAID n the brain because I'm currently resizing the
partitions for a ZFS RAID so I can try out btrfs... and it's all your
fault for starting this thread and piquing my interest in btrfs again!


-- 
Neil Bothwick

Secret hacker rule #11: hackers read manuals.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] btrfs conversion: first impressions

[Stefan G. Weichinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 14.05.2014 11:26, schrieb Neil Bothwick:

>> What RAID? I think of a laptop with only one SSD inside. You mean
>> the duplicated metadata in this case?
> 
> Ah right. I've got RAID n the brain because I'm currently resizing
> the partitions for a ZFS RAID so I can try out btrfs... and it's
> all your fault for starting this thread and piquing my interest in
> btrfs again!

I didn't start it! :-P

I also consider moving the data on a ZFS raid on one of my backup
servers to a btrfs-based RAID ...

But the encryption topic for me is interesting because I right now
have it mixed on my thinkpad:

sda1 - /boot/efi
sda2 - btrfs -> root etc
sda3 - cryptsetup-partition -> /home on it with ext4-fs

S


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTczeoAAoJEClcuD1V0PzmT9YQAJKe8jnZN2kcOMuLV+uRfFcH
ec8wihdOd9eWGB49GjgETgCdtyN3V36AEsaGjkOCWOOJ5z7e6C7N7Mg00eTpFCe+
9yXcXL3g/HKDO/iRGNdh8aMqTmkTV+EtL4V/H0UiCH6AOpTT/ruGX0xBJtsVtJ9T
p2/wt+uFKWJ7qexEcmS1o2DUDZoD2hzUIIxooiuK1U1V5DTzWFyW67/CuBfG0ol4
n48VfjzMWYN8EtHow3Op3mg1bQB3Bley/eqMUoZcgYG+GsCt1q9z0Cxb94Wiz5Kg
l4BNOJDgdE8RZyRbOZ01D5gEk305PKHRaOBnsSmahIGjhB+ukkCv+YaPAecZHy9q
7hVZfyPMN2jqYliQO3kpoukifjcPuhefNor2inpdC0VbKnDs7HbJvihYunbdBJf8
hrP0+ZO2qg7EJQb+jy23YtPSMFmd3Rwvx2MibrkFvjYmqFU7ljDJcqmLYAlxL3uB
VekUCHTZlOY9POcV3bt2nrA7kR6rQjdERJEd+Gw1FnPhLGt8c9iRue+bjE/Wg2QG
+kbxWVzZ50Z7HCe7aqKE6aCuUSB96miBNntK+MQ+OqF6L9yjMUPlsS2jabqKDeUj
VYN21rHj8/SxC2J5vEyAM88Lb4JxLZWqrizicjqCWgxVkQojnr0aWPfQOmC2YJPA
WNv7VJxCgHzqXVwxmaOo
=R2z9
-----END PGP SIGNATURE-----

Re: [gentoo-user] btrfs conversion: first impressions

[Stefan G. Weichinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 14.05.2014 11:30, schrieb Stefan G. Weichinger:

> But the encryption topic for me is interesting because I right now 
> have it mixed on my thinkpad:
> 
> sda1 - /boot/efi sda2 - btrfs -> root etc sda3 -
> cryptsetup-partition -> /home on it with ext4-fs

I just now converted the ext4 to btrfs and edited pam_mount.conf.xml
... works!

I just have to clean up the top level volume of btrfs as one should
use a subvolume for the actual data.

Now to changing the luks-passphrase and teaching gnome-something to
unlock it. Weak old passphrase/password ...



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTdFwNAAoJEClcuD1V0PzmTJEQALC4p9+CxPJYhQrD7GIi4oqH
tulczyMgi21GmmGKmsonhR//ZlwIWn2XAljwvEpmHREaDq/o+7dqh1aLABVjFeQN
a+adtBZBNEBW8gWuRksXpf4AjQSB1OKRvcZ6nabnrgKTUMgiB7OcaxE7JBowPwwk
BlxyVGmcmXKuFEXyMsQCdfcEU5/jPVFNB14hcMS4DmxLjVlLMhgEyGVY8qKjdPOs
vFlxX3t01KQWam5kHtQ2BoUmBGVRTO4DqPtrLHuRVBOT6MToBhbTJZJupzx54kDE
kLjie57K3gmoptAdLEVJjcv44hzK+VpPa+d0Fusl6VXNkB1CTaJl2CF53f5rZWXj
aWc+psEgEsXibyXQSoj6vctQft9OMgn0bpTwKd7nA11jS+ANOcusOJbL8rDnSXNu
zMk4KSL+CPLo+XkwtBIjI8k+fJMNh1rufvdcTvqBADiJSi67wMIsdlH8hal4O2Cb
/X8BqApZQfTSWyDEq8KVUecMEIK5fINENqU+PQHHvAs36OemFMQ2Bf+Z6gjN9w3T
cA14SOQquXVBkK2sd4BLmVReY0Gd8ulGw1EmVxSJ7l0VMk/amSsvq689UsFfxLC0
aCkn57ih4aEPRbOzwlsQg1tNhjD7D3rTVfJWUtblZSDddgKTyzqnzeHtwHMwIbJ7
xaRJGuOn8Lf7DXHbbMti
=hUHb
-----END PGP SIGNATURE-----

Re: [gentoo-user] planned btrfs conversion: questions

[Stefan G. Weichinger]

Am 11.05.2014 18:17, schrieb Stefan G. Weichinger:

> "mount" does not show me the subvol-id or subvol-name of a mounted
> btrfs-subvolume.
> 
> Seems like a bug in util-linux to me, in other distros that seems to work.
> 
> /proc/mounts and findmnt also don't display that info, does anyone know
> how to check that? Would be very handy when dealing with snapshots etc ...

https://bugs.gentoo.org/show_bug.cgi?id=510148

Re: [gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 6729 bytes --]

Am Thu, 08 May 2014 19:57:34 +0800
schrieb William Kenworthy <billk@iinet.net.au>:

> On 05/07/14 07:51, Marc Joliet wrote:
> > Am Wed, 07 May 2014 06:56:12 +0800
> > schrieb William Kenworthy <billk@iinet.net.au>:
> >
> >> On 05/06/14 18:18, Marc Joliet wrote:
> >>> Hi all,
> >>>
> >>> I've become increasingly motivated to convert to btrfs.  From what I've seen,
> >>> it has become increasingly stable; enough so that it is apparently supposed to
> >>> become the default FS on OpenSuse in 13.2.
> >>>
> >>> I am motivated by various reasons:
> >> ....
> >>
> >> My btrfs experience:
> >>
> >> I have been using btrfs seriously (vs testing) for a while now with
> >> mixed results but the latest kernel/tools seem to be holding up quite well.
> >>
> >> ~ 2yrs on a Apple/gentoo laptop (I handed it back to work a few months
> >> back) - never a problem! (mounted with discard/trim)
> > That's one HDD, right? From what I've read, that's the most tested and stable
> > use case for btrfs, so it doesn't surprise me that much that it worked so well.
> >
> Yes, light duty using the builtin ssd chips on the motherboard.

SSD chips on the motherboard? I just did a quick googled (well, "duckduckwent"
would be more accurate; funny how language works), do you mean something like
the Intel Z68 chipset?

> >> btrfs on a 128MB intel ssd (linux root drive) had to secure reset a few
> >> times as btrfs said the filesystem was full, but there was 60G+ free -
> >> happens after multiple crashes and it seemed the btrfs metadata and the
> >> ssd disagreed on what was actually in use - reset drive and restore from
> >> backups :(  Now running ext4 on that drive with no problems - will move
> >> back to btrfs at some point.
> > All the more reason to stick with EXT4 on the SSD for now.
> I have had had very poor luck with ext anything and would hesitate it to
> recommend it except for this very specific case where there is little
> alternative - reiserfs is far better on platters for instance.

I, like Stefan, am interested in precisely what kind of negative experiences
you've had with ext*.

I used to use reiserfs (from waaay back when I still used SuSE), but the only
remnant of that is a broken external HDD that I want to attempt a ddrescue on
someday (really the only reason I still keep around reiserfs support in my
kernel). The only thing I really miss is tail packing of small files; my actual
disk usage grew noticeably after my switch to ext4. But ext* have the distinct
advantage of being used pretty much everywhere, which, as we all know, is an
important factor in finding bugs. Reiserfs, in comparison, is AFAIK
unmaintained now (of course, it's maintained in the sense that the existing
code is kept working, but that's beside the point).

> > [snip interesting but irrelevant ceph scenario]
> Its relevant because it keeps revealing bugs in btrfs by stressing it -
> one of those reported by me to ceph was reported upstream by the ceph
> team and fixed last year - bugs still exist in btrfs !

Sorry, I read "ceph" and immediately thought "OK, clustering file system, way
outside of experience" and didn't realise you were talking about outright bugs
in btrfs (I kind of assumed a situation similar to btrfs and swap files: one
piece of software (e.g., the kernel swap code) making assumptions that don't
hold for btrfs).

> >> 3 x raid 0+1 (btrfs raid 1 with 3 drives) - working well for about a month
> > That last one is particularly good to know. I expect RAID 0, 1 and 10 to work
> > fairly well, since those are the oldest supported RAID levels.
> >
> >> ~10+ gentoo VM's, one ubuntu and 3 x Win VM's with kvm/qemu storage on
> >> btrfs - regular scrubs show an occasional VM problem after system crash
> >> (VM server), otherwise problem free since moving to pure btrfs from
> >> ceph.  Gentoo VM's were btrfs in raw qemu containers and are now
> >> converted to qcow2 - no problems since moving from ceph.  Fragmentation
> >> on VM's is a problem but "cp --reflink vm1 vm2" for vm's is really
> >> really cool!
> > That matches the scenario from the ars technica article; the author is a huge
> > fan of file cloning in btrfs :) .
> >
> > And yeah, too bad autodefrag is not yet stable.
> Not that its not stable but that it cant deal with large files that
> change randomly on a continual basis like VM virtual disks.

Oh, I thought it was still considered new and "unpolished" (I did not mean
buggy!).

> >> I have a clear impression that btrfs has been incrementally improving
> >> and the current kernel and recovery tools are quite good but its still
> >> possible to end up with an unrecoverable partition (in the sense that
> >> you might be able to get to some of the the data using recovery tools,
> >> but the btrfs mount itself is toast)
> >>
> >> Backups using dirvish - was getting an occasional corruption (mainly
> >> checksum) that seemed to coincide with network problems during a backup
> >> sequence - have not seen it for a couple of months now.  Only lost whole
> >> partition once :(  Dirvish really hammers a file system and ext4 usually
> >> dies very quickly so even now btrfs is far better here.
> > I use rsnapshot here with an external hard drive formatted to EXT4. I'm not
> > *that* worried about the FS dying, more that it dies at an inopportune moment
> > where I can't immediately restore it.
> >
> > [again, snip interesting but irrelevant ceph scenario]
> as I said above - if it fails under ceph, its likely going to fail under
> similar stresses using other software - I am not talking ceph bugs (of
> which there are many) but actual btrfs corruption.

Got it, thanks!

> >> I am slowly moving my systems from reiserfs to btrfs as my confidence in
> >> it and its tools builds.  I really dislike ext4 and its ability to lose
> >> valuable data (though that has improved dramaticaly) but it still seems
> >> better than btrfs on solid state and hard use - but after getting burnt
> >> I am avoiding that scenario so need to retest.
> > Rising confidence: good to hear :) .
> >
> > Perhaps this will turn out similarly to when I was using the xf86-video-ati
> > release candidates and bleeding edge gentoo-sources/mesa/libdrm/etc. (for 3D
> > support in the r600 driver): I start using it shortly before it starts truly
> > stabilising :) .
> >
> More exposure, more bugs will surface and be fixed - its getting there.

I sure hope so, I'll be making the switch either tomorrow or Monday :-) .

> BillK

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[William Kenworthy]

On 05/08/14 19:57, William Kenworthy wrote:
> On 05/07/14 07:51, Marc Joliet wrote:
>> Am Wed, 07 May 2014 06:56:12 +0800
>> schrieb William Kenworthy <billk@iinet.net.au>:
>>
>>> On 05/06/14 18:18, Marc Joliet wrote:
>>>> Hi all,
>>>>
>>>> I've become increasingly motivated to convert to btrfs.  From what I've seen,
>>>> it has become increasingly stable; enough so that it is apparently supposed to
>>>> become the default FS on OpenSuse in 13.2.
>>>>
>>>> I am motivated by various reasons:
>>> ....
>>>

scrub status for 20a18d40-e4e6-4e94-88e6-23eacd629cba
        scrub started at Sat May 10 14:27:35 2014, running for 10520 seconds
        total bytes scrubbed: 676.67GiB with 2 errors
        error details: csum=2
        corrected errors: 0, uncorrectable errors: 2, unverified errors: 0

and from /var/log/messages

May 10 16:15:26 moriah kernel: btrfs: checksum error at logical
486690582528 on dev /dev/mapper/vg1-backups, sector 965263992, root 5,
inode 10156876, offset 3179999232, length 4096, links 1 (path:
olympus/20140426/tree/mnt/data/vm/asterisk/asterisk_disk1.qcow2)
May 10 16:15:26 moriah kernel: btrfs: bdev /dev/mapper/vg1-backups errs:
wr 0, rd 14, flush 0, corrupt 11, gen 0
May 10 16:15:26 moriah kernel: btrfs: unable to fixup (regular) error at
logical 486690582528 on dev /dev/mapper/vg1-backups


I'll delete the file(s) (its just one of numerous dated backups)

No event I can put it down to, nothing I can find in the logs - just
happens on an irregular basis!

Its btrfs using compress=LZO on an LVM2 partition after running a few
dirvish multiple backup sessions.  Note that as I said in my original
email, "dirvish" really hammers a file system and only reiserfs seems to
withstand it though I have gotten errors with it in the past.  Ive tried
ext4 (takes only a couple of backup sessions and its unrecoverable,
btrfs an occasional error with two complete losses of the
partition/filesystem since Christmas and reiserfs gets rare errors.

Note that hardware and other items have been changed over time so I
doubt its a specific problem there.

Not quite there yet ...

BillK

Re: [gentoo-user] planned btrfs conversion: questions

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1454 bytes --]

On Saturday 10 May 2014 10:33:19 William Kenworthy wrote:
> Note that as I said in my original
> email, "dirvish" really hammers a file system and only reiserfs seems to
> withstand it though I have gotten errors with it in the past.  Ive tried
> ext4 (takes only a couple of backup sessions and its unrecoverable,
> btrfs an occasional error with two complete losses of the
> partition/filesystem since Christmas and reiserfs gets rare errors.


I moved away from reisefs to ext4 because I was getting some random lockups 
when I/O was high.  While on reiserfs I also had a couple of corrupt mysql 
files and all around poor performance.  Now, this was on a machine with a 
deficient PSU (I replaced a couple of capacitors since then and it is now 
working properly) so I don't want to blame the filesystem because of this 
hardware problem.  In any case, under these impaired conditions ext4 was a 
much better performing filesystem than reiserfs.  No lock ups, significantly 
faster and no corruption was observed in normal operation - I didn't try to 
hammer it.

So I read your paragraph above with surprise, because in my experience the 
opposite was true.  At the time I thought that reiserfs was perhaps suffering 
from bitrot, because these symptoms had gotten worse over time.  This is on an 
installation running since 2005.  Not sure what to conclude from these 
anecdotal observations ...  :-/

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[William Kenworthy]

On 05/11/14 16:53, Mick wrote:
> On Saturday 10 May 2014 10:33:19 William Kenworthy wrote:
>> Note that as I said in my original
>> email, "dirvish" really hammers a file system and only reiserfs seems to
>> withstand it though I have gotten errors with it in the past.  Ive tried
>> ext4 (takes only a couple of backup sessions and its unrecoverable,
>> btrfs an occasional error with two complete losses of the
>> partition/filesystem since Christmas and reiserfs gets rare errors.
> 
> 
> I moved away from reisefs to ext4 because I was getting some random lockups 
> when I/O was high.  While on reiserfs I also had a couple of corrupt mysql 
> files and all around poor performance.  Now, this was on a machine with a 
> deficient PSU (I replaced a couple of capacitors since then and it is now 
> working properly) so I don't want to blame the filesystem because of this 
> hardware problem.  In any case, under these impaired conditions ext4 was a 
> much better performing filesystem than reiserfs.  No lock ups, significantly 
> faster and no corruption was observed in normal operation - I didn't try to 
> hammer it.
> 
> So I read your paragraph above with surprise, because in my experience the 
> opposite was true.  At the time I thought that reiserfs was perhaps suffering 
> from bitrot, because these symptoms had gotten worse over time.  This is on an 
> installation running since 2005.  Not sure what to conclude from these 
> anecdotal observations ...  :-/
> 

Everyone's use case and experience with filesystems seems a bit
different. One reason I am moving from reiserfs is bitrot - I can see
that reiserfs is losing favour.  btrfs has potential ...

BillK

Re: [gentoo-user] planned btrfs conversion: questions

[Peter Humphrey]

On Sunday 11 May 2014 18:43:53 William Kenworthy wrote:

> One reason I am moving from reiserfs is bitrot - I can see that reiserfs is
> losing favour.

I'd been given that impression too, so a month or two ago I converted my few 
reiserfs partitions to ext4. When I came to restoring the /usr/portage 
partition from backup I had to double its size to fit the files in (or I could 
have tried fiddling with the arguments to mkfs.ext4, but resizing was easier at 
the time).

-- 
Regards
Peter

Re: [gentoo-user] planned btrfs conversion: questions

[Alan McKinnon]

On 11/05/2014 14:29, Peter Humphrey wrote:
> On Sunday 11 May 2014 18:43:53 William Kenworthy wrote:
> 
>> One reason I am moving from reiserfs is bitrot - I can see that reiserfs is
>> losing favour.
> 
> I'd been given that impression too, so a month or two ago I converted my few 
> reiserfs partitions to ext4. When I came to restoring the /usr/portage 
> partition from backup I had to double its size to fit the files in (or I could 
> have tried fiddling with the arguments to mkfs.ext4, but resizing was easier at 
> the time).
> 

Reiserfs bitrot was always a sad thing for me with the portage tree.

Reiser always truly excelled at storing, accessing and deleting
thousands of small files, just like the tree contains.

Like you, I ended up making my ext4 volumes for the tree twice as big
and got over it. I still miss the reiser performance though

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 2659 bytes --]

Am Sun, 11 May 2014 09:53:10 +0100
schrieb Mick <michaelkintzios@gmail.com>:

> On Saturday 10 May 2014 10:33:19 William Kenworthy wrote:
> > Note that as I said in my original
> > email, "dirvish" really hammers a file system and only reiserfs seems to
> > withstand it though I have gotten errors with it in the past.  Ive tried
> > ext4 (takes only a couple of backup sessions and its unrecoverable,
> > btrfs an occasional error with two complete losses of the
> > partition/filesystem since Christmas and reiserfs gets rare errors.
> 
> 
> I moved away from reisefs to ext4 because I was getting some random lockups 
> when I/O was high.  While on reiserfs I also had a couple of corrupt mysql 
> files and all around poor performance.  Now, this was on a machine with a 
> deficient PSU (I replaced a couple of capacitors since then and it is now 
> working properly) so I don't want to blame the filesystem because of this 
> hardware problem.  In any case, under these impaired conditions ext4 was a 
> much better performing filesystem than reiserfs.  No lock ups, significantly 
> faster and no corruption was observed in normal operation - I didn't try to 
> hammer it.
> 
> So I read your paragraph above with surprise, because in my experience the 
> opposite was true.  At the time I thought that reiserfs was perhaps suffering 
> from bitrot, because these symptoms had gotten worse over time.  This is on an 
> installation running since 2005.  Not sure what to conclude from these 
> anecdotal observations ...  :-/

I remember that OpenSuse also changed their default from ReiserFS to Ext3 for
similar reasons [0]. I never experienced any problems with ReiserFS, but I also
got the impression that it was going nowhere and that if I wanted a reliable
system, Ext4 was the way to go.

Although funnily enough, when browsing through the btrfs ML, Duncan [1] shares
William's experience of ReiserFS being more resilient than even Ext4. In fact,
he writes that one reason he trusts btrfs to the degree he does is that Chris
Mason (the btrfs lead developer, for those who don't know) has a history with
ReiserFS (he used to work on it for SuSE [2]).

So yeah, as has been said already, everyone has different experiences.

[0] https://en.wikipedia.org/wiki/Reiserfs#Move_away_from_ReiserFS_to_ext3
[1] He used to be a regular in gentoo-amd64, when it was still active; yes, you
know, the guy who wrote the longest emails ;) .
[2] https://en.wikipedia.org/wiki/Btrfs#History
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[Peter Humphrey]

On Monday 12 May 2014 16:08:00 Marc Joliet wrote:

> [1] He used to be a regular in gentoo-amd64, when it was still active; yes,
> you know, the guy who wrote the longest emails ;) .

...and is the only person ever to have found his way into my kill file.

-- 
Regards
Peter

Re: [gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

Am Mon, 12 May 2014 15:39:41 +0100
schrieb Peter Humphrey <peter@prh.myzen.co.uk>:

> On Monday 12 May 2014 16:08:00 Marc Joliet wrote:
> 
> > [1] He used to be a regular in gentoo-amd64, when it was still active; yes,
> > you know, the guy who wrote the longest emails ;) .
> 
> ...and is the only person ever to have found his way into my kill file.

Personally, I appreciated his responses. He was also always friendly. When I
didn't feel like reading them, I simply skipped them. That, to me at least, is
less work than putting him in a kill file. There have been, however, plenty of
other candidates that I *almost* put in a kill file, but didn't, because no one
has yet shown themselves to be simultaneously overly annoying and
unknowledgeable.

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[Dale]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

Marc Joliet wrote:
> Am Mon, 12 May 2014 15:39:41 +0100
> schrieb Peter Humphrey <peter@prh.myzen.co.uk>:
>
>> On Monday 12 May 2014 16:08:00 Marc Joliet wrote:
>>
>>> [1] He used to be a regular in gentoo-amd64, when it was still
active; yes,
>>> you know, the guy who wrote the longest emails ;) .
>>
>> ...and is the only person ever to have found his way into my kill file.
>
> Personally, I appreciated his responses. He was also always friendly.


+1.  Where did he go anyway?

Dale

:-)  :-)

-- 
I am only responsible for what I said ... Not for what you understood or
how you interpreted my words!


[-- Attachment #2: Type: text/html, Size: 1185 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]

Am Mon, 12 May 2014 11:15:21 -0500
schrieb Dale <rdalek1967@gmail.com>:

> Marc Joliet wrote:
> > Am Mon, 12 May 2014 15:39:41 +0100
> > schrieb Peter Humphrey <peter@prh.myzen.co.uk>:
> >
> >> On Monday 12 May 2014 16:08:00 Marc Joliet wrote:
> >>
> >>> [1] He used to be a regular in gentoo-amd64, when it was still
> active; yes,
> >>> you know, the guy who wrote the longest emails ;) .
> >>
> >> ...and is the only person ever to have found his way into my kill file.
> >
> > Personally, I appreciated his responses. He was also always friendly.
> 
> 
> +1.  Where did he go anyway?

Well, he's definitely a regular on the Btrfs mailing list. He also still posts
to gentoo-amd64 whenever the (nowadays very rare) question pops up; the last
thread was back in March. He's probably also active on various other lists.

I don't think he ever subscribed to gentoo-user.

> Dale
> 
> :-)  :-)
> 

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] planned btrfs conversion: questions

[Daniel Frey]

On 05/12/2014 08:04 AM, Marc Joliet wrote:
> There have been, however, plenty of
> other candidates that I *almost* put in a kill file, but didn't, because no one
> has yet shown themselves to be simultaneously overly annoying and
> unknowledgeable.
> 

Is that a challenge? ;-)

Dan

[gentoo-user] Re: planned btrfs conversion: questions

[Jonathan Callen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 05/06/2014 06:18 AM, Marc Joliet wrote:
> Hi all,
> 
> I've become increasingly motivated to convert to btrfs.  From what I've seen, it has become
> increasingly stable; enough so that it is apparently supposed to become the default FS on
> OpenSuse in 13.2.
> 
> I am motivated by various reasons:
> 
> - The experience of insignificant data loss after a reboot after no (visible) problems during
> normal operations (no similar occurrence since). I suspect this would have been discovered
> sooner by btrfs' data checksumming.
> 
> While the data lost was of a small amount, not to mention insignificant (internet pictures) and
> could be retrieved again easily, I worry that this sort of silent corruption might happen to
> data that I *do* care about. My backups would become less than useless if I were to discover
> such corruption too late.
> 
> - The outright sexy multiple device support :) .
> 
> This migration will occur in conjunction with a migration of / + /usr to a cheap SSD that I
> just bought (Crucial M500 120 GB). The overall plan is thus as follows:
> 
> Replace
> 
> /boot on /dev/md1 (EXT3, RAID 1) / (with assorted sub-directories, sans /usr) on /dev/md2
> (EXT4, RAID 10) the rest on LVM on /dev/md3 (all LVs EXT4, RAID 10)
> 
> with
> 
> / + /boot + /usr + swapfile on the SSD (EXT4) the rest (/home, my media partitions) on a btrfs
> RAID 10
> 
> (which replaces an older plan to recreate the RAID 10 + LVM to use the whole disks with the
> current 1.2 metadata format)
> 
> The goals are, in addition to alleviating my data safety concerns above, to guarantee that I
> don't need an initramfs at boot (hence the SSD), and to greatly simplify my partitioning scheme
> (I just have too many separate logical volumes ;-) ). Any added performance is "just" a nice
> bonus.
> 
> The reason why I would choose EXT4 for the SSD is that btrfs still lacks support for swap files
> and I worry about creating a swap partition on the SSD. Is that warranted, or will the
> wear-levelling of the SSD handle that just fine? Do swap partitions support SSDs specially?
> Also, does anyone know whether EXT4 goes beyond "merely" supporting TRIM? That is, the btrfs
> wiki advertises the following:
> 
> "SSD (Flash storage) awareness (TRIM/Discard for reporting free blocks for reuse) and
> optimizations (e.g. avoiding unnecessary seek optimizations, sending writes in clusters, even
> if they are from unrelated files. This results in larger write operations and faster write
> throughput)"
> 
> Does EXT4 also implement such optimisations for SSDs?
> 
> So I guess I want to know: does anybody have any further suggestions to make? Is btrfs a good
> choice for / after all? And should I be using the most recent kernel versions? (I would go with
> no, despite the advice from upstream, because the changes in the last two versions don't seem
> to be particularly user visible, at least to me, from reading kernelnewbies.org.)
> 
> I also have a more specific question regarding RAID 10: the btrfs wiki says that you can add
> devices with different sizes to a multiple device setup, but I don't think it says to which
> RAID levels this applies and how. From [0] I would say it works with RAID 10 (since that's what
> the example uses), but thought maybe somebody here knows more details and/or gotchas. From my
> understanding, this means that I can iteratively upgrade my RAID 10 to larger drives and have 
> btrfs use all of the available space (or at least as much as is possible). This is important to
> me because I currently have 4 320 GB HDDs + 1 (possibly broken, must check) spare and wish to
> be able to upgrade without having to buy four HDDs at once.
> 
> Now to the migration plan: first, partition the SSD and copy all relevant file systems to it;
> this will be done from a Live-CD (SystemRescueCD). After I have configured the mount options
> appropriately and can boot from that, I should be able to mount the other file systems (/home
> and my media partitions) read-only from the actual system and do the RAID + LVM -> btrfs
> migration from there.
> 
> Note that I will in general follow the advice from [1-3], and if people recommend btrfs on /,
> then I will also try to get relevant information from [4].
> 
> [0] https://btrfs.wiki.kernel.org/index.php/SysadminGuide#RAID_and_data_replication [1]
> https://wiki.archlinux.org/index.php/Solid_State_Drives [2]
> https://wiki.archlinux.org/index.php/Btrfs [3] http://wiki.gentoo.org/wiki/Btrfs [4]
> http://wiki.gentoo.org/wiki/Btrfs_system_root
> 
> Greetings and thanks in advance for any help given
> 

As I understand it, when you use BTRFS's internal "RAID1" implementation, the filesystem ensures
that all the data is on at least two different drives, but otherwise doesn't require that the
drives are the same size, etc.  The amount of available space will effectively be the lesser of
50% of the total space on all drives or 100% of the total of all the drives *except* the largest
(in which case, it would be likely that *all* of your data would be stored on the largest drive in
addition to being split among the other drives).

- -- 
Jonathan Callen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJTaZt/AAoJELHSF2kinlg4geoP/Rz1Kn4Bu0c/hXr5kuPkP/wo
BXrfmKQySrWRaeXUx2Am1K7S0ZplM3gLm75NIM6dceeeucSj71x4AVFUY6Ds8B4q
7WGmALfdSfzFNcwLhiLdI1WCkCGGQr3deWCRg9AUFt742D3Uo9n6k2XQecbxsw56
CFwcazECZkDOMkbXr+gL+E0uj2RUM09gquhuZkoXZGIp6bbS7bDCgcX6KKv8wz/s
etIbgd/mcvqBBQrm6LzjYthYlNB8osH9Szvq7boCMOkzLF5l3JDtmLdN6c30H04H
DUXSOqt5ZAzH7uXAj5sgT01/1DpqOg0WRDLvVw+y+bagwIsN1stns6rWAIZI64Pf
Do1J8pX5iAcPsMr+YjmiclgwJxRQlC77HHhn48qFqjrMaUhdqk4XGYoWFhvpvGs1
u4yhZMOt0JpBH5sj8IlJEcjTMu8RS6Zsido6CxlwOOAGikrqRJq9XTgDvVUVFnUQ
AyAeIbI5CHjbe0nbvbUJvz+gsHTdzeY2F+Q7APK4LTzSDMzVIDgC5f4WcJSCnVz5
u4IwEPd6R/m2jII4x3gU+0A5LKaP4CyPFsTjeZjOu6rgIyITN6Cmnb248u3Z0jVh
nYX2amJNn4/53jQfiLC2PW35O3QsDn1VrSL23b74xHJIvsTENCtjwBTI+yav3val
Yr64ISgBtRdbSeh5WWD4
=E3vP
-----END PGP SIGNATURE-----

[gentoo-user] experience thus far (was: planned btrfs conversion: questions)

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

So, a week has passed since my conversion to btrfs.

So far there seem to have been no problems, my system has been running as if
nothing has changed :) . Which, as a friend pointed out, is how it should be.

I don't think there is anything particularly interesting to mention in addition
to what I already wrote. I can just say that I think the effort was worth it.

The one thing that I can tell from reading the past two weeks of the btrfs ML
is that the 3.15 Linux kernel series will contain lots of bug fixes (for
example in balancing, error handling, and send/receive), and that I will want
to use that sooner rather than later. Of course, the severity of the problems
varies, and a lot are triggered under odd, or at least uncommon, circumstances.
Still, its worth paying attention to.

Also, a lot of problem reports I saw came from people using other volume
management below btrfs, interestingly enough.

As for the future, I think I will wait a while, and get some experience with
btrfs first.  I suspect that by the time btrfs supports swap files, it will be
stable enough that I would consider converting my SSD to also use btrfs
anyway :) . Possibly before that, once I am fully convinced of btrfs'
stability, I will also convert my backup drive and switch to using snapshots
and send/receive to perform backups. Perhaps somebody will have written a
backup solution on top of snapshots by then.

Have a nice weekend,
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] experience thus far (was: planned btrfs conversion: questions)

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

Am Fri, 16 May 2014 22:15:58 +0200
schrieb Marc Joliet <marcec@gmx.de>:

[...]
> The one thing that I can tell from reading the past two weeks of the btrfs ML
> is that the 3.15 Linux kernel series will contain lots of bug fixes (for
> example in balancing, error handling, and send/receive), and that I will want
> to use that sooner rather than later. Of course, the severity of the problems
> varies, and a lot are triggered under odd, or at least uncommon, circumstances.
> Still, its worth paying attention to.
[...]

Sorry, obviously I didn't mean 3.15, which had its first RC in April, but
3.16 (and later, depending on when the patches get merged), though 3.15 will
most likely also have lots of bug fixes.

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] experience thus far

[William Kenworthy]

On 17/05/14 04:15, Marc Joliet wrote:
> So, a week has passed since my conversion to btrfs.
> 
> So far there seem to have been no problems, my system has been running as if
> nothing has changed :) . Which, as a friend pointed out, is how it should be.
> 
> I don't think there is anything particularly interesting to mention in addition
> to what I already wrote. I can just say that I think the effort was worth it.
> 
> The one thing that I can tell from reading the past two weeks of the btrfs ML
> is that the 3.15 Linux kernel series will contain lots of bug fixes (for
> example in balancing, error handling, and send/receive), and that I will want
> to use that sooner rather than later. Of course, the severity of the problems
> varies, and a lot are triggered under odd, or at least uncommon, circumstances.
> Still, its worth paying attention to.
> 
> Also, a lot of problem reports I saw came from people using other volume
> management below btrfs, interestingly enough.
> 
> As for the future, I think I will wait a while, and get some experience with
> btrfs first.  I suspect that by the time btrfs supports swap files, it will be
> stable enough that I would consider converting my SSD to also use btrfs
> anyway :) . Possibly before that, once I am fully convinced of btrfs'
> stability, I will also convert my backup drive and switch to using snapshots
> and send/receive to perform backups. Perhaps somebody will have written a
> backup solution on top of snapshots by then.
> 
> Have a nice weekend,
> 

Don't forget to have a maintenance program - run a scrub regularly once
a week or so - I have enough btrfs drives (22 qemu files, 4 WD Greens
att) to see about one or two scrub fixable errors a week with no obvious
cause, sometimes serious (in a critical file).  My experience is that if
you ignore these errors they seem to increase over time resulting in a
crash and burn.  Keep an eye on your logs as btrfs will list the errors
there as well ("grep -i btrfs /var/log/messages").  For the ones scrub
cant fix, delete the file and restore from backup.  Errors that require
off-line fixing (btrfsck) are the ones where I have lost file systems -
though I have not seen this in the last 6 months.

I am quite practised in restoring from backups because of btrfs :)

BillK

Re: [gentoo-user] experience thus far

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 4146 bytes --]

Am Sat, 17 May 2014 08:08:17 +0800
schrieb William Kenworthy <billk@iinet.net.au>:

> On 17/05/14 04:15, Marc Joliet wrote:
> > So, a week has passed since my conversion to btrfs.
> > 
> > So far there seem to have been no problems, my system has been running as if
> > nothing has changed :) . Which, as a friend pointed out, is how it should be.
> > 
> > I don't think there is anything particularly interesting to mention in addition
> > to what I already wrote. I can just say that I think the effort was worth it.
> > 
> > The one thing that I can tell from reading the past two weeks of the btrfs ML
> > is that the 3.15 Linux kernel series will contain lots of bug fixes (for
> > example in balancing, error handling, and send/receive), and that I will want
> > to use that sooner rather than later. Of course, the severity of the problems
> > varies, and a lot are triggered under odd, or at least uncommon, circumstances.
> > Still, its worth paying attention to.
> > 
> > Also, a lot of problem reports I saw came from people using other volume
> > management below btrfs, interestingly enough.
> > 
> > As for the future, I think I will wait a while, and get some experience with
> > btrfs first.  I suspect that by the time btrfs supports swap files, it will be
> > stable enough that I would consider converting my SSD to also use btrfs
> > anyway :) . Possibly before that, once I am fully convinced of btrfs'
> > stability, I will also convert my backup drive and switch to using snapshots
> > and send/receive to perform backups. Perhaps somebody will have written a
> > backup solution on top of snapshots by then.
> > 
> > Have a nice weekend,
> > 
> 
> Don't forget to have a maintenance program - run a scrub regularly once
> a week or so - I have enough btrfs drives (22 qemu files, 4 WD Greens
> att) to see about one or two scrub fixable errors a week with no obvious
> cause, sometimes serious (in a critical file).  My experience is that if
> you ignore these errors they seem to increase over time resulting in a
> crash and burn.  Keep an eye on your logs as btrfs will list the errors
> there as well ("grep -i btrfs /var/log/messages").  For the ones scrub
> cant fix, delete the file and restore from backup.  Errors that require
> off-line fixing (btrfsck) are the ones where I have lost file systems -
> though I have not seen this in the last 6 months.

I did not forget about scrubbing, though so far I have run them manually (once
on Monday after a weekend away from the computer, and once tonight, both
without error). Nevertheless, thanks for the reminder and extra info :) .

BTW: I came across an interesting tool called dstat (indirectly while looking
for which package contained iostat, which was mentioned on the btrfs ML). With
"dstat -df", you can monitor the I/O of each individual drive. It's fun
watching them be used in parallel :) .

Anyway, with dstat I discovered that my drives have noticeably different
throughput. Of course, I might have deduced that earlier:

# btrfs scrub status -d /home 
scrub status for 472c9290-3ff2-4096-9c47-0612d3a52cef
scrub device /dev/sda (id 1) history
	scrub started at Sat May 17 00:23:33 2014 and finished after 2536 seconds
	total bytes scrubbed: 215.42GiB with 0 errors
scrub device /dev/sdb (id 2) history
	scrub started at Sat May 17 00:23:33 2014 and finished after 3519 seconds
	total bytes scrubbed: 216.32GiB with 0 errors
scrub device /dev/sdc (id 3) history
	scrub started at Sat May 17 00:23:33 2014 and finished after 2346 seconds
	total bytes scrubbed: 216.57GiB with 0 errors
scrub device /dev/sdd (id 4) history
	scrub started at Sat May 17 00:23:33 2014 and finished after 2346 seconds
	total bytes scrubbed: 215.68GiB with 0 errors

Boy, is sdb slow! I might replace it with sde, which is laying around as a
spare for now, and make sdb the spare instead.

> I am quite practised in restoring from backups because of btrfs :)

Haha :) .

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-user] experience thus far

[William Kenworthy]

On 17/05/14 08:08, William Kenworthy wrote:
> On 17/05/14 04:15, Marc Joliet wrote:
>> So, a week has passed since my conversion to btrfs.
>>
>...
>>
>> Have a nice weekend,
>>
> 
> Don't forget to have a maintenance program - run a scrub regularly once
> a week or so - I have enough btrfs drives (22 qemu files, 4 WD Greens
> att) to see about one or two scrub fixable errors a week with no obvious
> cause, sometimes serious (in a critical file).  My experience is that if
> you ignore these errors they seem to increase over time resulting in a
> crash and burn.  Keep an eye on your logs as btrfs will list the errors
> there as well ("grep -i btrfs /var/log/messages").  For the ones scrub
> cant fix, delete the file and restore from backup.  Errors that require
> off-line fixing (btrfsck) are the ones where I have lost file systems -
> though I have not seen this in the last 6 months.
> 
> I am quite practised in restoring from backups because of btrfs :)
> 
> BillK
> 
> 
> 
This is from this mornings grep of the log:
May 15 07:00:34 myth kernel: btrfs: checksum error at logical 1775247360
on dev /dev/vda3, sector 5580816, root 5, inode 6423718, offset 1839104,
length 4096, links 1 (path:
var/log/mythtv/old/mythbackend.20140421061150.6275.log-20140515)
May 15 07:00:34 myth kernel: btrfs: bdev /dev/vda3 errs: wr 0, rd 0,
flush 0, corrupt 1, gen 0
May 15 07:00:34 myth kernel: btrfs: unable to fixup (regular) error at
logical 1775247360 on dev /dev/vda3
May 15 07:41:40 myth kernel: btrfs: bdev /dev/vda3 errs: wr 0, rd 0,
flush 0, corrupt 1, gen 0

and

May 16 20:40:33 moriah kernel: btrfs: bdev /dev/mapper/vg1-backups errs:
wr 0, rd 250, flush 0, corrupt 13, gen 0

sigh ....

BillK

Re: [gentoo-user] experience thus far

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 2152 bytes --]

On Saturday 17 May 2014 04:02:35 William Kenworthy wrote:
> On 17/05/14 08:08, William Kenworthy wrote:
> > On 17/05/14 04:15, Marc Joliet wrote:
> >> So, a week has passed since my conversion to btrfs.
> >
> >...
> >
> >> Have a nice weekend,
> > 
> > Don't forget to have a maintenance program - run a scrub regularly once
> > a week or so - I have enough btrfs drives (22 qemu files, 4 WD Greens
> > att) to see about one or two scrub fixable errors a week with no obvious
> > cause, sometimes serious (in a critical file).  My experience is that if
> > you ignore these errors they seem to increase over time resulting in a
> > crash and burn.  Keep an eye on your logs as btrfs will list the errors
> > there as well ("grep -i btrfs /var/log/messages").  For the ones scrub
> > cant fix, delete the file and restore from backup.  Errors that require
> > off-line fixing (btrfsck) are the ones where I have lost file systems -
> > though I have not seen this in the last 6 months.
> > 
> > I am quite practised in restoring from backups because of btrfs :)
> > 
> > BillK
> 
> This is from this mornings grep of the log:
> May 15 07:00:34 myth kernel: btrfs: checksum error at logical 1775247360
> on dev /dev/vda3, sector 5580816, root 5, inode 6423718, offset 1839104,
> length 4096, links 1 (path:
> var/log/mythtv/old/mythbackend.20140421061150.6275.log-20140515)
> May 15 07:00:34 myth kernel: btrfs: bdev /dev/vda3 errs: wr 0, rd 0,
> flush 0, corrupt 1, gen 0
> May 15 07:00:34 myth kernel: btrfs: unable to fixup (regular) error at
> logical 1775247360 on dev /dev/vda3
> May 15 07:41:40 myth kernel: btrfs: bdev /dev/vda3 errs: wr 0, rd 0,
> flush 0, corrupt 1, gen 0
> 
> and
> 
> May 16 20:40:33 moriah kernel: btrfs: bdev /dev/mapper/vg1-backups errs:
> wr 0, rd 250, flush 0, corrupt 13, gen 0

Thank you all for sharing your experiences with btrfs.  It looks like it will 
be come the fs of choice in the future.

I am not clear on one thing:  is the corruption that you show above *because* 
of btrfs, or it would occur silently with any other fs, like e.g. ext4?

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] experience thus far

[Rich Freeman]

On Sat, May 17, 2014 at 3:53 AM, Mick <michaelkintzios@gmail.com> wrote:
> I am not clear on one thing:  is the corruption that you show above *because*
> of btrfs, or it would occur silently with any other fs, like e.g. ext4?

That is something I'm curious about as well as I stumbled on this
thread.  I've been running btrfs on a 5 drive array set to raid1 for
both data and metadata for several months now, and I've yet to see a
single error in my weekly scrubs.  This is on a system that is up
24x7, running mysql, mythtv, postfix, and a daily rsync backup -
basically light disk activity at all times, and heavy activity
moderately often.  The only issue I've had with btrfs is ENOSPC when
it manages to allocate all of its chunks (more of a problem on a
smaller ssd running btrfs for /), and panics when I try to remove
several snapshots at once.

I'm not sure how easy it would be to test for silent corruption on
another fs, unless you tried using ZFS instead, or used tripwire or
some other integrity checker.  Testing the drive itself would be
straightforward if you didn't need to use it in any kind of production
capacity - write patterns to it and try to read them back in a few
days.

Rich

Re: [gentoo-user] experience thus far

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Sat, 17 May 2014 08:08:17 +0800, William Kenworthy wrote:

> Don't forget to have a maintenance program - run a scrub regularly once
> a week or so - I have enough btrfs drives (22 qemu files, 4 WD Greens
> att) to see about one or two scrub fixable errors a week with no obvious
> cause, sometimes serious (in a critical file).

That's a bit scary. I'm running ZFS on several systems, scrub twice a
week and have never seen any corruption. Do you think this is because of
btrfs, your hardware or your usage?


-- 
Neil Bothwick

Experience is what you get when you didn't get what you wanted.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] experience thus far

[William Kenworthy]

On 17/05/14 18:07, Neil Bothwick wrote:
> On Sat, 17 May 2014 08:08:17 +0800, William Kenworthy wrote:
> 
>> Don't forget to have a maintenance program - run a scrub regularly once
>> a week or so - I have enough btrfs drives (22 qemu files, 4 WD Greens
>> att) to see about one or two scrub fixable errors a week with no obvious
>> cause, sometimes serious (in a critical file).
> 
> That's a bit scary. I'm running ZFS on several systems, scrub twice a
> week and have never seen any corruption. Do you think this is because of
> btrfs, your hardware or your usage?
> 
> 

Usage and bugs in btrfs - the backup drive is using dirvish which
hammers the drive quite hard as its uses hardlinks for duplicate files
(faster, saves space).  Reiserfs which I used to use (same hardware)
only ever got an error whih a cause like power outage in the middle of a
backup - easily recovered if time consuming.  Its also on an LVM running
across a mixture of drives raging from an old ide to WD green.  Part of
the reason for converting it to btrfs is because if it survives the
hammering (and its getting better), its a sign its getting robust.  I
did try ext 4 (twice) before reiserfs - both attempts lasted around a
week.  All this is on roughly the same hardware - Ive had a couple of
disks fail, and move partitions around on the LVM as necessary.  The
motherboard is an old core2 but running 32bit.  Kernels are all 3.12.13
gentoo-sources.

For the VM's, they were still getting an occasional error when I had
them on a reiserfs  storage.  I tried ceph on btrfs for awhile but not
having enough hardware to do it properly (mostly btrfs failures it must
be said) its now on a 3 disk btrfs raid1.  Since creating that one just
a few weeks ago I have not seen an error, though btrfs VM's stored on it
have had errors.  The VM's 10-12 gentoo, 3xwin (usually 6 running, not
all 15 or so running at the same time) use a supermicro motherboard,
dual quad zeons and 16G ram.

BillK