[gentoo-user] Organising btrfs subvolumes

[gentoo-user] Organising btrfs subvolumes

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

I'm working on this btrfs malarkey and have a question about best
practice. It is recommended to leave the root volume empty and create a
subvolume for the root filesystem which is set with btrfs subvolume
set-default, which I have done.

What is the recommended way to create subvolumes that are mounted further
down the filesystem? Let's say I was usr and var on their own subvolumes.
Do I create them in the btrfs root, which means they have to be mounted
from /etc/fstab - or do I create hem below the subvolume called root?

That raises another question. Assuming I've done it wrong (well, my wife
always does) is there an equivalent to the zfs rename command to move or
rename a subvolume?


-- 
Neil Bothwick

Unsupported service (adj): Broken (see Demon)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Organising btrfs subvolumes

[Stefan G. Weichinger]

Am 22.05.2014 18:12, schrieb Neil Bothwick:
> I'm working on this btrfs malarkey and have a question about best 
> practice. It is recommended to leave the root volume empty and
> create a subvolume for the root filesystem which is set with btrfs
> subvolume set-default, which I have done.

Alternative: mount the subvol via option "subvolid" etc in fstab .... if
you plan to mount different snapshots, for example.

> What is the recommended way to create subvolumes that are mounted
> further down the filesystem? Let's say I was usr and var on their
> own subvolumes. Do I create them in the btrfs root, which means
> they have to be mounted from /etc/fstab - or do I create hem below
> the subvolume called root?


I saw more examples mounting every dir via a
separate line in fstab (which also adds the choice to mount them with
different options, think compression etc).

My understanding is:

* create and use subvols for entities you want to be able to snapshot
and rollback individually.

* create and use subvols for entities you want to apply special
options to: compression, (no)COW, quota ...

I would mount each subvol via extra line and create them in parallel ...

> That raises another question. Assuming I've done it wrong (well, my
> wife always does) is there an equivalent to the zfs rename command
> to move or rename a subvolume?

As far as I understand you are allowed to mount the root volume (or
academic: any subvol in a higher level) and use plain "mv" to rename
the subvols as if you renamed sub-dirs.

Stefan

Re: [gentoo-user] Organising btrfs subvolumes

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]

On Fri, 23 May 2014 00:34:25 +0200, Stefan G. Weichinger wrote:

> > I'm working on this btrfs malarkey and have a question about best 
> > practice. It is recommended to leave the root volume empty and
> > create a subvolume for the root filesystem which is set with btrfs
> > subvolume set-default, which I have done.  
> 
> Alternative: mount the subvol via option "subvolid" etc in fstab .... if
> you plan to mount different snapshots, for example.

I went with set-default for the root subvolume, if I need the root volume
I can mount it with subvolid=0.

> > What is the recommended way to create subvolumes that are mounted
> > further down the filesystem? Let's say I was usr and var on their
> > own subvolumes. Do I create them in the btrfs root, which means
> > they have to be mounted from /etc/fstab - or do I create hem below
> > the subvolume called root?  
> 
> I saw more examples mounting every dir via a
> separate line in fstab (which also adds the choice to mount them with
> different options, think compression etc).

That makes sense, and will be useful should we ever get encryption.

> My understanding is:
> 
> * create and use subvols for entities you want to be able to snapshot
> and rollback individually.
> 
> * create and use subvols for entities you want to apply special
> options to: compression, (no)COW, quota ...
> 
> I would mount each subvol via extra line and create them in parallel ...

That's what i ended up doing, thanks. I did have an issue with systemd
failing to mount them because the correct symlinks hadn't been created
when I run cryptsetup in the initramfs, because it doesn't use udev, but
that was fairly easy to fix.

> > That raises another question. Assuming I've done it wrong (well, my
> > wife always does) is there an equivalent to the zfs rename command
> > to move or rename a subvolume?  
> 
> As far as I understand you are allowed to mount the root volume (or
> academic: any subvol in a higher level) and use plain "mv" to rename
> the subvols as if you renamed sub-dirs.

rust me to overlook the easy way of doing things, I was looking for an
equivalent to "zfs rename" and never considered mv. So far, btrfs looks
good on my laptop - time to think about putting it on my desktop.


-- 
Neil Bothwick

Ifyoucanreadthis,youspendtoomuchtimefiguringouttaglines.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Organising btrfs subvolumes

[Stefan G. Weichinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 27.05.2014 09:59, schrieb Neil Bothwick:

>> Alternative: mount the subvol via option "subvolid" etc in fstab
>> .... if you plan to mount different snapshots, for example.
> 
> I went with set-default for the root subvolume, if I need the root
> volume I can mount it with subvolid=0.

Yes, just a question of preference.

I prefer to specify a subvolid ... this won't lead to side-effects if
I ever might set-default to something else.

But it's easier to use the defaul subvolid if you refer to that subvol
via grub2 or dracut etc ...

Something I still learn about ...

>> As far as I understand you are allowed to mount the root volume
>> (or academic: any subvol in a higher level) and use plain "mv" to
>> rename the subvols as if you renamed sub-dirs.
> 
> rust me to overlook the easy way of doing things, I was looking for
> an equivalent to "zfs rename" and never considered mv.

It feels somehow "wrong" to only mv them, right? ;-)

> So far, btrfs looks good on my laptop - time to think about putting
> it on my desktop.

Yeah, good luck with that. I am quite happy with btrfs so far ... no
problems or disadvantages so far.

And the hourly snapshots of / and /home on my desktop are really nice
to have ;-)

Stefan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJThG+2AAoJEClcuD1V0PzmqXsP/jkK/dYlatk90CbXwc/H71Y3
VPuT8y0P8zrsrMdwziG8BzvL5ivL6xTEMmxXmi5frWHgC9751UmqZVlCI6I2+dYQ
ffjZYLqf3Q6FXKIcZ09Sbuxg+FsuEqw2LLrWXmSgWDrf5YQ0GAOjZupL4ii8dtUt
heBGUXWHMdR6JQOgqAl5+g+f1HlDEnVDf49L5AbKE5wkeCirNPZzx4I54k8SNykG
N7TLUe+AGzCsDAOL8BWpLrAyCyuWkMc01sfZN0sp36wzt4IsMxQYQFxq1ndmtEWf
LneGfQzAalXY26YnKvpAczyeTLcwdWw5TK0Yd8ftvnJz8Q++h4KQOAzqyaVBlMKi
oYis1WugSeKOG5JebOd5vNYQ/Po4NaGmidLkvTARFf7lSyLdND+dOQRkCBuzN2Q0
i/G+C5OFVSI8Mp+7qmHfWIOaKdi36Zi8luKV937gJD2+y/oS0tJiOWICSo4bqAIh
5/yaSWg3rUfxQ0aYzQG8U055CpcvhfZntqt0HKn4/NOuw42T4/r/nFplIhB1/Qgz
qIGmkvCupUNRrpSfY+CjbNKMeO4LDpNpruU2qThwYBPMWQ/m3Iq1zEAkR4vYNetA
3oFdqEZmJsOI/e/uTBfpSTGsk9bP9y9rEIEjzmjTXEN8OjwfpMoB/Dn9C0J9uK25
MdA7u6Xx0JfbJ1qOXNQl
=b71z
-----END PGP SIGNATURE-----

Re: [gentoo-user] Organising btrfs subvolumes

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1591 bytes --]

On Tuesday 27 May 2014 11:57:58 Stefan G. Weichinger wrote:
> Am 27.05.2014 09:59, schrieb Neil Bothwick:
> >> Alternative: mount the subvol via option "subvolid" etc in fstab
> >> .... if you plan to mount different snapshots, for example.
> > 
> > I went with set-default for the root subvolume, if I need the root
> > volume I can mount it with subvolid=0.
> 
> Yes, just a question of preference.
> 
> I prefer to specify a subvolid ... this won't lead to side-effects if
> I ever might set-default to something else.
> 
> But it's easier to use the defaul subvolid if you refer to that subvol
> via grub2 or dracut etc ...
> 
> Something I still learn about ...
> 
> >> As far as I understand you are allowed to mount the root volume
> >> (or academic: any subvol in a higher level) and use plain "mv" to
> >> rename the subvols as if you renamed sub-dirs.
> > 
> > rust me to overlook the easy way of doing things, I was looking for
> > an equivalent to "zfs rename" and never considered mv.
> 
> It feels somehow "wrong" to only mv them, right? ;-)
> 
> > So far, btrfs looks good on my laptop - time to think about putting
> > it on my desktop.
> 
> Yeah, good luck with that. I am quite happy with btrfs so far ... no
> problems or disadvantages so far.
> 
> And the hourly snapshots of / and /home on my desktop are really nice
> to have ;-)
> 
> Stefan

I recall that zfs needed a lot of RAM >= 8M, is it the same with BTRFS?

Also how big is each snapshot of / and why are these necessary on an hourly 
basiszfs ?

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Organising btrfs subvolumes

[Rich Freeman]

On Tue, May 27, 2014 at 7:25 AM, Mick <michaelkintzios@gmail.com> wrote:
>
> Also how big is each snapshot of / and why are these necessary on an hourly
> basiszfs ?

Btrfs is COW, so snapshots only consume space as files change.  If you
have a read-only filesystem and snapshot it hourly the only space
consumed by a snapshot will be a few metadata records.

Snapshotting hourly would mostly be a convenience - in theory it
should get you time-machine-like functionality just like hourly
backups would, but with far less overhead and space use.

In practice I stopped doing this, as btrfs can misbehave when you
start getting a lot of snapshots accumulated (we're talking
thousands).  It probably doesn't help that I have VM images
snapshotted (though these images have fairly low write volumes - the
most active one does most of its writing to an nfs volume so only OS
updates, logs, etc change the VM).  When snapper would go to cleanup
snapshots I'd get panics.  I ended up having to write a script that
deleted one snapshot every 30min over the course of days to clean up
from that.  Now I only manually snapshot periodically and I haven't
had a problem with it.

I suspect that as with many things btrfs-related that it will be
worked out in time, though snapshots will always cause fragmentation
as long as the filesystem does partial diffs.

Rich

Re: [gentoo-user] Organising btrfs subvolumes

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

On Tue, 27 May 2014 07:38:01 -0400, Rich Freeman wrote:

> Snapshotting hourly would mostly be a convenience - in theory it
> should get you time-machine-like functionality just like hourly
> backups would, but with far less overhead and space use.
> 
> In practice I stopped doing this, as btrfs can misbehave when you
> start getting a lot of snapshots accumulated (we're talking
> thousands). 

I've read that too. I'm looking at doing a kind of round robin thing
where a script makes a snapshot and then deletes the oldest one.


-- 
Neil Bothwick

... "I'm simply not a nice girl", she whispered tartly.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Organising btrfs subvolumes

[Stefan G. Weichinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 27.05.2014 13:25, schrieb Mick:

> I recall that zfs needed a lot of RAM >= 8M, is it the same with
> BTRFS?

I assume you mean ">8GB" ?

As far as I know and researched: no, btrfs is less memory hungry and
was designed to even work fine on small devices like phones or so.

It depends if you use features like deduplication which is very
ressource-intensive ...

> Also how big is each snapshot of / and why are these necessary on
> an hourly basiszfs ?


Snapshots don't have any size initially.

With filesystems like btrfs and zfs a snapshot is more of a pointer to
a specific status of the whole fs-tree in time and in consequence also
happens instantly. The size of the snapshot ... oh, Rich already
replied as well :-)

Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJThHlUAAoJEClcuD1V0PzmApcP/RCBeASWCqKOxDhtyJ7Pj+c3
LgDObKZ23svRCU2qiwF04e1enJTUF+FcaL/KyB4690x1EATo+0ciYOajzgIRng16
24bGB/ZCRhE4uZ99aVLymAslhBXUyILcZ+AqaJCM2XL72Ttp7gqB/PW0bJWOgQ0Y
wC4FWtzDJN6vm2VfC3gDMhdUaoysWuCEQqhppJe5RpSbOP3kk9PzXiFLPpJkKs2V
xglvGDy5SCu79APTqUrDRZtHCQBpLTkMDBQr9ytdtVLPDDx6WyG0BZ351tp2fzMp
CMUy4T1z417l+TgDMZ2iYBh1+Ctnqkr+SfLdlkkF+0AJfpFUpNjztDHyhV8Muzan
AQdEDW1ccXc/cSR62gbf8+Y+Aj2QTBPImXtHNHQHPRiszVgJM7E5ufnxrUm+1qT9
aWM+SSLbDjtvmRBSOExNawonprMT44Vhqc4j0UfLhbLsLZfVbUghfczqOr5pDD6l
32dIkXRtT/zkR/tDpWX5n6ZotIlnAuVh7xIe8zID3KTLXdcuM2sP3UDFUGd4FqPU
5ovfAIhi133BHpdz7FHmTbR9if7iGeF+mWFfCt/cWYQY9vw3yLmUaJg3pxP0gKvC
C3SXt8V30ubWKglrCXd0a3eqm0fbonFaACdB3eyeQgi5S4FBYHIATW8XDGP+/VSG
t81Av1TKapRTzHgQF35U
=lMUl
-----END PGP SIGNATURE-----

Re: [gentoo-user] Organising btrfs subvolumes

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 352 bytes --]

On Tue, 27 May 2014 12:25:48 +0100, Mick wrote:

> I recall that zfs needed a lot of RAM >= 8M, is it the same with BTRFS?

If you means 8GB, it doesn't. I am/was using it on several systems with
4GB. you can control the amount of memory used for its caches.


-- 
Neil Bothwick

If you consult enough experts, you can confirm any opinion.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Organising btrfs subvolumes

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

On Tue, 27 May 2014 12:57:58 +0200, Stefan G. Weichinger wrote:

> > Trust me to overlook the easy way of doing things, I was looking for
> > an equivalent to "zfs rename" and never considered mv.  
> 
> It feels somehow "wrong" to only mv them, right? ;-)

It's just too easy, there must be a catch :)

> > So far, btrfs looks good on my laptop - time to think about putting
> > it on my desktop.  
> 
> Yeah, good luck with that. I am quite happy with btrfs so far ... no
> problems or disadvantages so far.
> 
> And the hourly snapshots of / and /home on my desktop are really nice
> to have ;-)

I have zfs-snapshot making snapshots at 15 minute, hourly, daily, monthly
and weekly intervals - and it cleans up after itself. There isn't
anything quite like that for btrfs, so I'm knocking up a python script to
take care of it. I want automated snapshots before I risk it on my
desktop.


-- 
Neil Bothwick

Do you steal taglines too?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Organising btrfs subvolumes

[Stefan G. Weichinger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 27.05.2014 13:49, schrieb Neil Bothwick:

> I have zfs-snapshot making snapshots at 15 minute, hourly, daily,
> monthly and weekly intervals - and it cleans up after itself. There
> isn't anything quite like that for btrfs, so I'm knocking up a
> python script to take care of it. I want automated snapshots before
> I risk it on my desktop.

Oh, I have something like that.

Copied here:

http://marc.merlins.org/perso/btrfs/post_2014-03-21_Btrfs-Tips_-How-To-Setup-Netapp-Style-Snapshots.html

I wrapped some helper stuff around it to make it work with 2
btrfs-pools and systemd (and a timer there to even avoid running cron).

Stefan

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJThHx8AAoJEClcuD1V0Pzmnj4QAKcZIghxoDgVVbI7a1rIltxl
fAPdRvY5g5We0v/VJ7RL9vJe1tJ9F2UnzDBC/rEPw8XSPR1zJxhrFyCrdmumMYY+
qac4XSRCSS92subHwnYfjWxigqLk7DT86ATTEIo841YmQTdrfRyzmxwVHwvoONOB
APasAal8Re7PFd01UAGpCurAzA20lMs1CFlqK/sBcWwM9RGsRzPwUZsfLiKcmdmX
Ta4YUO/RtOv/RWntyHYJZ0FgcxmNo+HLCE6821mOxg+CyDp9Vag/6HWlXbZW1MRM
/CVHQ8kim+azcEuX0NU7WDIfotPrXtb8PGZt+2QJtJm12CJQficQrUOZpmgoi3yM
7n1ux1VkM7avnXkoJ90/T5eWznq8d/NqTaESUHLnOPueE7JciR5gc3kNEgcriD7Y
qHM9ZDG4vzRScUcLowZydsK32FHZdFRHUpNkwE2Admffz7+Sv3TfZqog2rs0WVoM
zVBoKr3DY1LQo7NvHQJT2sfMKzBJq0kFoBY7yvwnJkzf2htlnhT2jOFg+XzTC0Hr
IEHzEu/yzfmwKp490U3+jAfCxEnaGNIW1YZ+lA+kzS+6+9DWR+XYuaV/EV/wvE9s
J2oPqzwzpcaBLxLY+Mq8cz/sXOASf6ICYYy1w+GfH82e/DYzfn/j+FTIxmjK6KVe
t6/tx2rwGeNL1AhKaSmY
=QHMO
-----END PGP SIGNATURE-----

Re: [gentoo-user] Organising btrfs subvolumes

[Rich Freeman]

On Tue, May 27, 2014 at 7:49 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> I have zfs-snapshot making snapshots at 15 minute, hourly, daily, monthly
> and weekly intervals - and it cleans up after itself. There isn't
> anything quite like that for btrfs, so I'm knocking up a python script to
> take care of it. I want automated snapshots before I risk it on my
> desktop.

There is snapper, which is even in the tree now.  It isn't 100%
flexible but supports any number of hourly, daily, monthly, and yearly
snapshots, with retention policies for each.

My problem was that the snapshots were created hourly, but cleaned up
daily, which meant 24 deleted in parallel at a time.

In general I've tended to notice that btrfs tends to suffer from
hurry-up-and-wait issues.  It will happily accept a ton of writes
(even from an ioniced process) which I imagine go into the log, and
then the whole filesystem will come to a halt at the next checkpoint
(every 30s by default).  It makes ionice just about useless, since the
filesystem accepts more data than it can handle, and then blocks even
realtime-priority processes while it is catching up.

I suspect that it was having a related issue with snapshot removals.
24 huge snapshot removal commands complete in almost zero time, and
then in 30s the debt comes due.

In order to be a bit more ionice-friendly the filesystem needs to
figure out what it can sustain and throttle writes to a reasonable
rate.  I'm fine with having some allowance for bursting, but having
all disk access block for 10-20 seconds isn't acceptable.

Oh, and chromium just loves its disk cache - it will happily wait for
20 seconds to read a cache entry that it could have downloaded online
in less than a second.

Rich

Re: [gentoo-user] Organising btrfs subvolumes

[Stefan G. Weichinger]

Am 27.05.2014 14:12, schrieb Rich Freeman:

> There is snapper, which is even in the tree now.  It isn't 100%
> flexible but supports any number of hourly, daily, monthly, and yearly
> snapshots, with retention policies for each.

no systemd-unitfiles yet, correct? I merged it and took a quick look, as
far as I understand it needs a service running ...

I will check back later this evening and see how to get that working here.

Stefan

Re: [gentoo-user] Organising btrfs subvolumes

[Rich Freeman]

On Tue, May 27, 2014 at 8:58 AM, Stefan G. Weichinger <lists@xunil.at> wrote:
> Am 27.05.2014 14:12, schrieb Rich Freeman:
>
>> There is snapper, which is even in the tree now.  It isn't 100%
>> flexible but supports any number of hourly, daily, monthly, and yearly
>> snapshots, with retention policies for each.
>
> no systemd-unitfiles yet, correct? I merged it and took a quick look, as
> far as I understand it needs a service running ...
>
> I will check back later this evening and see how to get that working here.

It uses crontab and portage hooks (though I don't know if the package
in portage installs hooks, and if you don't want hundreds of snapshots
you probably don't want them anyway).

Rich

Re: [gentoo-user] Organising btrfs subvolumes

[J. Roeleveld]

On Tuesday, May 27, 2014 12:57:58 PM Stefan G. Weichinger wrote:
> Am 27.05.2014 09:59, schrieb Neil Bothwick:
> > So far, btrfs looks good on my laptop - time to think about putting
> > it on my desktop.
> 
> Yeah, good luck with that. I am quite happy with btrfs so far ... no
> problems or disadvantages so far.
> 
> And the hourly snapshots of / and /home on my desktop are really nice
> to have ;-)

Hourly snapshots are nice, but I wonder how much need there is if the 
filesystem itself doesn't change very much.

I am still happily using LVM with snapshots. Those are instantaneous as well 
and I can then backup the snapshot, which on my server takes between 2 hours 
(incremental) and 3 weeks (full)
When a snapshot is backed up, it is removed.

The process to create the snapshots runs daily, but I could also configure it 
to run more often. This means that when I start a daily backup, the 
incrementals are piling up as snapshots.

With 15 different filesystems to backup, I didn't experience any issue with 
this.

I wonder how btrfs would deal with a situation like this?

--
Joost

Re: [gentoo-user] Organising btrfs subvolumes

[Rich Freeman]

On Tue, May 27, 2014 at 10:09 AM, J. Roeleveld <joost@antarean.org> wrote:
> I am still happily using LVM with snapshots. Those are instantaneous as well
> and I can then backup the snapshot, which on my server takes between 2 hours
> (incremental) and 3 weeks (full)
> When a snapshot is backed up, it is removed.
>
> The process to create the snapshots runs daily, but I could also configure it
> to run more often. This means that when I start a daily backup, the
> incrementals are piling up as snapshots.
>
> With 15 different filesystems to backup, I didn't experience any issue with
> this.
>
> I wonder how btrfs would deal with a situation like this?

btrfs wouldn't have any issues with this at all.  You'd have an
advantage in that you wouldn't have to unmount the filesystem to
cleanly create the snapshot (which you have to do with lvm).  If
you're concerned about application-level consistency you still need to
get applications to flush their writes/checkpoint/etc (which don't
have to be on disk, but they do have to be sent to the kernel).

If you want to get really crazy you could make use of btrfs send as
well - which is a filesystem-level function which tracks the actual
changes between snapshots.  Think of it like librsync with full file
comparisons (a very expensive mode that few use in practice) but it
doesn't need to actually read the files or have access to the
destination files to find the differences.  Doing this does require
keeping around a snapshot until all backups incrementally created
against it are done (if there are going to be any).

But, you can always just create a snapshot, write it to backup with
your favorite tool (it is just a directory tree), and then remove it
as soon as you're done with it.  Creating a snapshot is atomic at the
filesystem level, though again if you want application level
consistency you need to deal with that until somebody comes up with a
transactional way to store files on Linux that is more elegant that
fsyncing on every write.

Rich

[gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[J. Roeleveld]

On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote:
> On Tue, May 27, 2014 at 10:09 AM, J. Roeleveld <joost@antarean.org> wrote:
> > I am still happily using LVM with snapshots. Those are instantaneous as
> > well and I can then backup the snapshot, which on my server takes between
> > 2 hours (incremental) and 3 weeks (full)
> > When a snapshot is backed up, it is removed.
> > 
> > The process to create the snapshots runs daily, but I could also configure
> > it to run more often. This means that when I start a daily backup, the
> > incrementals are piling up as snapshots.
> > 
> > With 15 different filesystems to backup, I didn't experience any issue
> > with
> > this.
> > 
> > I wonder how btrfs would deal with a situation like this?
> 
> btrfs wouldn't have any issues with this at all.  You'd have an
> advantage in that you wouldn't have to unmount the filesystem to
> cleanly create the snapshot (which you have to do with lvm).

That, or a "sync" prior to creating the snapshot. :)

> If
> you're concerned about application-level consistency you still need to
> get applications to flush their writes/checkpoint/etc (which don't
> have to be on disk, but they do have to be sent to the kernel).

Application-level consistency, for some of the filesystems, means stopping the 
application, taking a backup of the database, creating a snapshot and then 
restarting the application.
For all the applications I run, the entire nightly process takes 2 minutes in 
total. During this time, services become temporarily unavailable.
This is acceptable.

> If you want to get really crazy you could make use of btrfs send as
> well - which is a filesystem-level function which tracks the actual
> changes between snapshots.  Think of it like librsync with full file
> comparisons (a very expensive mode that few use in practice) but it
> doesn't need to actually read the files or have access to the
> destination files to find the differences.  Doing this does require
> keeping around a snapshot until all backups incrementally created
> against it are done (if there are going to be any).

I have a yearly (full), monthly, weekly and daily. Each incremental is against 
the most recent one of itself or longer period.
That means having to keep multiple snapshots active, which I prefer to avoid.

But, it is a good idea for backing up desktops and laptops.

> But, you can always just create a snapshot, write it to backup with
> your favorite tool (it is just a directory tree), and then remove it
> as soon as you're done with it.  Creating a snapshot is atomic at the
> filesystem level, though again if you want application level
> consistency you need to deal with that until somebody comes up with a
> transactional way to store files on Linux that is more elegant that
> fsyncing on every write.

That would require a method to keep database and filesystem perfectly in sync 
when they are not necessarily on the same machine.

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[J. Roeleveld]

On Tuesday, May 27, 2014 05:12:50 PM J. Roeleveld wrote:
> On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote:
> > On Tue, May 27, 2014 at 10:09 AM, J. Roeleveld <joost@antarean.org> wrote:

<snipped....>

Forgot to add:

For fileservers, I am starting to feel that ZFS or BTRFS snapshots are easier 
to work with as it makes restoring files simpler.
Does anyone know how these will handle (and perform) with a possible 300+ 
snapshots per filesystem (or volume, as I think it's called)?

--
Joost

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Rich Freeman]

On Tue, May 27, 2014 at 11:21 AM, J. Roeleveld <joost@antarean.org> wrote:
> Does anyone know how these will handle (and perform) with a possible 300+
> snapshots per filesystem (or volume, as I think it's called)?

I can't speak for zfs.  I had upwards of 1000 snapshots on my system
before I stopped creating them hourly and and just started having
issues with it.

I wouldn't really say it is ready for prime time, but it is workable.
Of course, you'll still want backups - a million snapshots does you no
good if some bug wipes out your filesystem.  For one of my ENOSPC
incidents I ended up just wiping the entire filesystem and restoring
from backup, though if I kept at it I'd probably have been able to fix
it.

Oh, one other tip if you use btrfs - be sure you have a rescue disk
that supports it.  Hint, the old Gentoo install CD I had lying around
didn't.  You'll probably want to keep a rescue CD with a recent kernel
and btrfs-tools handy at all times.

Rich

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 630 bytes --]

On Tue, 27 May 2014 11:32:22 -0400, Rich Freeman wrote:

> Oh, one other tip if you use btrfs - be sure you have a rescue disk
> that supports it.  Hint, the old Gentoo install CD I had lying around
> didn't.  You'll probably want to keep a rescue CD with a recent kernel
> and btrfs-tools handy at all times.

I have a couple of System Rescue Cd ISOs in /boot and GRUB entries to boot
them. I have two because I wanted the latest for btrfs but couldn't be
bothered remastering for ZFS so i have an older, premastered image for
that.


-- 
Neil Bothwick

Mr. bullfrog says: "time's fun when you're having flies."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Joost Roeleveld]

On Tuesday 27 May 2014 11:32:22 Rich Freeman wrote:
> On Tue, May 27, 2014 at 11:21 AM, J. Roeleveld <joost@antarean.org> wrote:
> > Does anyone know how these will handle (and perform) with a possible 300+
> > snapshots per filesystem (or volume, as I think it's called)?
> 
> I can't speak for zfs.  I had upwards of 1000 snapshots on my system
> before I stopped creating them hourly and and just started having
> issues with it.

Ok, it can handle my backup schedule then. :)

> I wouldn't really say it is ready for prime time, but it is workable.
> Of course, you'll still want backups - a million snapshots does you no
> good if some bug wipes out your filesystem.  For one of my ENOSPC
> incidents I ended up just wiping the entire filesystem and restoring
> from backup, though if I kept at it I'd probably have been able to fix
> it.

Agreed, this is why the backup system would be adjusted for BTRFS on the 
fileserver, when I get round to it. I would probably keep snapshots active for 
the past 2 weeks.

> Oh, one other tip if you use btrfs - be sure you have a rescue disk
> that supports it.  Hint, the old Gentoo install CD I had lying around
> didn't.  You'll probably want to keep a rescue CD with a recent kernel
> and btrfs-tools handy at all times.

Always important. I just saw the other email which states that the latest 
sysresccd supports it. That is fine for me.

--
Joost

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

On Wed, 28 May 2014 12:03 +0200, Joost Roeleveld wrote:

> Always important. I just saw the other email which states that the
> latest sysresccd supports it. That is fine for me.

Sysresccd has supported btrfs for some time. I realise my mail could have
been read otherwise, but the reason for keeping a recent CD for btrfs is
that it is evolving and the recommendation is to always use the latest
kernel and userspace available.


-- 
Neil Bothwick

If you can smile when things go wrong, you have someone in mind to blame.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Rich Freeman]

On Tue, May 27, 2014 at 11:12 AM, J. Roeleveld <joost@antarean.org> wrote:
> On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote:
>> btrfs wouldn't have any issues with this at all.  You'd have an
>> advantage in that you wouldn't have to unmount the filesystem to
>> cleanly create the snapshot (which you have to do with lvm).
>
> That, or a "sync" prior to creating the snapshot. :)

If the filesystem is still mounted, I'm not sure that a sync is
guaranteed to give you a clean remount.  It only flushes the
caches/etc.  You need to remount read-only or unmount before doing the
sync (and the sync probably isn't actually necessary as I'd think LVM
would snapshot the contents of the cache as well).

> I have a yearly (full), monthly, weekly and daily. Each incremental is against
> the most recent one of itself or longer period.
> That means having to keep multiple snapshots active, which I prefer to avoid.

You only need to store snapshots for use with incremental backups.
So, if all your backups are full, then you don't need to retain any
snapshots (and you wouldn't use btrfs send anyway).  If your yearly is
full and your monthlies are incremental against the yearly then you
need to keep your yearly snapshot for a year.  If your yearly is full
and your monthlies are incremental against the last month, then you
only need to keep the yearly until the next monthly.  If your
monthlies are full then you only need to keep the current monthly
assuming your dailies are incremental against those, but if they're
incremental from the last daily then you never need to keep anything
for more than a day.

>
> But, it is a good idea for backing up desktops and laptops.

It is really intended more for something like datacenter replication.
Snapshot every 5 min, send the data to the backup datacenter, delete
the snapshots upon confirmation of successful receipt.  In such a
scenario you wouldn't retain the sent files but just keep playing them
against the replicate filesystem.

They'd be fine for backups as well, as long as you can store the
snapshots online until no longer needed for incrementals.

>
>> But, you can always just create a snapshot, write it to backup with
>> your favorite tool (it is just a directory tree), and then remove it
>> as soon as you're done with it.  Creating a snapshot is atomic at the
>> filesystem level, though again if you want application level
>> consistency you need to deal with that until somebody comes up with a
>> transactional way to store files on Linux that is more elegant that
>> fsyncing on every write.
>
> That would require a method to keep database and filesystem perfectly in sync
> when they are not necessarily on the same machine.
>

Well, right now we can't even guarantee consistency when everything is
written by a single process on the same machine.  The best we have is
a clunky fsync operation which kills the write cache and destroys
performance, and even that doesn't do anything if you have more than
one file that must be consistent.

The result is journals on top of journals as nobody can trust the next
layer down to do its job correctly.

Going across machines does complicate things further as there are more
failure modes that take out one part of the overall system but not
another.  However, I'd like to think that an OS that natively supports
transactions could at least standardize things so that every layer
along the path isn't storing its own journal.

In fact, many of the optimizations possible with zfs and btrfs are due
to the fact that you eliminate all those layers.

Rich

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Joost Roeleveld]

On Tuesday 27 May 2014 11:28:17 Rich Freeman wrote:
> On Tue, May 27, 2014 at 11:12 AM, J. Roeleveld <joost@antarean.org> wrote:
> > On Tuesday, May 27, 2014 10:31:26 AM Rich Freeman wrote:
> >> btrfs wouldn't have any issues with this at all.  You'd have an
> >> advantage in that you wouldn't have to unmount the filesystem to
> >> cleanly create the snapshot (which you have to do with lvm).
> > 
> > That, or a "sync" prior to creating the snapshot. :)
> 
> If the filesystem is still mounted, I'm not sure that a sync is
> guaranteed to give you a clean remount.  It only flushes the
> caches/etc.  You need to remount read-only or unmount before doing the
> sync (and the sync probably isn't actually necessary as I'd think LVM
> would snapshot the contents of the cache as well).

I do this for the OS-partitions of my VMs.
In the VM, run "sync", then on the host, take an LVM snapshot and then mount 
the snapshot on the host.
I have not had any errors from this.

> > I have a yearly (full), monthly, weekly and daily. Each incremental is
> > against the most recent one of itself or longer period.
> > That means having to keep multiple snapshots active, which I prefer to
> > avoid.
> You only need to store snapshots for use with incremental backups.
> So, if all your backups are full, then you don't need to retain any
> snapshots (and you wouldn't use btrfs send anyway).  If your yearly is
> full and your monthlies are incremental against the yearly then you
> need to keep your yearly snapshot for a year.  If your yearly is full
> and your monthlies are incremental against the last month, then you
> only need to keep the yearly until the next monthly.  If your
> monthlies are full then you only need to keep the current monthly
> assuming your dailies are incremental against those, but if they're
> incremental from the last daily then you never need to keep anything
> for more than a day.

That makes for an interesting option. Not sure if I would implement it that 
way.

> > But, it is a good idea for backing up desktops and laptops.
> 
> It is really intended more for something like datacenter replication.
> Snapshot every 5 min, send the data to the backup datacenter, delete
> the snapshots upon confirmation of successful receipt.  In such a
> scenario you wouldn't retain the sent files but just keep playing them
> against the replicate filesystem.
> 
> They'd be fine for backups as well, as long as you can store the
> snapshots online until no longer needed for incrementals.

"app-backup/dar" uses catalogues for the incrementals. I think I will stick to 
that for the foreseeable future.

> >> But, you can always just create a snapshot, write it to backup with
> >> your favorite tool (it is just a directory tree), and then remove it
> >> as soon as you're done with it.  Creating a snapshot is atomic at the
> >> filesystem level, though again if you want application level
> >> consistency you need to deal with that until somebody comes up with a
> >> transactional way to store files on Linux that is more elegant that
> >> fsyncing on every write.
> > 
> > That would require a method to keep database and filesystem perfectly in
> > sync when they are not necessarily on the same machine.
> 
> Well, right now we can't even guarantee consistency when everything is
> written by a single process on the same machine.  The best we have is
> a clunky fsync operation which kills the write cache and destroys
> performance, and even that doesn't do anything if you have more than
> one file that must be consistent.

Yep, and that's why those filesystems are actually umounted prior to creating 
the LVM snapshot. Umounting forces the filesystem to be put into a consistent 
state.

> The result is journals on top of journals as nobody can trust the next
> layer down to do its job correctly.
> 
> Going across machines does complicate things further as there are more
> failure modes that take out one part of the overall system but not
> another.  However, I'd like to think that an OS that natively supports
> transactions could at least standardize things so that every layer
> along the path isn't storing its own journal.
> 
> In fact, many of the optimizations possible with zfs and btrfs are due
> to the fact that you eliminate all those layers.

Either of those 2, probably btrfs as I prefer native support in the kernel, 
will be implemented when I get the opportunity to get the NAS on bare metal 
and remove the virtualization for that component. I need to find a different 
host for the other services first.
That might take a while.

--
Joost

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Rich Freeman]

On Wed, May 28, 2014 at 6:13 AM, Joost Roeleveld <joost@antarean.org> wrote:
> "app-backup/dar" uses catalogues for the incrementals. I think I will stick to
> that for the foreseeable future.
>

I used to use that and sarab (which is a wrapper).  I moved on to
duplicity.  The problem with dar is that it uses quite a bit of RAM as
the number of files being backed up grows I think.  So, if you have
6TB full of multimedia it might not be a huge problem, but if you have
6TB full of portage trees good luck with that.

The other problem with dar is that if a file changes it stores a
complete copy of it.  Duplicity uses librsync, so if a file changes it
only stores the parts that actually changed.  It also uses catalogs,
and supports things like caching catalogs (so you don't need the last
incremental mounted), and a bunch of storage backends (like S3).

However, dar definitely is more useful than tar if you want the option
for random access.

Rich

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Alan McKinnon]

On 27/05/2014 17:12, J. Roeleveld wrote:
> I have a yearly (full), monthly, weekly and daily. Each incremental is against 
> the most recent one of itself or longer period.
> That means having to keep multiple snapshots active, which I prefer to avoid.
> 
> But, it is a good idea for backing up desktops and laptops.


I'm curious why you have yearly snapshots. I've yet to find any sane
production system where a yearly backup had any worth at all. Even
monthly is pushing it...

Or do you do it to have a decent start point for incrementals?


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Joost Roeleveld]

On Tuesday 27 May 2014 23:35:26 Alan McKinnon wrote:
> On 27/05/2014 17:12, J. Roeleveld wrote:
> > I have a yearly (full), monthly, weekly and daily. Each incremental is
> > against the most recent one of itself or longer period.
> > That means having to keep multiple snapshots active, which I prefer to
> > avoid.
> > 
> > But, it is a good idea for backing up desktops and laptops.
> 
> I'm curious why you have yearly snapshots. I've yet to find any sane
> production system where a yearly backup had any worth at all. Even
> monthly is pushing it...
> 
> Or do you do it to have a decent start point for incrementals?

It's to have a decent start point for incrementals.
Below are the 2 biggest shares on the NAS:

/dev/xvda17             7.1T  5.9T  1.2T  84% /data/unsorted
/dev/xvda16             3.0T  2.4T  517G  83% /data/software

It is impossible to do a full backup on a daily or even weekly basis.

Previously, I had 1 full backup and then a daily incremental. This appears 
like a good idea, untill you need to restore the filesystem from backups when 
the crash occured 2 years later.
That is 1 full backup and over 700 incrementals....

Currently, I do the following:
Every year, a full backup
Then, every month, I have an incremental based on either the yearly or 
previous monthly.
Ditto for the weekly (but then based on monthly or weekly)
And again for the daily.

--
Joost

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Alan McKinnon]

On 28/05/2014 11:58, Joost Roeleveld wrote:
> On Tuesday 27 May 2014 23:35:26 Alan McKinnon wrote:
>> On 27/05/2014 17:12, J. Roeleveld wrote:
>>> I have a yearly (full), monthly, weekly and daily. Each incremental is
>>> against the most recent one of itself or longer period.
>>> That means having to keep multiple snapshots active, which I prefer to
>>> avoid.
>>>
>>> But, it is a good idea for backing up desktops and laptops.
>>
>> I'm curious why you have yearly snapshots. I've yet to find any sane
>> production system where a yearly backup had any worth at all. Even
>> monthly is pushing it...
>>
>> Or do you do it to have a decent start point for incrementals?
> 
> It's to have a decent start point for incrementals.
> Below are the 2 biggest shares on the NAS:
> 
> /dev/xvda17             7.1T  5.9T  1.2T  84% /data/unsorted
> /dev/xvda16             3.0T  2.4T  517G  83% /data/software
> 
> It is impossible to do a full backup on a daily or even weekly basis.
> 
> Previously, I had 1 full backup and then a daily incremental. This appears 
> like a good idea, untill you need to restore the filesystem from backups when 
> the crash occured 2 years later.
> That is 1 full backup and over 700 incrementals....
> 
> Currently, I do the following:
> Every year, a full backup
> Then, every month, I have an incremental based on either the yearly or 
> previous monthly.
> Ditto for the weekly (but then based on monthly or weekly)
> And again for the daily.



OK, that makes sense.

It reminds me of an issue my wife had with the data warehouse when she
worked at the bank. In a nutshell, they needed backups but backups were
impossible to achieve because physics says so. They needed to get data
off the disk 4 times faster than data comes off a disk - SCSI limits
being rather hard limits :-) That opinion didn't go down well when I
offered it

The solution was to do it much like your plan above.
With the benefit that the infrequent full backups would be done on a
fixed schedule in a change window with X hours downtime that was known
well in advance.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Joost Roeleveld]

On Wednesday 28 May 2014 13:07:49 Alan McKinnon wrote:
> On 28/05/2014 11:58, Joost Roeleveld wrote:
> > On Tuesday 27 May 2014 23:35:26 Alan McKinnon wrote:
> >> On 27/05/2014 17:12, J. Roeleveld wrote:
> >>> I have a yearly (full), monthly, weekly and daily. Each incremental is
> >>> against the most recent one of itself or longer period.
> >>> That means having to keep multiple snapshots active, which I prefer to
> >>> avoid.
> >>> 
> >>> But, it is a good idea for backing up desktops and laptops.
> >> 
> >> I'm curious why you have yearly snapshots. I've yet to find any sane
> >> production system where a yearly backup had any worth at all. Even
> >> monthly is pushing it...
> >> 
> >> Or do you do it to have a decent start point for incrementals?
> > 
> > It's to have a decent start point for incrementals.
> > Below are the 2 biggest shares on the NAS:
> > 
> > /dev/xvda17             7.1T  5.9T  1.2T  84% /data/unsorted
> > /dev/xvda16             3.0T  2.4T  517G  83% /data/software
> > 
> > It is impossible to do a full backup on a daily or even weekly basis.
> > 
> > Previously, I had 1 full backup and then a daily incremental. This appears
> > like a good idea, untill you need to restore the filesystem from backups
> > when the crash occured 2 years later.
> > That is 1 full backup and over 700 incrementals....
> > 
> > Currently, I do the following:
> > Every year, a full backup
> > Then, every month, I have an incremental based on either the yearly or
> > previous monthly.
> > Ditto for the weekly (but then based on monthly or weekly)
> > And again for the daily.
> 
> OK, that makes sense.
> 
> It reminds me of an issue my wife had with the data warehouse when she
> worked at the bank. In a nutshell, they needed backups but backups were
> impossible to achieve because physics says so. They needed to get data
> off the disk 4 times faster than data comes off a disk - SCSI limits
> being rather hard limits :-) That opinion didn't go down well when I
> offered it

Haha :)
I know the feeling.
I'd love to know the final solution they came up with.

> The solution was to do it much like your plan above.
> With the benefit that the infrequent full backups would be done on a
> fixed schedule in a change window with X hours downtime that was known
> well in advance.

Using snapshots, the downtime is the same couple of minutes each night.
The problem is that during the backup, the performance of the server is 
impacted. For a full backup, that means weeks...

--
Joost

Re: [gentoo-user] Backups and snapshots [Was: Organising btrfs subvolumes]

[Alan McKinnon]

On 28/05/2014 13:42, Joost Roeleveld wrote:
>>> Currently, I do the following:
>>> > > Every year, a full backup
>>> > > Then, every month, I have an incremental based on either the yearly or
>>> > > previous monthly.
>>> > > Ditto for the weekly (but then based on monthly or weekly)
>>> > > And again for the daily.
>> > 
>> > OK, that makes sense.
>> > 
>> > It reminds me of an issue my wife had with the data warehouse when she
>> > worked at the bank. In a nutshell, they needed backups but backups were
>> > impossible to achieve because physics says so. They needed to get data
>> > off the disk 4 times faster than data comes off a disk - SCSI limits
>> > being rather hard limits :-) That opinion didn't go down well when I
>> > offered it
> Haha :)
> I know the feeling.
> I'd love to know the final solution they came up with.

It was clever magic with LVM snapshots, but I'm not privy to the details
- it was a bank after all and there's only so much Mrs Alan and the
sysadmins could tell me.

But it went something like this:

Take a snapshot and copy that data to the SAN. This takes days or weeks
and it's only ever done once. Thereafter, snapshots only. The backup
system would take that last full + incrementals and create a new full
for it's own use, this process runs independent from everything else and
must take as long as it takes while the db carries on doing it's thing.
If two backup jobs start to overlap in time, one of them gets discarded.

Quite a clever scheme actually and it relies on storage being shared on
the SAN to work, plus some in-house magic to get the backup system to
recognise and use LVM snapshots natively. I believe performance impact
was kept to acceptable levels by cleverly setting priorities -
read/writes by the db take priority over backup reads which has to take
a back seat and just wait it's turn.


> 
>> > The solution was to do it much like your plan above.
>> > With the benefit that the infrequent full backups would be done on a
>> > fixed schedule in a change window with X hours downtime that was known
>> > well in advance.
> Using snapshots, the downtime is the same couple of minutes each night.
> The problem is that during the backup, the performance of the server is 
> impacted. For a full backup, that means weeks...


-- 
Alan McKinnon
alan.mckinnon@gmail.com