[gentoo-amd64] Quick rsync backup before changing a RAID

[gentoo-amd64] Quick rsync backup before changing a RAID

[Mark Knecht]

Hi,
   I need to change the RAID layout on my home server. This change
does _not_ involve the running RAID6 with Gentoo on it (at least at
the moment) but only a rather old, large and ill-placed second RAID6
that has some _very_ important Virtualbox VMs. Now, I have backup
scripts for protecting this stuff, and I use them every night, but
I've never had to do a restore so at this point I don't know whether
they really work. The files are there (checked by hand anyway) but who
knows if permissions or something else might cause a problem so I want
to be extra careful.

   Today, I created a new backup USB drive and want to do an rsync to
that drive as a second backup. I intend to restore from this drive
later today when the RAIDs are rebuilt and ready to go. I'd like some
ideas on best practices for using rsync to do this job.

   My current scripts use this basic format:

mark@c2RAID6 ~ $ cat ./DoBackups
mount /backups
rsync -avx /VirtualMachines/VMWare/. /backups/VMWare
rsync -avx /VirtualMachines/Virtualbox/. /backups/Virtualbox
rsync -avx /home/. /backups/home
cd
umount /backups
mark@c2RAID6 ~ $

   I'm wondering whether -avx is what others would use or whether
there's a better set of options.

   When the RAID comes back up I want to reverse and copy back from
USB drive to the /VirtualMachines directory.

Thanks in advance,
Mark

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Paul Hartman]

On Wed, Jun 19, 2013 at 1:59 PM, Mark Knecht <markknecht@gmail.com> wrote:
>    I'm wondering whether -avx is what others would use or whether
> there's a better set of options.

I think that is the typical rsync backup command set as far as I have
ever seen. It should work as you expect it to work. Note that if you
use ACLs or extended attributes, you would need to additionally
specify -A and/or -X since they are not implied by -a.

On a side note, I recently rsync'ed a drive between computers, one
running a Live CD and one not, over ssh, and somehow got all my
user/groups mixed up (I guess group 20 on one box was X but on another
box it was Y). Luckily I was able to identify it and fix it before
things got too broken. I'm not entirely sure what I did wrong, to be
honest.

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Mark Knecht]

On Wed, Jun 19, 2013 at 12:15 PM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Wed, Jun 19, 2013 at 1:59 PM, Mark Knecht <markknecht@gmail.com> wrote:
>>    I'm wondering whether -avx is what others would use or whether
>> there's a better set of options.
>
> I think that is the typical rsync backup command set as far as I have
> ever seen. It should work as you expect it to work. Note that if you
> use ACLs or extended attributes, you would need to additionally
> specify -A and/or -X since they are not implied by -a.
>
> On a side note, I recently rsync'ed a drive between computers, one
> running a Live CD and one not, over ssh, and somehow got all my
> user/groups mixed up (I guess group 20 on one box was X but on another
> box it was Y). Luckily I was able to identify it and fix it before
> things got too broken. I'm not entirely sure what I did wrong, to be
> honest.
>

Thanks for the reply Paul.

Is 'using ACLs' one of those things that if you don't know then you
are not? I just run basic Gentoo here. All installs done as per the
install guide. I don't even use LVM or anything fancy. (Which is
likely part of why I'm having to reconfigure as much as I am but so be
it.)

Anyway, thanks for the info. I'll just give it a try and then see
about maybe diffing the two drives or something like to to check.

Cheers,
Mark

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Rich Freeman]

On Wed, Jun 19, 2013 at 3:21 PM, Mark Knecht <markknecht@gmail.com> wrote:
>
> Is 'using ACLs' one of those things that if you don't know then you
> are not? I just run basic Gentoo here. All installs done as per the
> install guide. I don't even use LVM or anything fancy. (Which is
> likely part of why I'm having to reconfigure as much as I am but so be
> it.)

Gentoo can potentially make use of them if you're using POSIX
capabilities, but I think that has to be enabled by USE flag or
FEATURE.  I think you might also have to enable them in fstab (and the
kernel as well, though that is likely enabled by default).  Chances
are you'd know if you were using them, and it would only apply to
stuff installed by portage (generally binaries).  Oh, and using POSIX
capabilities is a good thing (less suid root binaries).

One thing to mention - I'm not sure how much data you're talking
about, but if it is a lot and downtime is a concern you could probably
get creative and avoid some copying.  You could copy the files onto an
LVM vg, and then after your RAID is set up you could move the logical
volume onto your RAID, which is an online operation.  (Ie set up your
USB drive as LVM, copy the files over, adjust your mounts so that the
USB drive is mounted where you want the files, and at this point
they're in production.  Then get your RAID set up, add it to the
volume group, and pvmove the USB drive so that it gets moved to your
RAID.  Then remove the USB drive from your volume group and you're in
the final config, with no further need to adjust mountpoints since
they're already pointed to LVM.)  I've done a lot of messing around
with online migrations of RAID and LVM - quite handy when you have
lots of stuff like mythtv recordings that you don't want to have
offline backups of.

Rich

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Mark Knecht]

On Wed, Jun 19, 2013 at 1:20 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Wed, Jun 19, 2013 at 3:21 PM, Mark Knecht <markknecht@gmail.com> wrote:
>>
>> Is 'using ACLs' one of those things that if you don't know then you
>> are not? I just run basic Gentoo here. All installs done as per the
>> install guide. I don't even use LVM or anything fancy. (Which is
>> likely part of why I'm having to reconfigure as much as I am but so be
>> it.)
>
> Gentoo can potentially make use of them if you're using POSIX
> capabilities, but I think that has to be enabled by USE flag or
> FEATURE.  I think you might also have to enable them in fstab (and the
> kernel as well, though that is likely enabled by default).  Chances
> are you'd know if you were using them, and it would only apply to
> stuff installed by portage (generally binaries).  Oh, and using POSIX
> capabilities is a good thing (less suid root binaries).
>
> One thing to mention - I'm not sure how much data you're talking
> about, but if it is a lot and downtime is a concern you could probably
> get creative and avoid some copying.  You could copy the files onto an
> LVM vg, and then after your RAID is set up you could move the logical
> volume onto your RAID, which is an online operation.  (Ie set up your
> USB drive as LVM, copy the files over, adjust your mounts so that the
> USB drive is mounted where you want the files, and at this point
> they're in production.  Then get your RAID set up, add it to the
> volume group, and pvmove the USB drive so that it gets moved to your
> RAID.  Then remove the USB drive from your volume group and you're in
> the final config, with no further need to adjust mountpoints since
> they're already pointed to LVM.)  I've done a lot of messing around
> with online migrations of RAID and LVM - quite handy when you have
> lots of stuff like mythtv recordings that you don't want to have
> offline backups of.
>
> Rich
>

Interesting info Rich. Thanks.

Approximately 250GB involved. Not worried about downtime. The data is
all Windows VMs which I use for stock & futures trading. Only need the
machine back up by tomorrow morning.

Also, I've never used LVM so for an afternoon project it's unlikely
I'd be successful going that direction I think. The second USB backup
drive is about 50% done so I should be able to start working on
reconfiguring the RAID in about an hour and hopefully be back up and
running by 5PM if things go well.

Cheers,
Mark

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Thanasis]

on 06/19/2013 10:15 PM Paul Hartman wrote the following:
> I recently rsync'ed a drive between computers, one
> running a Live CD and one not, over ssh, and somehow got all my
> user/groups mixed up 

You should have used the option --numeric-ids
(see rsync man page)

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Mark Knecht]

On Wed, Jun 19, 2013 at 12:37 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
> on 06/19/2013 10:15 PM Paul Hartman wrote the following:
>> I recently rsync'ed a drive between computers, one
>> running a Live CD and one not, over ssh, and somehow got all my
>> user/groups mixed up
>
> You should have used the option --numeric-ids
> (see rsync man page)
>

Ah, interesting option in Paul's case.

In my case, as I'm going back onto the same system, I think I won't need this.

Thanks,
Mark

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Paul Hartman]

On Wed, Jun 19, 2013 at 2:37 PM, Thanasis <thanasis@asyr.hopto.org> wrote:
> on 06/19/2013 10:15 PM Paul Hartman wrote the following:
>> I recently rsync'ed a drive between computers, one
>> running a Live CD and one not, over ssh, and somehow got all my
>> user/groups mixed up
>
> You should have used the option --numeric-ids
> (see rsync man page)

Ah-ha, that looks like it would have prevented it. I will try to
remember that for next time. Thank you very much for the tip.

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Panagiotis Christopoulos]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

On Wed, Jun 19, 2013 at 9:59 PM, Mark Knecht <markknecht@gmail.com> wrote:

> ...
>
    I'm wondering whether -avx is what others would use or whether

> there's a better set of options.
>
>    When the RAID comes back up I want to reverse and copy back from
> USB drive to the /VirtualMachines directory.
>
> Add --progress to track progress while copying. Also, be careful what
filesystem the USB drive will have (better to use what you're already
using), in order to preserve permissions correctly etc.

Panagiotis

[-- Attachment #2: Type: text/html, Size: 1032 bytes --]

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Mark Knecht]

On Wed, Jun 19, 2013 at 12:19 PM, Panagiotis Christopoulos
<pxrist@gmail.com> wrote:
> On Wed, Jun 19, 2013 at 9:59 PM, Mark Knecht <markknecht@gmail.com> wrote:
>>
>> ...
>
>     I'm wondering whether -avx is what others would use or whether
>>
>> there's a better set of options.
>>
>>    When the RAID comes back up I want to reverse and copy back from
>> USB drive to the /VirtualMachines directory.
>>
> Add --progress to track progress while copying. Also, be careful what
> filesystem the USB drive will have (better to use what you're already
> using), in order to preserve permissions correctly etc.
>
> Panagiotis

Hi Panagiotis,
   Good point about the file system type. Both are ext3 so no problems
expected, although on restore I suspect it will end up ext4.

Cheers,
Mark

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Benny Pedersen]

Mark Knecht skrev den 2013-06-19 20:59:

> mark@c2RAID6 ~ $ cat ./DoBackups
> mount /backups
> rsync -avx /VirtualMachines/VMWare/. /backups/VMWare

is not possible to make native vmware backup in blocklevel ?, or is 
file level backup needed ?

> rsync -avx /VirtualMachines/Virtualbox/. /backups/Virtualbox

same here, i know virtualbox can make blocklevel backup and its really 
more usefull if imho to make it that way

> rsync -avx /home/. /backups/home

this would make a tarball instaed ?

> cd
> umount /backups
> mark@c2RAID6 ~ $
>
> I'm wondering whether -avx is what others would use or whether
> there's a better set of options.

its possible to make filelevel backups to nas disks like qnap i have 
here with have rsync client/server, but if you more liked to migrade to 
another vmware/virtualbox then rsync is not the best option for backups

> When the RAID comes back up I want to reverse and copy back from
> USB drive to the /VirtualMachines directory.

will not create the virtualbox/vmware setup, if thats not needed then 
ok

-- 
senders that put my email into body content will deliver it to my own 
trashcan, so if you like to get reply, dont do it

Re: [gentoo-amd64] Quick rsync backup before changing a RAID

[Mark Knecht]

On Thu, Jun 20, 2013 at 6:44 AM, Benny Pedersen <me@junc.eu> wrote:
> Mark Knecht skrev den 2013-06-19 20:59:
>
>
>> mark@c2RAID6 ~ $ cat ./DoBackups
>> mount /backups
>> rsync -avx /VirtualMachines/VMWare/. /backups/VMWare
>
>
> is not possible to make native vmware backup in blocklevel ?, or is file
> level backup needed ?
>

Possibly, and it's sensible that Virtualbox & VMWare would provide
some mechanisms to do that, but then I'd have to back up a bunch of
different VMs using different tools, and that would only get the VMs
and not the other files on the old RAID.

<SNIP>
>> rsync -avx /home/. /backups/home
>
>
> this would make a tarball instaed ?
>

No, it's not a tarball, it's just a copy of the file system structure
that existed on the old RAID/

<SNIP>
>
> its possible to make filelevel backups to nas disks like qnap i have here
> with have rsync client/server, but if you more liked to migrade to another
> vmware/virtualbox then rsync is not the best option for backups
>

OK, I'm not sure I'm understanding you but I'm not trying to migrate
from one version fo VBox or VMWare to another. I'm only trying to
reduce the number of RAIDs on a single set of drives. The RAID6 that
has / is being enlarged and then the copied data will be added onto
that array in the same locations (/VirtualMachines) instead of
mounting a second RAID under /VirtualMachine. There is no migration
going on in terms of executable versions.

>
>> When the RAID comes back up I want to reverse and copy back from
>> USB drive to the /VirtualMachines directory.
>
>
> will not create the virtualbox/vmware setup, if thats not needed then ok

I guess I agree. It won't 'create' it because it's already there. In
fact I mounted the USB backup drive at /VirtualMachines and ran the
VMs from the USB drive as a test that it was working before I removed
the old RAID. I expect that when I rsync the USB backup back to
/VirtualMachines on the new, larger / RAID6 it will continue to work.

Thanks for the comments.

Cheers,
Mark