[gentoo-user] Clone live system as a simple backup?

[gentoo-user] Clone live system as a simple backup?

[Mark Knecht]

Hi,
   I'm interested in the idea of cloning a live, complicated hardware
system onto a single external hard drive as a simple backup. I would
like this external drive to be completely bootable. What's the best
way to approach doing this? I was considering just doing a Gentoo
install from scratch but figured maybe there's a way to clone enough
of the live system to get me there less painfully?

   The system I'm playing with has five 500MB hard drives with most
partitions in linked together in various forms of RAID. (1, 5 & 6)
That said, the total storage that this system presents KDE and the
users is about 600GB.

   I have an external 1TB eSATA drive which is therefore large enough
to hold everything on this system, albeit without the reliability of
RAID which is fine for this purpose.

   The system looks more or less like:

/dev/sda1 -> /boot (50MB)
/dev/sdb1 -> /boot copy
/dev/sdc1 -> /boot copy

c2stable ~ # df
Filesystem     1K-blocks      Used Available Use% Mounted on
rootfs          51612920  31862844  17128276  66% /
/dev/root       51612920  31862844  17128276  66% /
rc-svcdir           1024        92       932   9% /lib64/rc/init.d
udev               10240       476      9764   5% /dev
shm              6151284         0   6151284   0% /dev/shm
/dev/md7       389183252 350247628  19166232  95% /VirtualMachines
tmpfs            8388608         0   8388608   0% /var/tmp/portage
/dev/sda1          54416     29516     22091  58% /boot
c2stable ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4]
md6 : active raid5 sdb6[1] sdc6[2] sda6[0]
      494833664 blocks super 1.1 level 5, 64k chunk, algorithm 2 [3/3]
[UUU]

md7 : active raid6 sdb7[1] sdc7[2] sda7[0] sdd2[3] sde2[4]
      395387904 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]

md3 : active raid6 sdb3[1] sdc3[2] sda3[0] sdd3[3] sde3[4]
      157305168 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]

md126 : active raid1 sdc5[2] sda5[0] sdb5[1]
      52436032 blocks [3/3] [UUU]

unused devices: <none>
c2stable ~ #

   /dev/md3 is a second Gentoo installation that doesn't need to be
backed up at this time. md6 is an internal RAID used to back up md7
daily. It doesn't need to be backed up, but if the machine totally
failed killing all the drives that wouldn't survive so currently I
back up md126 to md6 daily, and then back up md6 weekly to an external
eSATA drive.

   What I'd like to do is clone

1) /boot (sda1) including grub and everything required to make it bootable
2) back up the system portions of dev/md126 (/ )
3) Add some swap space on the external drive
4) back up /dev/md7 which is all of my VMs
5) back up /home to a separate partition on the external drive
6) back up some special things like /var/lib/portage/world and
/usr/portage/packages

My thought is that this drive is basically bootable, but over time
gets out-of-sync with the system. However should the system fail I've
got a bootable external drive with all the binary packages required to
get it running again quickly. However I can always boot the drive, do
an emerge -ek @world, and basically be back to where I am as of the
last backup.

The external drive will look something like:

/dev/sdg1 -> /boot
/dev/sdg2 -> swap
/dev/sdg3 -> / (not including /home, /usr/portage/distfiles, etc)
/dev/sdg5 -> /usr/portage/packages
/dev/sdg6 -> /dev/md7

etc....

   I will of course have to modify grub.conf and /etc/fstab to work
from this drive but that's no big deal.

   What are folks best ideas about how to approach doing something like this?

Thanks,
Mark

Re: [gentoo-user] Clone live system as a simple backup?

[gandalf]

> Hi,
>    I'm interested in the idea of cloning a live, complicated hardware
> system onto a single external hard drive as a simple backup. I would
> like this external drive to be completely bootable. What's the best
> way to approach doing this? I was considering just doing a Gentoo
> install from scratch but figured maybe there's a way to clone enough
> of the live system to get me there less painfully?
>
>    The system I'm playing with has five 500MB hard drives with most
> partitions in linked together in various forms of RAID. (1, 5 & 6)
> That said, the total storage that this system presents KDE and the
> users is about 600GB.
>
>    I have an external 1TB eSATA drive which is therefore large enough
> to hold everything on this system, albeit without the reliability of
> RAID which is fine for this purpose.
>
>    The system looks more or less like:
>
> /dev/sda1 -> /boot (50MB)
> /dev/sdb1 -> /boot copy
> /dev/sdc1 -> /boot copy
>
> c2stable ~ # df
> Filesystem     1K-blocks      Used Available Use% Mounted on
> rootfs          51612920  31862844  17128276  66% /
> /dev/root       51612920  31862844  17128276  66% /
> rc-svcdir           1024        92       932   9% /lib64/rc/init.d
> udev               10240       476      9764   5% /dev
> shm              6151284         0   6151284   0% /dev/shm
> /dev/md7       389183252 350247628  19166232  95% /VirtualMachines
> tmpfs            8388608         0   8388608   0% /var/tmp/portage
> /dev/sda1          54416     29516     22091  58% /boot
> c2stable ~ # cat /proc/mdstat
> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
> [raid4]
> md6 : active raid5 sdb6[1] sdc6[2] sda6[0]
>       494833664 blocks super 1.1 level 5, 64k chunk, algorithm 2 [3/3]
> [UUU]
>
> md7 : active raid6 sdb7[1] sdc7[2] sda7[0] sdd2[3] sde2[4]
>       395387904 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5]
> [UUUUU]
>
> md3 : active raid6 sdb3[1] sdc3[2] sda3[0] sdd3[3] sde3[4]
>       157305168 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5]
> [UUUUU]
>
> md126 : active raid1 sdc5[2] sda5[0] sdb5[1]
>       52436032 blocks [3/3] [UUU]
>
> unused devices: <none>
> c2stable ~ #
>
>    /dev/md3 is a second Gentoo installation that doesn't need to be
> backed up at this time. md6 is an internal RAID used to back up md7
> daily. It doesn't need to be backed up, but if the machine totally
> failed killing all the drives that wouldn't survive so currently I
> back up md126 to md6 daily, and then back up md6 weekly to an external
> eSATA drive.
>
>    What I'd like to do is clone
>
> 1) /boot (sda1) including grub and everything required to make it bootable
> 2) back up the system portions of dev/md126 (/ )
> 3) Add some swap space on the external drive
> 4) back up /dev/md7 which is all of my VMs
> 5) back up /home to a separate partition on the external drive
> 6) back up some special things like /var/lib/portage/world and
> /usr/portage/packages
>
> My thought is that this drive is basically bootable, but over time
> gets out-of-sync with the system. However should the system fail I've
> got a bootable external drive with all the binary packages required to
> get it running again quickly. However I can always boot the drive, do
> an emerge -ek @world, and basically be back to where I am as of the
> last backup.
>
> The external drive will look something like:
>
> /dev/sdg1 -> /boot
> /dev/sdg2 -> swap
> /dev/sdg3 -> / (not including /home, /usr/portage/distfiles, etc)
> /dev/sdg5 -> /usr/portage/packages
> /dev/sdg6 -> /dev/md7
>
> etc....
>
>    I will of course have to modify grub.conf and /etc/fstab to work
> from this drive but that's no big deal.
>
>    What are folks best ideas about how to approach doing something like
> this?
>
> Thanks,
> Mark
>
>
Hi,
    Why don't you something like bind mount the folders you want to copy
and rsync them to the eSATA disk, after creating a similar partition
layout on it. Remember to exclude system files like /proc/*, /dev/*
and /sys/* as well as the ones you want to exclude yourself from the
rsync. When you want to sync the clone again just do the same again
and rsync the changes.

Regards,
Derek

Re: [gentoo-user] Clone live system as a simple backup?

[Joshua Murphy]

On Wed, Mar 7, 2012 at 8:55 AM,  <gandalf@d-danks.co.uk> wrote:
>> Hi,
>>    I'm interested in the idea of cloning a live, complicated hardware
>> system onto a single external hard drive as a simple backup. I would
>> like this external drive to be completely bootable. What's the best
>> way to approach doing this? I was considering just doing a Gentoo
>> install from scratch but figured maybe there's a way to clone enough
>> of the live system to get me there less painfully?
>>
>>    The system I'm playing with has five 500MB hard drives with most
>> partitions in linked together in various forms of RAID. (1, 5 & 6)
>> That said, the total storage that this system presents KDE and the
>> users is about 600GB.
>>
>>    I have an external 1TB eSATA drive which is therefore large enough
>> to hold everything on this system, albeit without the reliability of
>> RAID which is fine for this purpose.
>>
>>    The system looks more or less like:
>>
>> /dev/sda1 -> /boot (50MB)
>> /dev/sdb1 -> /boot copy
>> /dev/sdc1 -> /boot copy
>>
>> c2stable ~ # df
>> Filesystem     1K-blocks      Used Available Use% Mounted on
>> rootfs          51612920  31862844  17128276  66% /
>> /dev/root       51612920  31862844  17128276  66% /
>> rc-svcdir           1024        92       932   9% /lib64/rc/init.d
>> udev               10240       476      9764   5% /dev
>> shm              6151284         0   6151284   0% /dev/shm
>> /dev/md7       389183252 350247628  19166232  95% /VirtualMachines
>> tmpfs            8388608         0   8388608   0% /var/tmp/portage
>> /dev/sda1          54416     29516     22091  58% /boot
>> c2stable ~ # cat /proc/mdstat
>> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
>> [raid4]
>> md6 : active raid5 sdb6[1] sdc6[2] sda6[0]
>>       494833664 blocks super 1.1 level 5, 64k chunk, algorithm 2 [3/3]
>> [UUU]
>>
>> md7 : active raid6 sdb7[1] sdc7[2] sda7[0] sdd2[3] sde2[4]
>>       395387904 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5]
>> [UUUUU]
>>
>> md3 : active raid6 sdb3[1] sdc3[2] sda3[0] sdd3[3] sde3[4]
>>       157305168 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5]
>> [UUUUU]
>>
>> md126 : active raid1 sdc5[2] sda5[0] sdb5[1]
>>       52436032 blocks [3/3] [UUU]
>>
>> unused devices: <none>
>> c2stable ~ #
>>
>>    /dev/md3 is a second Gentoo installation that doesn't need to be
>> backed up at this time. md6 is an internal RAID used to back up md7
>> daily. It doesn't need to be backed up, but if the machine totally
>> failed killing all the drives that wouldn't survive so currently I
>> back up md126 to md6 daily, and then back up md6 weekly to an external
>> eSATA drive.
>>
>>    What I'd like to do is clone
>>
>> 1) /boot (sda1) including grub and everything required to make it bootable
>> 2) back up the system portions of dev/md126 (/ )
>> 3) Add some swap space on the external drive
>> 4) back up /dev/md7 which is all of my VMs
>> 5) back up /home to a separate partition on the external drive
>> 6) back up some special things like /var/lib/portage/world and
>> /usr/portage/packages
>>
>> My thought is that this drive is basically bootable, but over time
>> gets out-of-sync with the system. However should the system fail I've
>> got a bootable external drive with all the binary packages required to
>> get it running again quickly. However I can always boot the drive, do
>> an emerge -ek @world, and basically be back to where I am as of the
>> last backup.
>>
>> The external drive will look something like:
>>
>> /dev/sdg1 -> /boot
>> /dev/sdg2 -> swap
>> /dev/sdg3 -> / (not including /home, /usr/portage/distfiles, etc)
>> /dev/sdg5 -> /usr/portage/packages
>> /dev/sdg6 -> /dev/md7
>>
>> etc....
>>
>>    I will of course have to modify grub.conf and /etc/fstab to work
>> from this drive but that's no big deal.
>>
>>    What are folks best ideas about how to approach doing something like
>> this?
>>
>> Thanks,
>> Mark
>>
>>
> Hi,
>    Why don't you something like bind mount the folders you want to copy
> and rsync them to the eSATA disk, after creating a similar partition
> layout on it. Remember to exclude system files like /proc/*, /dev/*
> and /sys/* as well as the ones you want to exclude yourself from the
> rsync. When you want to sync the clone again just do the same again
> and rsync the changes.
>
> Regards,
> Derek
>
>

As an added note on this, rsync's --one-file-system (-x) flag is handy
for avoiding grabbing unneeded things, but will typically leave you
without the base few device nodes needed to boot the backup, those can
either be grabbed from a stage3, or created with (courtesy of Linux
From Scratch's section "6.2.1. Creating Initial Device Nodes"):

mknod -m 600 ${backup}/dev/console c 5 1
mknod -m 666 ${backup}/dev/null c 1 3

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] Clone live system as a simple backup?

[YoYo Siska]

On Wed, Mar 07, 2012 at 02:47:08PM -0500, Joshua Murphy wrote:
> On Wed, Mar 7, 2012 at 8:55 AM,  <gandalf@d-danks.co.uk> wrote:
> >> Hi,
> >>    I'm interested in the idea of cloning a live, complicated hardware
> >> system onto a single external hard drive as a simple backup. I would
> >> like this external drive to be completely bootable. What's the best
> >> way to approach doing this? I was considering just doing a Gentoo
> >> install from scratch but figured maybe there's a way to clone enough
> >> of the live system to get me there less painfully?
> >>
> >>    The system I'm playing with has five 500MB hard drives with most
> >> partitions in linked together in various forms of RAID. (1, 5 & 6)
> >> That said, the total storage that this system presents KDE and the
> >> users is about 600GB.
> >>
> >>    I have an external 1TB eSATA drive which is therefore large enough
> >> to hold everything on this system, albeit without the reliability of
> >> RAID which is fine for this purpose.
> >>
> >>    The system looks more or less like:
> >>
> >> /dev/sda1 -> /boot (50MB)
> >> /dev/sdb1 -> /boot copy
> >> /dev/sdc1 -> /boot copy
> >>
> >> c2stable ~ # df
> >> Filesystem     1K-blocks      Used Available Use% Mounted on
> >> rootfs          51612920  31862844  17128276  66% /
> >> /dev/root       51612920  31862844  17128276  66% /
> >> rc-svcdir           1024        92       932   9% /lib64/rc/init.d
> >> udev               10240       476      9764   5% /dev
> >> shm              6151284         0   6151284   0% /dev/shm
> >> /dev/md7       389183252 350247628  19166232  95% /VirtualMachines
> >> tmpfs            8388608         0   8388608   0% /var/tmp/portage
> >> /dev/sda1          54416     29516     22091  58% /boot
> >> c2stable ~ # cat /proc/mdstat
> >> Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
> >> [raid4]
> >> md6 : active raid5 sdb6[1] sdc6[2] sda6[0]
> >>       494833664 blocks super 1.1 level 5, 64k chunk, algorithm 2 [3/3]
> >> [UUU]
> >>
> >> md7 : active raid6 sdb7[1] sdc7[2] sda7[0] sdd2[3] sde2[4]
> >>       395387904 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5]
> >> [UUUUU]
> >>
> >> md3 : active raid6 sdb3[1] sdc3[2] sda3[0] sdd3[3] sde3[4]
> >>       157305168 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5]
> >> [UUUUU]
> >>
> >> md126 : active raid1 sdc5[2] sda5[0] sdb5[1]
> >>       52436032 blocks [3/3] [UUU]
> >>
> >> unused devices: <none>
> >> c2stable ~ #
> >>
> >>    /dev/md3 is a second Gentoo installation that doesn't need to be
> >> backed up at this time. md6 is an internal RAID used to back up md7
> >> daily. It doesn't need to be backed up, but if the machine totally
> >> failed killing all the drives that wouldn't survive so currently I
> >> back up md126 to md6 daily, and then back up md6 weekly to an external
> >> eSATA drive.
> >>
> >>    What I'd like to do is clone
> >>
> >> 1) /boot (sda1) including grub and everything required to make it bootable
> >> 2) back up the system portions of dev/md126 (/ )
> >> 3) Add some swap space on the external drive
> >> 4) back up /dev/md7 which is all of my VMs
> >> 5) back up /home to a separate partition on the external drive
> >> 6) back up some special things like /var/lib/portage/world and
> >> /usr/portage/packages
> >>
> >> My thought is that this drive is basically bootable, but over time
> >> gets out-of-sync with the system. However should the system fail I've
> >> got a bootable external drive with all the binary packages required to
> >> get it running again quickly. However I can always boot the drive, do
> >> an emerge -ek @world, and basically be back to where I am as of the
> >> last backup.
> >>
> >> The external drive will look something like:
> >>
> >> /dev/sdg1 -> /boot
> >> /dev/sdg2 -> swap
> >> /dev/sdg3 -> / (not including /home, /usr/portage/distfiles, etc)
> >> /dev/sdg5 -> /usr/portage/packages
> >> /dev/sdg6 -> /dev/md7
> >>
> >> etc....
> >>
> >>    I will of course have to modify grub.conf and /etc/fstab to work
> >> from this drive but that's no big deal.
> >>
> >>    What are folks best ideas about how to approach doing something like
> >> this?
> >>
> >> Thanks,
> >> Mark
> >>
> >>
> > Hi,
> >    Why don't you something like bind mount the folders you want to copy
> > and rsync them to the eSATA disk, after creating a similar partition
> > layout on it. Remember to exclude system files like /proc/*, /dev/*
> > and /sys/* as well as the ones you want to exclude yourself from the
> > rsync. When you want to sync the clone again just do the same again
> > and rsync the changes.
> >
> > Regards,
> > Derek
> >
> >
> 
> As an added note on this, rsync's --one-file-system (-x) flag is handy
> for avoiding grabbing unneeded things, but will typically leave you
> without the base few device nodes needed to boot the backup, those can
> either be grabbed from a stage3, or created with (courtesy of Linux
> From Scratch's section "6.2.1. Creating Initial Device Nodes"):
> 
> mknod -m 600 ${backup}/dev/console c 5 1
> mknod -m 666 ${backup}/dev/null c 1 3

The best way to copy a filesystem without any sub-mounts is to
mount-bind it to some directory and copy it from there:

mkdir /tmp/root
mount --bind / /tmp/root
rsync -a /tmp/root/ /mnt/backup/

Note that copying a rw filesystem (especially /) on a running system
isn't a very clean operation..

yoyo

Re: [gentoo-user] Clone live system as a simple backup?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

On Mar 8, 2012 2:50 AM, "Joshua Murphy" <poisonbl@gmail.com> wrote:
>

---- >8 snip

>
> As an added note on this, rsync's --one-file-system (-x) flag is handy
> for avoiding grabbing unneeded things, but will typically leave you
> without the base few device nodes needed to boot the backup, those can
> either be grabbed from a stage3, or created with (courtesy of Linux
> From Scratch's section "6.2.1. Creating Initial Device Nodes"):
>
> mknod -m 600 ${backup}/dev/console c 5 1
> mknod -m 666 ${backup}/dev/null c 1 3
>

... or just add another rsync invocation to backup /dev ...

Rgds,

[-- Attachment #2: Type: text/html, Size: 785 bytes --]

Re: [gentoo-user] Clone live system as a simple backup?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

On Thu, 8 Mar 2012 23:52:53 +0700, Pandu Poluan wrote:

> > As an added note on this, rsync's --one-file-system (-x) flag is handy
> > for avoiding grabbing unneeded things, but will typically leave you
> > without the base few device nodes needed to boot the backup, those can
> > either be grabbed from a stage3, or created with (courtesy of Linux
> > From Scratch's section "6.2.1. Creating Initial Device Nodes"):
> >
> > mknod -m 600 ${backup}/dev/console c 5 1
> > mknod -m 666 ${backup}/dev/null c 1 3
> >  
> 
> ... or just add another rsync invocation to backup /dev ...

That won't work because it will backup the full devfs mounted on /dev,
not the files that exist in the directory itself.


-- 
Neil Bothwick

Despite the cost of living, have you noticed how it remains so popular?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Clone live system as a simple backup?

[Mark Knecht]

On Thu, Mar 8, 2012 at 9:12 AM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Thu, 8 Mar 2012 23:52:53 +0700, Pandu Poluan wrote:
>
>> > As an added note on this, rsync's --one-file-system (-x) flag is handy
>> > for avoiding grabbing unneeded things, but will typically leave you
>> > without the base few device nodes needed to boot the backup, those can
>> > either be grabbed from a stage3, or created with (courtesy of Linux
>> > From Scratch's section "6.2.1. Creating Initial Device Nodes"):
>> >
>> > mknod -m 600 ${backup}/dev/console c 5 1
>> > mknod -m 666 ${backup}/dev/null c 1 3
>> >
>>
>> ... or just add another rsync invocation to backup /dev ...
>
> That won't work because it will backup the full devfs mounted on /dev,
> not the files that exist in the directory itself.
>
>
> --
> Neil Bothwick
>
> Despite the cost of living, have you noticed how it remains so popular?

Thanks for all the ideas. As I've not done this sort of thing before
every one of them has been worth my time thinking about.

I ended up going a slightly different direction, only made possible by
how little disk space costs these days. Here's what I did:

1) As a chroot, on the live system I created a new Gentoo install.
This is just a very basic single disk partition type install as per
the Gentoo install guide. No apps, no KDE, nothing. Just the basic
stuff.

2) Once that install was up & running I copied the server's world
file, /etc/portage/package.*, /etc/conf.d, /etc/X/xorg.conf & a few
other things into this new install and ran emerge -ek @world to get it
up and running.

3) When all of that was complete and functioning, at least in the
chroot, I then took the chroot offline and used rsync to get the
chroot installed copied to  the external drive.

4) WIth the rsync complete I then took a cut at modifying /etc/fstab
to be correct for booting from the external drive. Not sure about this
step at this time as I don't know for sure how the machine will name
the external drive.

I'm now looking to get /boot copied to the external drive, get grub
installed, and then see if I can boot using that drive.

Again, thanks for the ideas and keep 'em coming if you have more.

Cheers,
Mark

Re: [gentoo-user] Clone live system as a simple backup?

[Brian Wiborg]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

On 03/06/12 at 11:51AM -0800, Mark Knecht wrote:
>    What are folks best ideas about how to approach doing something like this?

Simple answer:
Just use ReaR [1], it is even provided in sunrise [2]

Regards,
bacce

[1] http://rear.sourceforge.net/
[2] http://overlays.gentoo.org/proj/sunrise/browser/sunrise/app-backup/rear


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]