[gentoo-user] backup to a cold-swap drive

[gentoo-user] backup to a cold-swap drive

[Iain Buchanan]

Hi,

A winblows colleague said he uses a utility to backup his internal hard
drive to an external disk, such that if his internal disk fails he can
replace it with the external disk and continue straight away.

Since I go to weird locations with unreliable power and sometimes drop
my laptop I thought it should be simple to do the same in Linux.  I have
an external disk the same size, but now what?

      * I want to copy changes intelligently (ie. no dd, gparted, or
        Ghost4Linux).
      * I want to copy a specific device only (no usb keys, etc) to a
        specific external device.
      * Windows partitions can be ignored.
      * It doesn't matter if the copy is not unmounted properly, eg. if
        power is shut of without shutting down.
      * The external disk must be able to be absent

Can md use one internal and one external disk in a RAID 1 setup, with
the external disk not always there?  Any other suggestions?

thanks :)
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Better tried by twelve than carried by six.
		-- Jeff Cooper

Re: [gentoo-user] backup to a cold-swap drive

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 2206 bytes --]

Am 28.04.2010 03:41, schrieb Iain Buchanan:
> Hi,
> 
> A winblows colleague said he uses a utility to backup his internal hard
> drive to an external disk, such that if his internal disk fails he can
> replace it with the external disk and continue straight away.
> 
> Since I go to weird locations with unreliable power and sometimes drop
> my laptop I thought it should be simple to do the same in Linux.  I have
> an external disk the same size, but now what?
> 
>       * I want to copy changes intelligently (ie. no dd, gparted, or
>         Ghost4Linux).
>       * I want to copy a specific device only (no usb keys, etc) to a
>         specific external device.
>       * Windows partitions can be ignored.
>       * It doesn't matter if the copy is not unmounted properly, eg. if
>         power is shut of without shutting down.
>       * The external disk must be able to be absent
> 
> Can md use one internal and one external disk in a RAID 1 setup, with
> the external disk not always there?  Any other suggestions?
> 
> thanks :)

md would be extremely slow because it has to rebuild/resync the complete
array.

I suggest you manually recreate the partitioning scheme, install grub,
then mount it with some little script and call some backup tool to do
the actual copying.

If you can live with just one big partition as a backup (probably with
separate /boot), you should replace fstab and grub.conf on the backup
medium and blacklist them from the files which you want to back up.

Concerning the backup tool, I would use `rsync --delete` plus all
relevant switches for permissions, times, acls, etc. If you use another
tool, just make sure it doesn't put some metadata onto the backup medium
and that it can delete files which no longer exist on the original medium.

With regard to your requirement to just 'pull the cord' without
umounting it: Better mount it with '-o sync' to increase your chance
that everything works fine afterwards. But in reality, nothing can
really protect you from filesystem corruption in this situation. If you
can afford, better keep two backup media which you round-robin.

Hope this helps,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] backup to a cold-swap drive

[Iain Buchanan]

Hi & thanks,

On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote:

> > Can md use one internal and one external disk in a RAID 1 setup, with
> > the external disk not always there?  Any other suggestions?
> > 
> > thanks :)
> 
> md would be extremely slow because it has to rebuild/resync the complete
> array.

so every time you unplug and re-plug the external disk, it will
essentially re-copy everything?  Damn, there goes that fine idea!

> If you can live with just one big partition as a backup (probably with
> separate /boot), you should replace fstab and grub.conf on the backup
> medium and blacklist them from the files which you want to back up.

why wouldn't I backup fstab and grub.conf as well?  If my internal disk
dies, I assume I'll swap them over, meaning grub and fstab will have to
be the same.

> Concerning the backup tool, I would use `rsync --delete` plus all
> relevant switches for permissions, times, acls, etc. If you use another
> tool, just make sure it doesn't put some metadata onto the backup medium
> and that it can delete files which no longer exist on the original medium.

I was thinking of rsync, but I didn't want to do it in an hourly cron
fashion, I was hoping for some "gamin" alteration-triggered idea.

> With regard to your requirement to just 'pull the cord' without
> umounting it:

I wasn't thinking of pulling the chord without unmounting, I was
thinking of the machine dying, hence leaving the disk in a non-shutdown
state.

thanks for the tips :)  rsync will at least get me going quickly.
Yesterday I tried iotop to with dd - some slowness but otherwise quite
nice.

-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Real computer scientists don't comment their code.  The identifiers are
so long they can't afford the disk space.

Re: [gentoo-user] backup to a cold-swap drive

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 2558 bytes --]

Am 29.04.2010 02:38, schrieb Iain Buchanan:
> Hi & thanks,
> 
> On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote:
[...]
> 
>> If you can live with just one big partition as a backup (probably with
>> separate /boot), you should replace fstab and grub.conf on the backup
>> medium and blacklist them from the files which you want to back up.
> 
> why wouldn't I backup fstab and grub.conf as well?  If my internal disk
> dies, I assume I'll swap them over, meaning grub and fstab will have to
> be the same.

I think you misunderstood me or I didn't explain it correctly. I try it
again:

You probably have a lot of partitions on your internal disk; one for /,
/usr and /home, for example. My reasoning was that it is probably easier
to have just one big partition on your backup medium instead of many
small ones. If you would then swap your backup medium in, with exactly
the same fstab as in your original installation, your system would try
to mount partitions which are simply not there. Therefore you need a
different fstab and most likely also another grub.conf.

> 
>> Concerning the backup tool, I would use `rsync --delete` plus all
>> relevant switches for permissions, times, acls, etc. If you use another
>> tool, just make sure it doesn't put some metadata onto the backup medium
>> and that it can delete files which no longer exist on the original medium.
> 
> I was thinking of rsync, but I didn't want to do it in an hourly cron
> fashion, I was hoping for some "gamin" alteration-triggered idea.
> 

Ah, I see what you mean. I've never worked with the file alteration
monitor (FAM) but once evaluated inotify for some administrative
purposes. AFAIK they are not scalable good enough to work on a system
wide basis. For example, I think the default limit of observable files
with inotify is 8192.

>> With regard to your requirement to just 'pull the cord' without
>> umounting it:
> 
> I wasn't thinking of pulling the chord without unmounting, I was
> thinking of the machine dying, hence leaving the disk in a non-shutdown
> state.
> 

Okay, I thought you meant the unreliable power at those "weird
locations" you were talking about. Such a black-out or brown-out is
basically the same as pulling the cord.

> thanks for the tips :)  rsync will at least get me going quickly.
> Yesterday I tried iotop to with dd - some slowness but otherwise quite
> nice.
> 

To reduce the performance impact, you can also use the ionice command.

Hope this helps,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] backup to a cold-swap drive

[Iain Buchanan]

On Fri, 2010-04-30 at 16:24 +0200, Florian Philipp wrote:
> Am 29.04.2010 02:38, schrieb Iain Buchanan:
> > Hi & thanks,
> > 
> > On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote:
> [...]
> > 
> >> If you can live with just one big partition as a backup (probably with
> >> separate /boot), you should replace fstab and grub.conf on the backup
> >> medium and blacklist them from the files which you want to back up.
> > 
> > why wouldn't I backup fstab and grub.conf as well?  If my internal disk
> > dies, I assume I'll swap them over, meaning grub and fstab will have to
> > be the same.
> 
> I think you misunderstood me or I didn't explain it correctly. I try it
> again:

[snip]

ah, NOW I get it :)

[snip]

> Ah, I see what you mean. I've never worked with the file alteration
> monitor (FAM) but once evaluated inotify for some administrative
> purposes. AFAIK they are not scalable good enough to work on a system
> wide basis. For example, I think the default limit of observable files
> with inotify is 8192.

hm, there goes that idea!  Is there any kernel based "watch" on all file
based I/O that I could queue up somehow?  Just thinking aloud here.  I
know "everything is a file", but no doubt I could watch all write
operations; filter out /dev and put the rest into a file; and then use
it like an rsync file-list...

> > thanks for the tips :)  rsync will at least get me going quickly.
> > Yesterday I tried iotop to with dd - some slowness but otherwise quite
> > nice.
> > 
> 
> To reduce the performance impact, you can also use the ionice command.

whoops, that's what I meant.  Even with ionice, there was some
noticeable delay when switching screens, opening programs, etc.  More so
when my RAM had been swapped from the large amount of I/O (I assume).  I
didn't try ionice with nice.

thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

One family builds a wall, two families enjoy it.

Re: [gentoo-user] backup to a cold-swap drive

[Frank Steinmetzger]

[-- Attachment #1: Type: Text/Plain, Size: 1876 bytes --]

Am Mittwoch, 28. April 2010 schrieb Iain Buchanan:
> Hi,
> 
> A winblows colleague said he uses a utility to backup his internal hard
> drive to an external disk, such that if his internal disk fails he can
> replace it with the external disk and continue straight away.
> 
> Since I go to weird locations with unreliable power and sometimes drop
> my laptop I thought it should be simple to do the same in Linux.  I have
> an external disk the same size, but now what?
> 
>       * I want to copy changes intelligently (ie. no dd, gparted, or
>         Ghost4Linux).
>       * I want to copy a specific device only (no usb keys, etc) to a
>         specific external device.
>       * Windows partitions can be ignored.
>       * It doesn't matter if the copy is not unmounted properly, eg. if
>         power is shut of without shutting down.
>       * The external disk must be able to be absent
> 
> Can md use one internal and one external disk in a RAID 1 setup, with
> the external disk not always there?  Any other suggestions?

After I upgraded my laptop with an internal HDD of 500 GB, I started using my 
old external 500 GB drive as backup. Though of different dimensions and 
makers, they both have the same number of sectors. So I dd'ed the entire disk 
first, which gave me an exact mirror of the internal disk. I though this would 
be faster, because I have lots of small files in some places.

But now I can update the backup by a simple call to rsync:

rsync -aX --delete / /dev/<backup root partition>/

-a (archive) copies permissions, ownerships and the likes
-X stops at file system boundaries, i.e. it will only backup the actual root 
partition, without other mounted file systems such as /proc, /dev and /home.
-- 
Gruß | Greetings | Qapla'
Why did the tachyon cross the road?
Because it was on the other side.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] backup to a cold-swap drive

[Iain Buchanan]

Hi,

On Wed, 2010-04-28 at 23:16 +0200, Frank Steinmetzger wrote:

> After I upgraded my laptop with an internal HDD of 500 GB, I started using my 
> old external 500 GB drive as backup. Though of different dimensions and 
> makers, they both have the same number of sectors. So I dd'ed the entire disk 
> first, which gave me an exact mirror of the internal disk. I though this would 
> be faster, because I have lots of small files in some places.

I tried that, but after dd finished, I was left with strange partitions
and id's that I couldn't mount.  The two disks are both 160Gb with the
same sector size...

It might be easier to do the fdisk-ing by hand.

thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Necessity is the plea for every infringement of human freedom.
It is the argument of tyrants; it is the creed of slaves.
		-- William Pitt, 1783

Re: [gentoo-user] backup to a cold-swap drive

[Iain Buchanan]

On Wed, 2010-04-28 at 23:16 +0200, Frank Steinmetzger wrote:

> rsync -aX --delete / /dev/<backup root partition>/
> 
> -a (archive) copies permissions, ownerships and the likes
> -X stops at file system boundaries, i.e. it will only backup the actual root 
> partition, without other mounted file systems such as /proc, /dev and /home.

actually, lower case x is --one-file-system or "don't cross filesystem
boundaries".  Upper case X is --xattrs or "preserve extended attributes"

:)
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

When you don't know what to do, walk fast and look worried.

Re: [gentoo-user] backup to a cold-swap drive

[Alex Schuster]

Iain Buchanan writes:

> A winblows colleague said he uses a utility to backup his internal hard
> drive to an external disk, such that if his internal disk fails he can
> replace it with the external disk and continue straight away.

I do the same, but with a 2nd internal drive. The drive is partitioned 
similar, with some partitions being a bit larger so I can do incremental 
backups, too.

I am using rdiff-backup, which makes use of rsync. The backup partition 
has exactly the same contents as the source partition, except for an 
additional 'rdiff-backup' directory that contains incremental backups of 
files that were modified from backup to backup, gzipped.

Some other partitions are handled differently: /boot is just being dd'ed, 
contents of /usr/src are tarred each, and /var/portage/packages/ is just 
plain rsynced. Some unnecessary stuff like .ccache and /var/tmp/portage is 
excluded.

All my partitions are LVM volumes, so before the backup starts, I make a 
LVM snapshot of the partition. This way I can modify it while the backup 
is still in progress.

I wrote a shell script to do this, so I do not have to issue a lot of 
commands every time I want to do the backup. As there are now some others 
using this script, adapted to their needs, I started to rewrite it in a 
way that it reads a config file, and no modification of the script itself 
is necessary. If anyone is interested, send me an email.


Some time ago my first drive started having bad blocks. Without LVM, I 
could just have swapped the disks, but so I had to rename the backup 
volume group to the original volume group from a live cd. And the system 
was running from the new drive as it was before - only that I no longer 
had a backup until the new drive arrived. This makes an uneasy feeling 
with these 1.5 TB drives.

	Wonko

Re: [gentoo-user] backup to a cold-swap drive

[Iain Buchanan]

On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:

[snip]
> All my partitions are LVM volumes, so before the backup starts, I make a 
> LVM snapshot of the partition. This way I can modify it while the backup 
> is still in progress.

hmm, never got into LVM.  Sounds interesting though...

[snip]
> I wrote a shell script to do this, so I do not have to issue a lot of 
> commands every time I want to do the backup.

I don't use too many commands, something like this
in /etc/cron.daily/custom-backup:

sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
--delete-delay --delete-excluded --partial
--human-readable / /<unique-mount-of-external-drive> || echo external
backup failed!

>  As there are now some others 
> using this script, adapted to their needs, I started to rewrite it in a 
> way that it reads a config file, and no modification of the script itself 
> is necessary. If anyone is interested, send me an email.

interested! So is it on sourceforge yet ;)

thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

BOFH Excuse #418:

Sysadmins busy fighting SPAM.

Re: [gentoo-user] backup to a cold-swap drive

[Alex Schuster]

Iain Buchanan writes:

> On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:
> 
> [snip]
> 
> > All my partitions are LVM volumes, so before the backup starts, I
> > make a LVM snapshot of the partition. This way I can modify it while
> > the backup is still in progress.
> 
> hmm, never got into LVM.  Sounds interesting though...

It is. it gives you great flexibility. Your /home is becoming too small? 
So just add a Gigabyte:
  lvresize -L +1G /dev/system/home 
  resize2fs /dev/system/home

Takes half a minute, and you don't need to unmount anything before that.

You want to backup your large /home partition, and want to continue 
working meanwhile? Create a snapshot, mount it, and backup this:
  lvcreate -s -L 1G -n home-snapshot /dev/system/home
  mount /dev/system/home-snapshot /mnt


> [snip]
> 
> > I wrote a shell script to do this, so I do not have to issue a lot of
> > commands every time I want to do the backup.
> 
> I don't use too many commands, something like this
> in /etc/cron.daily/custom-backup:
> 
> sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> --delete-delay --delete-excluded --partial
> --human-readable / /<unique-mount-of-external-drive> || echo external
> backup failed!

That's okay, I did it in this way, too. Well, the backup may be 
inconsistent when you are modifying the root partition in the meantime, so 
I would not emerge things when the backup is running.
But now I am using LVM for everything, and so I just create a snapshot.
And I prefer to use rdiff-backup, so I can have incremental backups.

> >  As there are now some others
> > using this script, adapted to their needs, I started to rewrite it in
> > a way that it reads a config file, and no modification of the script
> > itself is necessary. If anyone is interested, send me an email.
> 
> interested! So is it on sourceforge yet ;)

Um, no... not yet. Not really sure if it belongs there...

The script has become a little large, but I think it's working now. I will 
do some testing, and inform you when it's done. I'm a little ill at the 
moment, so it may take 1-2 more days.

	Wonko

snackup (was: Re: [gentoo-user] backup to a cold-swap drive)

[Alex Schuster]

Iain Buchanan writes:

> On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:

> > using this script, adapted to their needs, I started to rewrite it in
> > a way that it reads a config file, and no modification of the script
> > itself is necessary. If anyone is interested, send me an email.
> 
> interested! So is it on sourceforge yet ;)

Sorry, it took a little longer. I was ill, and then these scripts tend to 
grow and grow until I am satisfied with them. It's still not perfect, but 
I think at the moment it does what it should do.

Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup
The name is silly, but I kinda like it now. A backup utilizing LVM 
snapshots... snapshot backup... snackup!

It needs a config file that is looked up in some default locations, or can 
be specified with the -c option. Use option -T to generate a template that 
has information on the syntax and some examples. In short:

The config file defines targets you give as arguments to snackup. So 
'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz 
file of your /etc. The config file is sourced, so you can put any bash 
stuff you want into there, like some shell functions you want to have 
executed. For example, I want my /var/log/portage/*.log files to be 
compressed before I backup my /var partition.

A target is started by a colon, followed by name, type, source and 
destination (and optionally more), separated by one or more tab 
characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only 
tested dd, tar and rdiff so far, though. The source directory may be 
prefixed by the volume group and LVM of that partition in order to create 
an lvm snapshot first. You may add a LUKS key if the partition is 
encrypted. The source may contain wildcards. In that case, all matching 
files are backed up to the destination. If the type has a * appended, all 
matching files are backed up individually. Some examples:

:	tar 		kde		.kde*	/backup/kde.%s

Target 'kde' will backup all your .kde* directories to /backup/kde.tar 
(the %s is replaced by a suffix, usually tar). You need to be in your home 
directory, otherwise the path will not be found. This is not necessary if
you use ~//.kde* instead.

:	kernels	tar*	/usr/src//linux-*		/backup/src/%f.%s		-j	-	lsf

Target 'kernels' will backup your /usr/src/linux-* directories 
individually - one tar file for each (note the * appendended to the tar 
target type). The -z option will be passed to the tar command, and the 
suffix is changed to tgz accordingly.
The // denotes that the backup should be done locally from the directory 
left to the //. Otherwise, the full path would be included in the tar 
file.

:	tar		etc		/etc		/backup/etc_%d.%s	-j

Target 'etc' will tar your /etc directory, compressed with bzip2. The %d 
will be replaced by the current date. You can change the date format by 
re-defining a date() shell function.
You will get a message about tar removing the leading /, you can avoid it 
by replacing /etc by ///etc, meaning that your local directroy will be /.

:	boot		dd		/dev/sda5		/dev/sdb5		bs=32M

Target 'boot' will backup a boot partition with dd. Options like the 
bs=32M are added directly to the dd command.

:	home		rsync	system/home::.	/backup/home/

Target 'home' will create a LVM snapshot of the /dev/system/home logical 
volume, mount it and back it up with rdiff-backup. The '.' could also be a 
/ or left off.

:	var		rsync	system/var::/	/backup/var/	-	ziplog

Similar, but the command ziplog will be executed before the backup, 
compressing some log files. It must be defined in the config file.

The script has some options:

  -c file location of config file
  -C      clear destination first
  -d      dummy mode, just show what would be done
  -f      force backup (the initial rdiff-backup may need this)
  -h      show this help
  -l      output to log file, too
  -L      log file (default:/home/wonko/log/snackup.log); may be a
          directory (add a trailing slash) to create target-specific logs
  -n l    use nice level l (default:10, 0 to turn off)
  -N l    use ionice level l (default:3, 0 to turn off)
  -o      extra options that will added to the actual backup command
  -s size size of LVM snapshot (default:2G)
  -S char replace tab as delimiter for targets in config file
  -T      output config file template
  -v      verbose output; may be given multiple times
          1: some extra output; 2: add -v to commands; 3: set -xv
  -V      output version information and exit

Most important are:
  -c to specify the location of the config file
  -d to output what would be done, you will see the final tar / rdiff-
backup / whatever command with all its options.
  -T to see a template. Save it with snackup -T > ~/.snackup and then 
modify your ~/.snackup file.
  -o to add whichever options you like to the final command. These (and -j 
and -z) override options already defined in the config file.

Too complicated? Overkill? Maybe. But I like it, and now I do not have to 
write another script for every system I want to backup. And I like the LVm 
snapshotting to be done automatically. It's not that hard, but it's 
annoying to do it by hand, even more as all my volumes are encrypted.

These are my own targets:

:	boot		dd		/dev/boot					/dev/sdb5						-	-		-
:	backup	rdiff	weird/backup::/etc/key::.	/backup/weird/backup			-	-		-
:	local	rdiff	weird/local::/etc/key::.	/backup/weird/local			-	-		-
:	home		rdiff	weird/home::/etc/key::.	/backup/weird/home				-	-		-
:	med		rdiff	weird/med::/etc/key::.		/backup/weird/med				-	-		-
:	mp3		rdiff	weird/mp3::/etc/key::.		/backup/weird/mp3				-	-		-
:	mpeg		rdiff	weird/mpeg::/etc/key::.	/backup/weird/mpeg				-	-		-
:	opt		rdiff	weird/opt::/etc/key::.		/backup/weird/opt				-	-		-
:	root		rdiff	weird/root::/etc/key::.	/backup/weird/root				-	-		-
:	usr		rdiff	weird/usr::/etc/key::.		/backup/weird/usr				-	-		-
:	var		rdiff	weird/var::/etc/key::.		/backup/weird/var				-	zipLog	-
:	etc		tar		///etc					/backup/conf/etc/etc_%d.%s		-z	-		lsf
:	kernels	tar*	/usr/src//linux-*				/backup/weird/src/%f.%s		-j	-		lsf
:	portage	rsync	/var/portage/packages		/backup/weird/portage			-	-		-
:	kde		tar	~//.kde*						/backup/wonko/home/kde-%d.%s	-z	-		lsf
:	dot		tar	~//.??*						/backup/extern/dot-%d.%s		-z	-		lsf


At last, the disclaimer: Use it at your own risk! This is a script of some
800 lines of bash code some guy wrote, it messes around with LVM stuff,
creates and removes temporary directories and even has an option to wipe
the destination completely. I'm quite confident that no harm will be done,
and I do not suggest you to take a backup before daring to try this script,
but anyway.

Use the -d option first to test what actually would be going on, before you
do a real backup!

Hope this is of good use to someone. For me it is, I will be using it on 
several systems, and now I only have to make little changes in the config
file for each system, but not at the script.

Suggestions and bug reports are welcome of course.

	Wonko

Re: snackup (was: Re: [gentoo-user] backup to a cold-swap drive)

[covici]

Alex Schuster <wonko@wonkology.org> wrote:

> Iain Buchanan writes:
> 
> > On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote:
> 
> > > using this script, adapted to their needs, I started to rewrite it in
> > > a way that it reads a config file, and no modification of the script
> > > itself is necessary. If anyone is interested, send me an email.
> > 
> > interested! So is it on sourceforge yet ;)
> 
> Sorry, it took a little longer. I was ill, and then these scripts tend to 
> grow and grow until I am satisfied with them. It's still not perfect, but 
> I think at the moment it does what it should do.
> 
> Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup
> The name is silly, but I kinda like it now. A backup utilizing LVM 
> snapshots... snapshot backup... snackup!
> 
> It needs a config file that is looked up in some default locations, or can 
> be specified with the -c option. Use option -T to generate a template that 
> has information on the syntax and some examples. In short:
> 
> The config file defines targets you give as arguments to snackup. So 
> 'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz 
> file of your /etc. The config file is sourced, so you can put any bash 
> stuff you want into there, like some shell functions you want to have 
> executed. For example, I want my /var/log/portage/*.log files to be 
> compressed before I backup my /var partition.
> 
> A target is started by a colon, followed by name, type, source and 
> destination (and optionally more), separated by one or more tab 
> characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only 
> tested dd, tar and rdiff so far, though. The source directory may be 
> prefixed by the volume group and LVM of that partition in order to create 
> an lvm snapshot first. You may add a LUKS key if the partition is 
> encrypted. The source may contain wildcards. In that case, all matching 
> files are backed up to the destination. If the type has a * appended, all 
> matching files are backed up individually. Some examples:
> 
> :	tar 		kde		.kde*	/backup/kde.%s
> 
> Target 'kde' will backup all your .kde* directories to /backup/kde.tar 
> (the %s is replaced by a suffix, usually tar). You need to be in your home 
> directory, otherwise the path will not be found. This is not necessary if
> you use ~//.kde* instead.
> 
> :	kernels	tar*	/usr/src//linux-*		/backup/src/%f.%s		-j	-	lsf
> 
> Target 'kernels' will backup your /usr/src/linux-* directories 
> individually - one tar file for each (note the * appendended to the tar 
> target type). The -z option will be passed to the tar command, and the 
> suffix is changed to tgz accordingly.
> The // denotes that the backup should be done locally from the directory 
> left to the //. Otherwise, the full path would be included in the tar 
> file.
> 
> :	tar		etc		/etc		/backup/etc_%d.%s	-j
> 
> Target 'etc' will tar your /etc directory, compressed with bzip2. The %d 
> will be replaced by the current date. You can change the date format by 
> re-defining a date() shell function.
> You will get a message about tar removing the leading /, you can avoid it 
> by replacing /etc by ///etc, meaning that your local directroy will be /.
> 
> :	boot		dd		/dev/sda5		/dev/sdb5		bs=32M
> 
> Target 'boot' will backup a boot partition with dd. Options like the 
> bs=32M are added directly to the dd command.
> 
> :	home		rsync	system/home::.	/backup/home/
> 
> Target 'home' will create a LVM snapshot of the /dev/system/home logical 
> volume, mount it and back it up with rdiff-backup. The '.' could also be a 
> / or left off.
> 
> :	var		rsync	system/var::/	/backup/var/	-	ziplog
> 
> Similar, but the command ziplog will be executed before the backup, 
> compressing some log files. It must be defined in the config file.
> 
> The script has some options:
> 
>   -c file location of config file
>   -C      clear destination first
>   -d      dummy mode, just show what would be done
>   -f      force backup (the initial rdiff-backup may need this)
>   -h      show this help
>   -l      output to log file, too
>   -L      log file (default:/home/wonko/log/snackup.log); may be a
>           directory (add a trailing slash) to create target-specific logs
>   -n l    use nice level l (default:10, 0 to turn off)
>   -N l    use ionice level l (default:3, 0 to turn off)
>   -o      extra options that will added to the actual backup command
>   -s size size of LVM snapshot (default:2G)
>   -S char replace tab as delimiter for targets in config file
>   -T      output config file template
>   -v      verbose output; may be given multiple times
>           1: some extra output; 2: add -v to commands; 3: set -xv
>   -V      output version information and exit
> 
> Most important are:
>   -c to specify the location of the config file
>   -d to output what would be done, you will see the final tar / rdiff-
> backup / whatever command with all its options.
>   -T to see a template. Save it with snackup -T > ~/.snackup and then 
> modify your ~/.snackup file.
>   -o to add whichever options you like to the final command. These (and -j 
> and -z) override options already defined in the config file.
> 
> Too complicated? Overkill? Maybe. But I like it, and now I do not have to 
> write another script for every system I want to backup. And I like the LVm 
> snapshotting to be done automatically. It's not that hard, but it's 
> annoying to do it by hand, even more as all my volumes are encrypted.
> 
> These are my own targets:
> 
> :	boot		dd		/dev/boot					/dev/sdb5						-	-		-
> :	backup	rdiff	weird/backup::/etc/key::.	/backup/weird/backup			-	-		-
> :	local	rdiff	weird/local::/etc/key::.	/backup/weird/local			-	-		-
> :	home		rdiff	weird/home::/etc/key::.	/backup/weird/home				-	-		-
> :	med		rdiff	weird/med::/etc/key::.		/backup/weird/med				-	-		-
> :	mp3		rdiff	weird/mp3::/etc/key::.		/backup/weird/mp3				-	-		-
> :	mpeg		rdiff	weird/mpeg::/etc/key::.	/backup/weird/mpeg				-	-		-
> :	opt		rdiff	weird/opt::/etc/key::.		/backup/weird/opt				-	-		-
> :	root		rdiff	weird/root::/etc/key::.	/backup/weird/root				-	-		-
> :	usr		rdiff	weird/usr::/etc/key::.		/backup/weird/usr				-	-		-
> :	var		rdiff	weird/var::/etc/key::.		/backup/weird/var				-	zipLog	-
> :	etc		tar		///etc					/backup/conf/etc/etc_%d.%s		-z	-		lsf
> :	kernels	tar*	/usr/src//linux-*				/backup/weird/src/%f.%s		-j	-		lsf
> :	portage	rsync	/var/portage/packages		/backup/weird/portage			-	-		-
> :	kde		tar	~//.kde*						/backup/wonko/home/kde-%d.%s	-z	-		lsf
> :	dot		tar	~//.??*						/backup/extern/dot-%d.%s		-z	-		lsf
> 
> 
> At last, the disclaimer: Use it at your own risk! This is a script of some
> 800 lines of bash code some guy wrote, it messes around with LVM stuff,
> creates and removes temporary directories and even has an option to wipe
> the destination completely. I'm quite confident that no harm will be done,
> and I do not suggest you to take a backup before daring to try this script,
> but anyway.
> 
> Use the -d option first to test what actually would be going on, before you
> do a real backup!
> 
> Hope this is of good use to someone. For me it is, I will be using it on 
> several systems, and now I only have to make little changes in the config
> file for each system, but not at the script.
> 
> Suggestions and bug reports are welcome of course.
 
I have a question -- where would lvm put a snapshot and how could I pass
some list of excludes to rdiff-backup.  I have an lvm which is taking
all the PEs and a snapshot would take up lots of disk space -- or would
it.  Would I need some free pes to put the snapshot?

Thanks.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

[gentoo-user] Re: snackup

[Alex Schuster]

covici@ccs.covici.com writes:

> I have a question -- where would lvm put a snapshot and how could I
> pass some list of excludes to rdiff-backup.  I have an lvm which is
> taking all the PEs and a snapshot would take up lots of disk space --
> or would it.  Would I need some free pes to put the snapshot?

An LVM snapshot has to be in the same volume group as the LVM. If all your 
physical extends are full, this will not work I'm afraid.
But you can reduce the size of one LVM with lvreduce. Of course you have 
to resize the file system inside first. This is a little more complicated 
than extending the size, because you have to specify the size when 
reducing the file system and the LVM. And the file system has to be 
unmounted :(

Let's say you want to reduce your data partition of 15G to 10G:

  umount /dev/myvg/data
  fsck -f /dev/myvg/data
  resize2fs /dev/myvg/data 9G
  lvresize -L 10G /dev/myvg/data
  resize2fs /dev/myvg/data
  mount /dev/myvg/data
 
The 2nd resize2fs maximizes the size of the fs inside the LVM. I do not 
know (does anyone else?) if you could skip this and reduce it to 10G in 
the first resize2fs step. Just to be on the safe side I reduce it a little 
more, and let it adapt do the reduced LVM size afterwards.

The snapshot itself takes nearly no space at all - it only keeps the 
changes that occur in the LVM while the snapshot is in place. So it grows 
when you modify the LVM you snapshotted. When you do not much 
modifications, 15-20% is enough according to the lvcreate man page. And I 
think I had it much lower without problems. I would expect that it can be 
really small when you do not change the original LVm much. snackup uses 2G 
as default, change this with option -s. Of course, when you do large 
modifications, like creating larger files, this may be too small.

Excludes can be given with the -x option (multiple times). And have a look 
at the config template that snackup -T gives you. Near the bottom, the 
variable oXclude is defined. It is an array, just change it to your needs. 
it already excludes things like ccache, kdecache-* directories, 
*/tmp/portage, and the dreaded nepomuk directory fo KDE4 because this 
sometimes gets really REALLY large here.

snackup -x dip -x dap would exclude the stuff already pre-defined and dip 
and dap. If you want to exclude dip and dap only, call snackup -x "" -x 
dip -x dap. But I find it easier to adapt the oXclude array.

	Wonko

Re: [gentoo-user] Re: snackup

[covici]

Alex Schuster <wonko@wonkology.org> wrote:

> covici@ccs.covici.com writes:
> 
> > I have a question -- where would lvm put a snapshot and how could I
> > pass some list of excludes to rdiff-backup.  I have an lvm which is
> > taking all the PEs and a snapshot would take up lots of disk space --
> > or would it.  Would I need some free pes to put the snapshot?
> 
> An LVM snapshot has to be in the same volume group as the LVM. If all your 
> physical extends are full, this will not work I'm afraid.
> But you can reduce the size of one LVM with lvreduce. Of course you have 
> to resize the file system inside first. This is a little more complicated 
> than extending the size, because you have to specify the size when 
> reducing the file system and the LVM. And the file system has to be 
> unmounted :(
> 
> Let's say you want to reduce your data partition of 15G to 10G:
> 
>   umount /dev/myvg/data
>   fsck -f /dev/myvg/data
>   resize2fs /dev/myvg/data 9G
>   lvresize -L 10G /dev/myvg/data
>   resize2fs /dev/myvg/data
>   mount /dev/myvg/data
>  
> The 2nd resize2fs maximizes the size of the fs inside the LVM. I do not 
> know (does anyone else?) if you could skip this and reduce it to 10G in 
> the first resize2fs step. Just to be on the safe side I reduce it a little 
> more, and let it adapt do the reduced LVM size afterwards.
> 
> The snapshot itself takes nearly no space at all - it only keeps the 
> changes that occur in the LVM while the snapshot is in place. So it grows 
> when you modify the LVM you snapshotted. When you do not much 
> modifications, 15-20% is enough according to the lvcreate man page. And I 
> think I had it much lower without problems. I would expect that it can be 
> really small when you do not change the original LVm much. snackup uses 2G 
> as default, change this with option -s. Of course, when you do large 
> modifications, like creating larger files, this may be too small.
> 
> Excludes can be given with the -x option (multiple times). And have a look 
> at the config template that snackup -T gives you. Near the bottom, the 
> variable oXclude is defined. It is an array, just change it to your needs. 
> it already excludes things like ccache, kdecache-* directories, 
> */tmp/portage, and the dreaded nepomuk directory fo KDE4 because this 
> sometimes gets really REALLY large here.
> 
> snackup -x dip -x dap would exclude the stuff already pre-defined and dip 
> and dap. If you want to exclude dip and dap only, call snackup -x "" -x 
> dip -x dap. But I find it easier to adapt the oXclude array.

Thanks, very interesting and I will have a look.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

[gentoo-user] Re: snackup

[Alex Schuster]

I wrote:

> Still not on sourceforge, but here:
> http://www.wonkology.org/utils/snackup

Whoops, access denied. After a chmod o+r snackup, it is accessible now.

In case anyone already wrote me about this issue, I had lost my domain for 
two days, and all the e-mails going to wonkology.org.

	Wonko