From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1OASvC-0001ty-QZ for garchives@archives.gentoo.org; Fri, 07 May 2010 19:09:15 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id E689EE07D7; Fri, 7 May 2010 19:08:28 +0000 (UTC) Received: from wlym.com (wlym.com [66.135.63.43]) by pigeon.gentoo.org (Postfix) with ESMTP id C1A2DE07D7 for ; Fri, 7 May 2010 19:08:28 +0000 (UTC) Received: from ccs.covici.com (pool-96-247-205-104.clppva.fios.verizon.net [96.247.205.104]) (authenticated bits=128) by wlym.com (8.14.3/8.14.3/Debian-5) with ESMTP id o47J5mFc016332 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Fri, 7 May 2010 14:05:50 -0500 Received: from ccs.covici.com (localhost [127.0.0.1]) by ccs.covici.com (8.14.4/8.14.2) with ESMTP id o47J8M6J021090 for ; Fri, 7 May 2010 15:08:25 -0400 To: gentoo-user@lists.gentoo.org Subject: Re: snackup (was: Re: [gentoo-user] backup to a cold-swap drive) In-reply-to: <201005072023.54772.wonko@wonkology.org> References: <1272418910.30160.4057.camel@localhost> <201004291644.34796.wonko@wonkology.org> <1272888790.30160.4194.camel@localhost> <201005072023.54772.wonko@wonkology.org> Comments: In-reply-to Alex Schuster message dated "Fri, 07 May 2010 20:23:53 +0200." X-Mailer: MH-E 8.2; nmh 1.3; GNU Emacs 23.1.1 Date: Fri, 07 May 2010 15:08:22 -0400 Message-ID: <21088.1273259302@ccs.covici.com> From: covici@ccs.covici.com Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Archives-Salt: c31e8785-b808-413f-8e85-c3e51abb0f85 X-Archives-Hash: 92fb50f2fa3dce920e51600d125adb60 Alex Schuster wrote: > Iain Buchanan writes: > > > On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote: > > > > using this script, adapted to their needs, I started to rewrite it in > > > a way that it reads a config file, and no modification of the script > > > itself is necessary. If anyone is interested, send me an email. > > > > interested! So is it on sourceforge yet ;) > > Sorry, it took a little longer. I was ill, and then these scripts tend to > grow and grow until I am satisfied with them. It's still not perfect, but > I think at the moment it does what it should do. > > Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup > The name is silly, but I kinda like it now. A backup utilizing LVM > snapshots... snapshot backup... snackup! > > It needs a config file that is looked up in some default locations, or can > be specified with the -c option. Use option -T to generate a template that > has information on the syntax and some examples. In short: > > The config file defines targets you give as arguments to snackup. So > 'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz > file of your /etc. The config file is sourced, so you can put any bash > stuff you want into there, like some shell functions you want to have > executed. For example, I want my /var/log/portage/*.log files to be > compressed before I backup my /var partition. > > A target is started by a colon, followed by name, type, source and > destination (and optionally more), separated by one or more tab > characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only > tested dd, tar and rdiff so far, though. The source directory may be > prefixed by the volume group and LVM of that partition in order to create > an lvm snapshot first. You may add a LUKS key if the partition is > encrypted. The source may contain wildcards. In that case, all matching > files are backed up to the destination. If the type has a * appended, all > matching files are backed up individually. Some examples: > > : tar kde .kde* /backup/kde.%s > > Target 'kde' will backup all your .kde* directories to /backup/kde.tar > (the %s is replaced by a suffix, usually tar). You need to be in your home > directory, otherwise the path will not be found. This is not necessary if > you use ~//.kde* instead. > > : kernels tar* /usr/src//linux-* /backup/src/%f.%s -j - lsf > > Target 'kernels' will backup your /usr/src/linux-* directories > individually - one tar file for each (note the * appendended to the tar > target type). The -z option will be passed to the tar command, and the > suffix is changed to tgz accordingly. > The // denotes that the backup should be done locally from the directory > left to the //. Otherwise, the full path would be included in the tar > file. > > : tar etc /etc /backup/etc_%d.%s -j > > Target 'etc' will tar your /etc directory, compressed with bzip2. The %d > will be replaced by the current date. You can change the date format by > re-defining a date() shell function. > You will get a message about tar removing the leading /, you can avoid it > by replacing /etc by ///etc, meaning that your local directroy will be /. > > : boot dd /dev/sda5 /dev/sdb5 bs=32M > > Target 'boot' will backup a boot partition with dd. Options like the > bs=32M are added directly to the dd command. > > : home rsync system/home::. /backup/home/ > > Target 'home' will create a LVM snapshot of the /dev/system/home logical > volume, mount it and back it up with rdiff-backup. The '.' could also be a > / or left off. > > : var rsync system/var::/ /backup/var/ - ziplog > > Similar, but the command ziplog will be executed before the backup, > compressing some log files. It must be defined in the config file. > > The script has some options: > > -c file location of config file > -C clear destination first > -d dummy mode, just show what would be done > -f force backup (the initial rdiff-backup may need this) > -h show this help > -l output to log file, too > -L log file (default:/home/wonko/log/snackup.log); may be a > directory (add a trailing slash) to create target-specific logs > -n l use nice level l (default:10, 0 to turn off) > -N l use ionice level l (default:3, 0 to turn off) > -o extra options that will added to the actual backup command > -s size size of LVM snapshot (default:2G) > -S char replace tab as delimiter for targets in config file > -T output config file template > -v verbose output; may be given multiple times > 1: some extra output; 2: add -v to commands; 3: set -xv > -V output version information and exit > > Most important are: > -c to specify the location of the config file > -d to output what would be done, you will see the final tar / rdiff- > backup / whatever command with all its options. > -T to see a template. Save it with snackup -T > ~/.snackup and then > modify your ~/.snackup file. > -o to add whichever options you like to the final command. These (and -j > and -z) override options already defined in the config file. > > Too complicated? Overkill? Maybe. But I like it, and now I do not have to > write another script for every system I want to backup. And I like the LVm > snapshotting to be done automatically. It's not that hard, but it's > annoying to do it by hand, even more as all my volumes are encrypted. > > These are my own targets: > > : boot dd /dev/boot /dev/sdb5 - - - > : backup rdiff weird/backup::/etc/key::. /backup/weird/backup - - - > : local rdiff weird/local::/etc/key::. /backup/weird/local - - - > : home rdiff weird/home::/etc/key::. /backup/weird/home - - - > : med rdiff weird/med::/etc/key::. /backup/weird/med - - - > : mp3 rdiff weird/mp3::/etc/key::. /backup/weird/mp3 - - - > : mpeg rdiff weird/mpeg::/etc/key::. /backup/weird/mpeg - - - > : opt rdiff weird/opt::/etc/key::. /backup/weird/opt - - - > : root rdiff weird/root::/etc/key::. /backup/weird/root - - - > : usr rdiff weird/usr::/etc/key::. /backup/weird/usr - - - > : var rdiff weird/var::/etc/key::. /backup/weird/var - zipLog - > : etc tar ///etc /backup/conf/etc/etc_%d.%s -z - lsf > : kernels tar* /usr/src//linux-* /backup/weird/src/%f.%s -j - lsf > : portage rsync /var/portage/packages /backup/weird/portage - - - > : kde tar ~//.kde* /backup/wonko/home/kde-%d.%s -z - lsf > : dot tar ~//.??* /backup/extern/dot-%d.%s -z - lsf > > > At last, the disclaimer: Use it at your own risk! This is a script of some > 800 lines of bash code some guy wrote, it messes around with LVM stuff, > creates and removes temporary directories and even has an option to wipe > the destination completely. I'm quite confident that no harm will be done, > and I do not suggest you to take a backup before daring to try this script, > but anyway. > > Use the -d option first to test what actually would be going on, before you > do a real backup! > > Hope this is of good use to someone. For me it is, I will be using it on > several systems, and now I only have to make little changes in the config > file for each system, but not at the script. > > Suggestions and bug reports are welcome of course. I have a question -- where would lvm put a snapshot and how could I pass some list of excludes to rdiff-backup. I have an lvm which is taking all the PEs and a snapshot would take up lots of disk space -- or would it. Would I need some free pes to put the snapshot? Thanks. -- Your life is like a penny. You're going to lose it. The question is: How do you spend it? John Covici covici@ccs.covici.com