From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1OA1y2-0007lt-5E for garchives@archives.gentoo.org; Thu, 06 May 2010 14:22:22 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id B2A66E07D5; Thu, 6 May 2010 14:22:07 +0000 (UTC) Received: from smtpout.karoo.kcom.com (smtpout.karoo.kcom.com [212.50.160.34]) by pigeon.gentoo.org (Postfix) with ESMTP id 371DDE07D5 for ; Thu, 6 May 2010 14:22:06 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.52,340,1270422000"; d="scan'208";a="193760755" Received: from 213-152-39-90.dsl.eclipse.net.uk (HELO compaq.stroller.uk.eu.org) ([213.152.39.90]) by smtpout.karoo.kcom.com with ESMTP; 06 May 2010 15:22:05 +0100 Received: from funf.stroller.uk.eu.org (funf.stroller.uk.eu.org [192.168.1.71]) by compaq.stroller.uk.eu.org (Postfix) with ESMTP id DAB8E3E116 for ; Thu, 6 May 2010 15:22:02 +0100 (BST) Message-Id: <20F546DD-F224-40CD-A4C0-7782D1D1C2A4@stellar.eclipse.co.uk> From: Stroller To: gentoo-user@lists.gentoo.org In-Reply-To: <1273103468.20354.45.camel@localhost> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: [gentoo-user] kernel notification of file system changes Date: Thu, 6 May 2010 15:22:03 +0100 References: <1273042474.20354.17.camel@localhost> <07250F7A-39A9-4417-A0E8-CBCD4E8CDDC6@stellar.eclipse.co.uk> <4BE1A9C5.5090709@f_philipp.fastmail.net> <1273103468.20354.45.camel@localhost> X-Mailer: Apple Mail (2.936) X-Archives-Salt: 842b38c4-e098-448b-a856-818e9170a841 X-Archives-Hash: 64fc5705094fc604e0309e1befefeb95 On 6 May 2010, at 00:51, Iain Buchanan wrote: > ... >>> ... Inotify has two drawbacks which make it hard or even impossible >>> to use for Iain's use case: >>> >>> a) It does not work recursively which means that you have to >>> create a >>> new handle for each subdirectory. Of course, this only means more >>> work >>> for the programmer but there is also the problem that >> >> Pardon me. I assumed that "files inside the directory" meant that foo >> would be be changed when foo/bar changed, thus monitoring grunt would >> reflect changes in grunt/foo/bar. I overlooked that a directory is >> not >> a file. > > isn't it? I thought a directory was just a file containing names (or > inodes) of other files? Which would explain why monitoring grunt > wouldn't show changes in grunt/foo/bar, since the directory/file > called > grunt remains the same (ie. contains the same list of inodes) even if > grunt/foo/bar changes. But if bar changes, the modification time of foo changes, which is a change in grunt. Apparently not. :( > Let me tell you what I actually want to do, which I may not have made > clear originally: > > I want to backup root to an external drive (or that could be rephrased > as "I want to backup any mount to any other mount), such that: > 1. My backup is an hour or so out of date (at most) > 2. I don't need to copy the entire filesystem every time > > To do that, I could either: > * Run rsync every hour over the entire filesystem (I'm doing this > now with ionice, takes about 10 minutes when there are no > changes) > * Use some file notification monitor to tell me which file was > just changed, and only rsync that file > > The problems with rsync is that during the rsync process, the > filesystem > is changing, so I will end up with a slightly inconsistent backup. I really appreciate where you're coming from, as I have the same concerns regarding the backing up of emails. I have systems with gigs of email on them - belonging to myself and to others - and in the time taken to rsync the lot, more mail is sure to have arrived. All one can do is rsync regularly - then you're only an hour or so out of date. > I'm considering LVM for it's snapshot capability, but I'd still have > to > rsync root. Indeed, snapshotting is ideal. > I would prefer a file notification method as well, so I can > just rsync the file that just changed. I think you're unrealistic to expect your backup to be uptodate to within minutes or seconds. I don't really see the benefit of that, to be honest. As far as I'm concerned I don't really care if /bin, /sbin &c are backed up or not, because I can reinstall Gentoo in a couple of hours. Here /etc does not generally change on a daily basis - more like a backup from last month would be fine! Your milage may vary, but I really don't see a day-old backup of /etc as a problem. So we are really only down to /home which needs backing up as frequently as hourly. I assume that you're aware that "live" backups require multiple copies. This is the RAID-is-not-a-backup argument - if a file becomes corrupt you don't want it overwriting your backup copy. Having said that you're "unrealistic" and that "I don't see the benefit", I do appreciate where you're coming from. It's the answer that us perfectionists want - it's the answer that all Linux users want. And thus, if there were a clean and elegant solution then it would be a well-trodden path. I think that if there was a clean and elegant solution to "the perfect backup" everyone would know what it was, and everyone would be using it, and you wouldn't even have to ask. This is why I stopped reading your thread early. Having said all that, if the stumbling block is snapshotting of your / partition, you might take a look at btrfs - it looks like it allows you to do this. brrfs is probably good enough now for everyday use, as long as you have a backup, and I think you could reasonably easily backup your snapshots on some regular basis to some other "safe" filesystem, such as ext[234]. I find it slightly interesting that none of the ionotify solutions do what you want. The point of such tools is to "do something when a file is written or changed", so they would seem an ideal fit to solve the "perfect backup problem" - in that context it would seem trivial to write a kernel module / userspace app to write a copy of each changed file to /path/to/backup/$UID and record some information about the original filename, $UID and the time of change. I imagine that some types of changes to files - such as bittorrent downloads, in which a binary file may change like A___E -> A_C_E -> ABCDE - are more difficult to handle than others, so this will be left to the reader as a practical programming exercise. Stroller.