Re: [gentoo-user] kernel notification of file system changes

[Stroller <stroller@stellar.eclipse.co.uk>]

On 6 May 2010, at 00:51, Iain Buchanan wrote:
> ...
>>> ... Inotify has two drawbacks which make it hard or even impossible
>>> to use for Iain's use case:
>>>
>>> a) It does not work recursively which means that you have to  
>>> create a
>>> new handle for each subdirectory. Of course, this only means more  
>>> work
>>> for the programmer but there is also the problem that
>>
>> Pardon me. I assumed that "files inside the directory" meant that foo
>> would be be changed when foo/bar changed, thus monitoring grunt would
>> reflect changes in grunt/foo/bar. I overlooked that a directory is  
>> not
>> a file.
>
> isn't it?  I thought a directory was just a file containing names (or
> inodes) of other files?  Which would explain why monitoring grunt
> wouldn't show changes in grunt/foo/bar, since the directory/file  
> called
> grunt remains the same (ie. contains the same list of inodes) even if
> grunt/foo/bar changes.

But if bar changes, the modification time of foo changes, which is a  
change in grunt. Apparently not. :(

> Let me tell you what I actually want to do, which I may not have made
> clear originally:
>
> I want to backup root to an external drive (or that could be rephrased
> as "I want to backup any mount to any other mount), such that:
>     1. My backup is an hour or so out of date (at most)
>     2. I don't need to copy the entire filesystem every time
>
> To do that, I could either:
>      * Run rsync every hour over the entire filesystem (I'm doing this
>        now with ionice, takes about 10 minutes when there are no
>        changes)
>      * Use some file notification monitor to tell me which file was
>        just changed, and only rsync that file
>
> The problems with rsync is that during the rsync process, the  
> filesystem
> is changing, so I will end up with a slightly inconsistent backup.

I really appreciate where you're coming from, as I have the same  
concerns regarding the backing up of emails. I have systems with gigs  
of email on them - belonging to myself and to others - and in the time  
taken to rsync the lot, more mail is sure to have arrived. All one can  
do is rsync regularly - then you're only an hour or so out of date.

> I'm considering LVM for it's snapshot capability, but I'd still have  
> to
> rsync root.

Indeed, snapshotting is ideal.

>  I would prefer a file notification method as well, so I can
> just rsync the file that just changed.

I think you're unrealistic to expect your backup to be uptodate to  
within minutes or seconds. I don't really see the benefit of that, to  
be honest. As far as I'm concerned I don't really care if /bin, /sbin  
&c are backed up or not, because I can reinstall Gentoo in a couple of  
hours. Here /etc does not generally change on a daily basis - more  
like a backup from last month would be fine! Your milage may vary, but  
I really don't see a day-old backup of /etc as a problem. So we are  
really only down to /home which needs backing up as frequently as  
hourly.

I assume that you're aware that "live" backups require multiple  
copies. This is the RAID-is-not-a-backup argument - if a file becomes  
corrupt you don't want it overwriting your backup copy.

Having said that you're "unrealistic" and that "I don't see the  
benefit", I do appreciate where you're coming from. It's the answer  
that us perfectionists want - it's the answer that all Linux users  
want. And thus, if there were a clean and elegant solution then it  
would be a well-trodden path. I think that if there was a clean and  
elegant solution to "the perfect backup" everyone would know what it  
was, and everyone would be using it, and you wouldn't even have to  
ask. This is why I stopped reading your thread early.

Having said all that, if the stumbling block is snapshotting of your /  
partition, you might take a look at btrfs - it looks like it allows  
you to do this. brrfs is probably good enough now for everyday use, as  
long as you have a backup, and I think you could reasonably easily  
backup your snapshots on some regular basis to some other "safe"  
filesystem, such as ext[234].

I find it slightly interesting that none of the ionotify solutions do  
what you want. The point of such tools is to "do something when a file  
is written or changed", so they would seem an ideal fit to solve the  
"perfect backup problem" - in that context it would seem trivial to  
write a kernel module / userspace app to write a copy of each changed  
file to /path/to/backup/$UID and record some information about the  
original filename, $UID and the time of change. I imagine that some  
types of changes to files - such as bittorrent downloads, in which a  
binary file may change like A___E -> A_C_E -> ABCDE - are more  
difficult to handle than others, so this will be left to the reader as  
a practical programming exercise.

Stroller.