[gentoo-user] kernel notification of file system changes

[gentoo-user] kernel notification of file system changes

[Iain Buchanan]

Hi,

I'm looking for some kernel-based notification of changes to my file
system.  I've been looking at inotify, but it's not exactly what I want.

Basically I want to know if _any_ write occurs anywhere.  I don't want
to register a whole bunch of files to watch, I just want to watch an
entire mount.  When a file is changed (ie. a write operation occurs), I
then want to add that file or fd to a list in RAM.  That's all.

I know this may be a lot of data, considering streams and devices, but I
can filter out /dev, /proc, etc. and just focus on "real" files.

Is there anything that can do this?

thanks :)
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

I hate it when my foot falls asleep during the day cause that means
it's going to be up all night.
		-- Steven Wright

Re: [gentoo-user] kernel notification of file system changes

[Bill Kenworthy]

Can the older dnotify do what you want? - it monitors files differently
to inotify.  There is also gammin/fam.

BillK


On Wed, 2010-05-05 at 16:24 +0930, Iain Buchanan wrote:
> Hi,
> 
> I'm looking for some kernel-based notification of changes to my file
> system.  I've been looking at inotify, but it's not exactly what I want.
> 
> Basically I want to know if _any_ write occurs anywhere.  I don't want
> to register a whole bunch of files to watch, I just want to watch an
> entire mount.  When a file is changed (ie. a write operation occurs), I
> then want to add that file or fd to a list in RAM.  That's all.
> 
> I know this may be a lot of data, considering streams and devices, but I
> can filter out /dev, /proc, etc. and just focus on "real" files.
> 
> Is there anything that can do this?
> 
> thanks :)

Re: [gentoo-user] kernel notification of file system changes

[Stroller]

On 5 May 2010, at 07:54, Iain Buchanan wrote:
> ...
> I'm looking for some kernel-based notification of changes to my file
> system.  I've been looking at inotify, but it's not exactly what I  
> want.
>
> Basically I want to know if _any_ write occurs anywhere.  I don't want
> to register a whole bunch of files to watch, I just want to watch an
> entire mount.

man inotify(7):
... When a directory is monitored, inotify will return events for the  
directory itself, and for files inside the directory.


Am I missing something?


This article was posted to a different froup recently:
http://www.ibm.com/developerworks/linux/library/l-inotify/index.html
It looks interesting.


Stroller.

Re: [gentoo-user] kernel notification of file system changes

[Helmut Jarausch]

On  5 May, Iain Buchanan wrote:
> Hi,
> 
> I'm looking for some kernel-based notification of changes to my file
> system.  I've been looking at inotify, but it's not exactly what I want.
> 
> Basically I want to know if _any_ write occurs anywhere.  I don't want
> to register a whole bunch of files to watch, I just want to watch an
> entire mount.  When a file is changed (ie. a write operation occurs), I
> then want to add that file or fd to a list in RAM.  That's all.
> 
> I know this may be a lot of data, considering streams and devices, but I
> can filter out /dev, /proc, etc. and just focus on "real" files.
> 
> Is there anything that can do this?
> 

Might be I've just asked a similar question on the ZSH mailing list.
Please have a look at inotifywatch from the sys-fs/inotify-tools
package. It can watch a directory tree recursively.

I hope this helps,
Helmut.

Re: [gentoo-user] kernel notification of file system changes

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1534 bytes --]

Am 05.05.2010 15:34, schrieb Stroller:
> 
> On 5 May 2010, at 07:54, Iain Buchanan wrote:
>> ...
>> I'm looking for some kernel-based notification of changes to my file
>> system.  I've been looking at inotify, but it's not exactly what I want.
>>
>> Basically I want to know if _any_ write occurs anywhere.  I don't want
>> to register a whole bunch of files to watch, I just want to watch an
>> entire mount.
> 
> man inotify(7):
> ... When a directory is monitored, inotify will return events for the
> directory itself, and for files inside the directory.
> 
> 
> Am I missing something?
> 
> 
> This article was posted to a different froup recently:
> http://www.ibm.com/developerworks/linux/library/l-inotify/index.html
> It looks interesting.
> 
> 
> Stroller.
> 

To repeat my comment on Iain's original "backup to a cold-swap drive"
thread, Inotify has two drawbacks which make it hard or even impossible
to use for Iain's use case:

a) It does not work recursively which means that you have to create a
new handle for each subdirectory. Of course, this only means more work
for the programmer but there is also the problem that

b) As far as I know, Inotify does not scale very good, at least not good
enough to monitor a whole system. /proc/sys/fs/inotify/max_user_watches
is 65535 on my system.

On the other hand, I've never tried to increase that limit and just let
it run on a deep directory structure. Who knows, maybe it actually works.

Hope this helps,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] kernel notification of file system changes

[Stroller]

On 5 May 2010, at 18:24, Florian Philipp wrote:
>> ...
>> man inotify(7):
>> ... When a directory is monitored, inotify will return events for the
>> directory itself, and for files inside the directory.
>> ...
>
> To repeat my comment on Iain's original "backup to a cold-swap drive"
> thread ...

Sorry, I started ignoring that almost immediately it was posted. He  
rejected too quickly too many workable solutions to basically  
functional backup. Perhaps Iain is a perfectionist, but I did not wish  
to follow the thread.

> ... Inotify has two drawbacks which make it hard or even impossible
> to use for Iain's use case:
>
> a) It does not work recursively which means that you have to create a
> new handle for each subdirectory. Of course, this only means more work
> for the programmer but there is also the problem that

Pardon me. I assumed that "files inside the directory" meant that foo  
would be be changed when foo/bar changed, thus monitoring grunt would  
reflect changes in grunt/foo/bar. I overlooked that a directory is not  
a file.

Stroller.

Re: [gentoo-user] kernel notification of file system changes

[Iain Buchanan]

On Wed, 2010-05-05 at 17:02 +0200, Helmut Jarausch wrote:

> Might be I've just asked a similar question on the ZSH mailing list.
> Please have a look at inotifywatch from the sys-fs/inotify-tools
> package. It can watch a directory tree recursively.

it does look interesting, thanks.  I would still run into the directory
limit if I wanted to watch something large like /

$ sudo find  / -xdev -type d | wc -l
71168

"...The default maximum is 8192; it can be increased by writing
to /proc/sys/fs/inotify/max_user_watches."

but it's an angle to follow.  I wonder how max_user_watches would handle
being 100k or more... no doubt you just need some RAM?!

thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

<drobbins> we should send him a commemorative gentoo crack pipe for all
 his contributions to this project

Re: [gentoo-user] kernel notification of file system changes

[Iain Buchanan]

On Wed, 2010-05-05 at 15:12 +0800, Bill Kenworthy wrote:
> Can the older dnotify do what you want? - it monitors files differently
> to inotify.  There is also gammin/fam.

dnotify locks the files or directories you want to watch, so it would
prevent external media from being unmounted.  dnotify also uses a file
descripter per watched item, which could get interesting for large
amounts of watches!

I'm not sure about FAM, but Gamin uses inotify or dnotify anyway (in
Linux).  I think they're all designed in a similar way: you have to
register a whole bunch of files or directories to watch.

thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

He that is giddy thinks the world turns round.
		-- William Shakespeare, "The Taming of the Shrew"

Re: [gentoo-user] kernel notification of file system changes

[Volker Armin Hemmann]

On Donnerstag 06 Mai 2010, Iain Buchanan wrote:
> On Wed, 2010-05-05 at 15:12 +0800, Bill Kenworthy wrote:
> > Can the older dnotify do what you want? - it monitors files differently
> > to inotify.  There is also gammin/fam.
> 
> dnotify locks the files or directories you want to watch, so it would
> prevent external media from being unmounted.  dnotify also uses a file
> descripter per watched item, which could get interesting for large
> amounts of watches!
> 
> I'm not sure about FAM, but Gamin uses inotify or dnotify anyway (in
> Linux).  I think they're all designed in a similar way: you have to
> register a whole bunch of files or directories to watch.
> 
> thanks,

oh god... fam... that crap caused me so much pain over the years. This bug 
ridden zombie is still around?

Re: [gentoo-user] kernel notification of file system changes

[Iain Buchanan]

On Wed, 2010-05-05 at 18:35 +0100, Stroller wrote:
> On 5 May 2010, at 18:24, Florian Philipp wrote:
> >> ...
> >> man inotify(7):
> >> ... When a directory is monitored, inotify will return events for the
> >> directory itself, and for files inside the directory.
> >> ...
> >
> > To repeat my comment on Iain's original "backup to a cold-swap drive"
> > thread ...
> 
> Sorry, I started ignoring that almost immediately it was posted.

ooohh, ouch! :)

>  He  
> rejected too quickly too many workable solutions to basically  
> functional backup. Perhaps Iain is a perfectionist, but I did not wish  
> to follow the thread.

Perhaps I am a bit of a perfectionist, but I think you misunderstood my
aim.  I rejected 3 options straight away (dd, gparted, and Ghost4Linux)
because they're not designed for backing up a live filesystem in a
"change only" fashion ("intelligent" is the word I used).

Beyond that I didn't reject anything else that anyone mentioned.

> > ... Inotify has two drawbacks which make it hard or even impossible
> > to use for Iain's use case:
> >
> > a) It does not work recursively which means that you have to create a
> > new handle for each subdirectory. Of course, this only means more work
> > for the programmer but there is also the problem that
> 
> Pardon me. I assumed that "files inside the directory" meant that foo  
> would be be changed when foo/bar changed, thus monitoring grunt would  
> reflect changes in grunt/foo/bar. I overlooked that a directory is not  
> a file.

isn't it?  I thought a directory was just a file containing names (or
inodes) of other files?  Which would explain why monitoring grunt
wouldn't show changes in grunt/foo/bar, since the directory/file called
grunt remains the same (ie. contains the same list of inodes) even if
grunt/foo/bar changes.

Let me tell you what I actually want to do, which I may not have made
clear originally:

I want to backup root to an external drive (or that could be rephrased
as "I want to backup any mount to any other mount), such that:
     1. My backup is an hour or so out of date (at most)
     2. I don't need to copy the entire filesystem every time

To do that, I could either:
      * Run rsync every hour over the entire filesystem (I'm doing this
        now with ionice, takes about 10 minutes when there are no
        changes)
      * Use some file notification monitor to tell me which file was
        just changed, and only rsync that file

The problems with rsync is that during the rsync process, the filesystem
is changing, so I will end up with a slightly inconsistent backup.  If I
use some notification method that tells me a file has changed, I can
greatly reduce any inconsistency, and I reduce my hour down to seconds
or minutes, depending on how much changes at any one time.

I'm considering LVM for it's snapshot capability, but I'd still have to
rsync root.  I would prefer a file notification method as well, so I can
just rsync the file that just changed.

So far all the file monitoring tools are based on individual files (even
the recursive ones), and you eventually reach a system limit.

Thanks, and willing to listen to any ideas from anyone (except
Stroller :p )

-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Usually, when a lot of men get together, it's called a war.
		-- Mel Brooks, "The Listener"

Re: [gentoo-user] kernel notification of file system changes

[Iain Buchanan]

On Thu, 2010-05-06 at 01:33 +0200, Volker Armin Hemmann wrote:

> oh god... fam... that crap caused me so much pain over the years. This bug 
> ridden zombie is still around?

thanks for the heads-up - I guess I should leave FAM to plan B?
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

I am a friend of the working man, and I would rather be his friend
than be one.
		-- Clarence Darrow

Re: [gentoo-user] kernel notification of file system changes

[Volker Armin Hemmann]

On Donnerstag 06 Mai 2010, Iain Buchanan wrote:
> On Thu, 2010-05-06 at 01:33 +0200, Volker Armin Hemmann wrote:
> > oh god... fam... that crap caused me so much pain over the years. This
> > bug ridden zombie is still around?
> 
> thanks for the heads-up - I guess I should leave FAM to plan B?

Plan Z really.  I can't even remember how many times I had to kill that pig to 
unmount something - or because it suddenly decided to eat CPU cycles like a 
fat stoner breakfast cereals.

Re: [gentoo-user] kernel notification of file system changes

[Iain Buchanan]

On Thu, 2010-05-06 at 08:54 +0930, Iain Buchanan wrote:

> but it's an angle to follow.  I wonder how max_user_watches would handle
> being 100k or more... no doubt you just need some RAM?!
> 
> thanks,

To answer my own questions, I'm now trying this:

# echo 100000 >/proc/sys/fs/inotify/max_user_watches

$ time sudo find  / -xdev -type d | sudo inotifywatch -v -t 1 -e modify,attrib,move,create,delete,delete_self,unmount --fromfile -
Establishing watches...
Total of 71169 watches.
Finished establishing watches, now collecting statistics.
Will listen for events for 1 seconds.
total  modify  filename
6      6       /tmp/
2      2       /dev/

real	0m3.177s
user	0m0.768s
sys	0m1.378s

This sets up a watch on all directories under / that aren't part of
another filesystem, and then exits after one second.  It's quite fast :)

The idea, off the top of my head, would be this:
     1. inotifywatch as above but without the time restriction
     2. wait for it to finishing "setting up"
     3. rsync the whole directory structure to the backup
     4. continuously do this loop:
             1. get list of changes from inotifywatch
             2. rsync those changes

Unfortunately inotifywatch only returns output on ctrl-c, which I don't
want to do or you loose anything changed between instances.  This could
be changed to another signal, no doubt.

How does that sound for a continuous running backup?  This is starting
to stray OT from Gentoo, but your thoughts are welcome :)
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

One cannot make an omelette without breaking eggs -- but it is amazing
how many eggs one can break without making a decent omelette.
		-- Professor Charles P. Issawi

Re: [gentoo-user] kernel notification of file system changes

[Stroller]

On 6 May 2010, at 00:51, Iain Buchanan wrote:
> ...
>>> ... Inotify has two drawbacks which make it hard or even impossible
>>> to use for Iain's use case:
>>>
>>> a) It does not work recursively which means that you have to  
>>> create a
>>> new handle for each subdirectory. Of course, this only means more  
>>> work
>>> for the programmer but there is also the problem that
>>
>> Pardon me. I assumed that "files inside the directory" meant that foo
>> would be be changed when foo/bar changed, thus monitoring grunt would
>> reflect changes in grunt/foo/bar. I overlooked that a directory is  
>> not
>> a file.
>
> isn't it?  I thought a directory was just a file containing names (or
> inodes) of other files?  Which would explain why monitoring grunt
> wouldn't show changes in grunt/foo/bar, since the directory/file  
> called
> grunt remains the same (ie. contains the same list of inodes) even if
> grunt/foo/bar changes.

But if bar changes, the modification time of foo changes, which is a  
change in grunt. Apparently not. :(

> Let me tell you what I actually want to do, which I may not have made
> clear originally:
>
> I want to backup root to an external drive (or that could be rephrased
> as "I want to backup any mount to any other mount), such that:
>     1. My backup is an hour or so out of date (at most)
>     2. I don't need to copy the entire filesystem every time
>
> To do that, I could either:
>      * Run rsync every hour over the entire filesystem (I'm doing this
>        now with ionice, takes about 10 minutes when there are no
>        changes)
>      * Use some file notification monitor to tell me which file was
>        just changed, and only rsync that file
>
> The problems with rsync is that during the rsync process, the  
> filesystem
> is changing, so I will end up with a slightly inconsistent backup.

I really appreciate where you're coming from, as I have the same  
concerns regarding the backing up of emails. I have systems with gigs  
of email on them - belonging to myself and to others - and in the time  
taken to rsync the lot, more mail is sure to have arrived. All one can  
do is rsync regularly - then you're only an hour or so out of date.

> I'm considering LVM for it's snapshot capability, but I'd still have  
> to
> rsync root.

Indeed, snapshotting is ideal.

>  I would prefer a file notification method as well, so I can
> just rsync the file that just changed.

I think you're unrealistic to expect your backup to be uptodate to  
within minutes or seconds. I don't really see the benefit of that, to  
be honest. As far as I'm concerned I don't really care if /bin, /sbin  
&c are backed up or not, because I can reinstall Gentoo in a couple of  
hours. Here /etc does not generally change on a daily basis - more  
like a backup from last month would be fine! Your milage may vary, but  
I really don't see a day-old backup of /etc as a problem. So we are  
really only down to /home which needs backing up as frequently as  
hourly.

I assume that you're aware that "live" backups require multiple  
copies. This is the RAID-is-not-a-backup argument - if a file becomes  
corrupt you don't want it overwriting your backup copy.

Having said that you're "unrealistic" and that "I don't see the  
benefit", I do appreciate where you're coming from. It's the answer  
that us perfectionists want - it's the answer that all Linux users  
want. And thus, if there were a clean and elegant solution then it  
would be a well-trodden path. I think that if there was a clean and  
elegant solution to "the perfect backup" everyone would know what it  
was, and everyone would be using it, and you wouldn't even have to  
ask. This is why I stopped reading your thread early.

Having said all that, if the stumbling block is snapshotting of your /  
partition, you might take a look at btrfs - it looks like it allows  
you to do this. brrfs is probably good enough now for everyday use, as  
long as you have a backup, and I think you could reasonably easily  
backup your snapshots on some regular basis to some other "safe"  
filesystem, such as ext[234].

I find it slightly interesting that none of the ionotify solutions do  
what you want. The point of such tools is to "do something when a file  
is written or changed", so they would seem an ideal fit to solve the  
"perfect backup problem" - in that context it would seem trivial to  
write a kernel module / userspace app to write a copy of each changed  
file to /path/to/backup/$UID and record some information about the  
original filename, $UID and the time of change. I imagine that some  
types of changes to files - such as bittorrent downloads, in which a  
binary file may change like A___E -> A_C_E -> ABCDE - are more  
difficult to handle than others, so this will be left to the reader as  
a practical programming exercise.

Stroller.