Re: [gentoo-user] OT: Fighting bit rot

[Alan McKinnon <alan.mckinnon@gmail.com>]

On Tue, 08 Jan 2013 17:16:32 +0100
Florian Philipp <lists@binarywings.net> wrote:

> Am 08.01.2013 08:55, schrieb Alan McKinnon:
> > On Tue, 08 Jan 2013 08:27:51 +0100
> > Florian Philipp <lists@binarywings.net> wrote:
> > 
> [...]
> >>
> >> As I said above, the point is that I need to detect the error as
> >> long as I still have a valid backup. Professional archive
> >> solutions do this on their own but I'm looking for something
> >> suitable for desktop usage.
> > 
> > rsync might be able to give you something close to what you want
> > easily
> > 
> > Use the -n switch for an rsync between your originals and the last
> > backup copy, and mail the output to yourself. Parse it looking for
> > ">" and "<" symbols and investigate why the file changed.
> > 
> > This strikes me as being a very easy solution that you could use
> > reliably with a suitable combination of options. 
> > 
> > 
> 
> Hmm, good idea, albeit similar to the `md5sum -c`. Either tool leaves
> you with the problem of distinguishing between legitimate changes
> (i.e. a user wrote to the file) and decay.
> 
> When you have completely static content, md5sum, rsync and friends are
> sufficient. But if you have content that changes from time to time,
> the number of false-positives would be too high. In this case, I
> think you could easily distinguish by comparing both file content and
> time stamps.
> 
> Now, that of course introduces the problem that decay could occur in
> the same time frame as a legitimate change, thus masking the decay. To
> reduce this risk, you have to reduce the checking interval.


I think your basic problem is that you are trying to detect a rare
event (corruption) that looks exactly like a common event (edits you
intended to make)

I don't know how to tell these apart except by somehow recording which
files have been written to - inotify is useful for this - and removing
those from the list of things rsync says have changed.

All of which leads to a massively complex lump of code that is sure to
cause many more problems than it is designed to solve....

I'm afraid I don't have any real solution to offer.



-- 
Alan McKinnon
alan.mckinnon@gmail.com