[gentoo-user] File system meta-data indexer / checker

[gentoo-user] File system meta-data indexer / checker

[Ciprian Dorin Craciun]

    Hello all!

    While struggling with managing various old backups --- just
imagine 10 or so copies of almost the same content, some with `rsync`,
some with `rdiff-backup`, yet some others on plain ISO's, all over a
range of a few years --- I stumbled upon the following missing piece
in the Linux tools ecosystem:  a file-system crawler that records
**only** meta-data, like all the info available through `stat`, plus
an assortment of hashes of the files (at least MD5 and SHA-1,2
family), and optionally ACL's and extended attributes.  Thus I was
wondering if someone knows a tool that fits this description.


    I must say I've tried to do my homework, and below are a few tools
that come close, but not quite...

    (A) `rdup` is probably the closest to what I'm searching.  However
there are a few issues:
    * it's output format is not very parsable, especially in cases
like symlinks, and a few other special cases;
    * it doesn't escape the file names --- and from some reason I have
files containing escape sequences in them...
    * it records only SHA-1;
    * it doesn't handle ACL's or extended attributes;

    (B) `mtree` from FreeBSD.  I found two ports of it for Linux,
however my main concern is how parsable is the output...

    (C) `md5deep` (or `sha*deep`), which only records the checksum not
other meta-data.


    Thus, are there any other alternatives?  (Just to be clear, I
don't need a "backup" solution, just something to record file-system
meta-data.  Maybe a "meta-backup" solution... :) )

    Ciprian.

Re: [gentoo-user] File system meta-data indexer / checker

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

Am 16.06.2013 23:14, schrieb Ciprian Dorin Craciun:
>     Hello all!
> 
>     While struggling with managing various old backups --- just
> imagine 10 or so copies of almost the same content, some with `rsync`,
> some with `rdiff-backup`, yet some others on plain ISO's, all over a
> range of a few years --- I stumbled upon the following missing piece
> in the Linux tools ecosystem:  a file-system crawler that records
> **only** meta-data, like all the info available through `stat`, plus
> an assortment of hashes of the files (at least MD5 and SHA-1,2
> family), and optionally ACL's and extended attributes.  Thus I was
> wondering if someone knows a tool that fits this description.
> 
> 
[...]
> 
>     Thus, are there any other alternatives?  (Just to be clear, I
> don't need a "backup" solution, just something to record file-system
> meta-data.  Maybe a "meta-backup" solution... :) )
> 
>     Ciprian.
> 

Do you mean something like `cp --attributes-only`?

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [gentoo-user] File system meta-data indexer / checker

[Ciprian Dorin Craciun]

On Mon, Jun 17, 2013 at 8:52 AM, Florian Philipp <lists@binarywings.net> wrote:
> Am 16.06.2013 23:14, schrieb Ciprian Dorin Craciun:
>>     Hello all!
>>
>>     While struggling with managing various old backups --- just
>> imagine 10 or so copies of almost the same content, some with `rsync`,
>> some with `rdiff-backup`, yet some others on plain ISO's, all over a
>> range of a few years --- I stumbled upon the following missing piece
>> in the Linux tools ecosystem:  a file-system crawler that records
>> **only** meta-data, like all the info available through `stat`, plus
>> an assortment of hashes of the files (at least MD5 and SHA-1,2
>> family), and optionally ACL's and extended attributes.  Thus I was
>> wondering if someone knows a tool that fits this description.
>>
>>
> [...]
>>
>>     Thus, are there any other alternatives?  (Just to be clear, I
>> don't need a "backup" solution, just something to record file-system
>> meta-data.  Maybe a "meta-backup" solution... :) )
>>
>>     Ciprian.
>
>
> Do you mean something like `cp --attributes-only`?


    Nop, wouldn't do the job, because `cp --attributes-only` requires
a target file system, however I need something which "records" those
attributes in a file that I can use afterwards to restore them (or
compare them with further versions, etc.).

    Ciprian.

Re: [gentoo-user] File system meta-data indexer / checker

[Peter Humphrey]

On Monday 17 Jun 2013 07:00:35 Ciprian Dorin Craciun wrote:
> On Mon, Jun 17, 2013 at 8:52 AM, Florian Philipp <lists@binarywings.net> 
wrote:
> > Do you mean something like `cp --attributes-only`?
> 
>     Nop, wouldn't do the job, because `cp --attributes-only` requires
> a target file system, ...

You could always cp to /dev/null.

-- 
Peter

Re: [gentoo-user] File system meta-data indexer / checker

[Ciprian Dorin Craciun]

On Mon, Jun 17, 2013 at 10:21 AM, Peter Humphrey
<peter@humphrey.ukfsn.org> wrote:
> On Monday 17 Jun 2013 07:00:35 Ciprian Dorin Craciun wrote:
>> On Mon, Jun 17, 2013 at 8:52 AM, Florian Philipp <lists@binarywings.net>
> wrote:
>> > Do you mean something like `cp --attributes-only`?
>>
>>     Nop, wouldn't do the job, because `cp --attributes-only` requires
>> a target file system, ...
>
>
> You could always cp to /dev/null.


    Sorry I don't follow...  What does imply to "cp to /dev/null", and
what would be the outcome of that?

Re: [gentoo-user] File system meta-data indexer / checker

[Paul Colquhoun]

[-- Attachment #1: Type: text/plain, Size: 2128 bytes --]

On Mon, 17 Jun 2013 00:14:17 Ciprian Dorin Craciun wrote:
>     Hello all!
> 
>     While struggling with managing various old backups --- just
> imagine 10 or so copies of almost the same content, some with `rsync`,
> some with `rdiff-backup`, yet some others on plain ISO's, all over a
> range of a few years --- I stumbled upon the following missing piece
> in the Linux tools ecosystem:  a file-system crawler that records
> **only** meta-data, like all the info available through `stat`, plus
> an assortment of hashes of the files (at least MD5 and SHA-1,2
> family), and optionally ACL's and extended attributes.  Thus I was
> wondering if someone knows a tool that fits this description.
> 
> 
>     I must say I've tried to do my homework, and below are a few tools
> that come close, but not quite...
> 
>     (A) `rdup` is probably the closest to what I'm searching.  However
> there are a few issues:
>     * it's output format is not very parsable, especially in cases
> like symlinks, and a few other special cases;
>     * it doesn't escape the file names --- and from some reason I have
> files containing escape sequences in them...
>     * it records only SHA-1;
>     * it doesn't handle ACL's or extended attributes;
> 
>     (B) `mtree` from FreeBSD.  I found two ports of it for Linux,
> however my main concern is how parsable is the output...
> 
>     (C) `md5deep` (or `sha*deep`), which only records the checksum not
> other meta-data.
> 
> 
>     Thus, are there any other alternatives?  (Just to be clear, I
> don't need a "backup" solution, just something to record file-system
> meta-data.  Maybe a "meta-backup" solution... :) )
> 
>     Ciprian.


Have a look at 'tripwire'. It's primarily an intrusion detection tool, but it does the job 
by recording file meta-data and checksums, then checking to see if they have 
changed. I can't remember if it handles ACL's, as it's been a few years since I used it.


-- 
Reverend Paul Colquhoun, ULC.     http://andor.dropbear.id.au/
  Asking for technical help in newsgroups?  Read this first:
     http://catb.org/~esr/faqs/smart-questions.html#intro


[-- Attachment #2: Type: text/html, Size: 9416 bytes --]

Re: [gentoo-user] File system meta-data indexer / checker

[Peter Humphrey]

On Monday 17 Jun 2013 08:25:08 Ciprian Dorin Craciun wrote:
> On Mon, Jun 17, 2013 at 10:21 AM, Peter Humphrey
> 
> <peter@humphrey.ukfsn.org> wrote:
> > On Monday 17 Jun 2013 07:00:35 Ciprian Dorin Craciun wrote:
> >> On Mon, Jun 17, 2013 at 8:52 AM, Florian Philipp
> >> <lists@binarywings.net>
> > 
> > wrote:
> >> > Do you mean something like `cp --attributes-only`?
> >> > 
> >>     Nop, wouldn't do the job, because `cp --attributes-only` requires
> >> 
> >> a target file system, ...
> > 
> > You could always cp to /dev/null.
> 
>     Sorry I don't follow...  What does imply to "cp to /dev/null", and
> what would be the outcome of that?

/dev/null would be the target file system you referred to. It's a bottomless 
empty pit, so no physically real copy would be made.

-- 
Peter

Re: [gentoo-user] File system meta-data indexer / checker

[Ciprian Dorin Craciun]

On Mon, Jun 17, 2013 at 6:56 PM, Peter Humphrey
<peter@humphrey.ukfsn.org> wrote:
> On Monday 17 Jun 2013 08:25:08 Ciprian Dorin Craciun wrote:
>> On Mon, Jun 17, 2013 at 10:21 AM, Peter Humphrey
>>
>> <peter@humphrey.ukfsn.org> wrote:
>> > On Monday 17 Jun 2013 07:00:35 Ciprian Dorin Craciun wrote:
>> >> On Mon, Jun 17, 2013 at 8:52 AM, Florian Philipp
>> >> <lists@binarywings.net>
>> >
>> > wrote:
>> >> > Do you mean something like `cp --attributes-only`?
>> >> >
>> >>     Nop, wouldn't do the job, because `cp --attributes-only` requires
>> >>
>> >> a target file system, ...
>> >
>> > You could always cp to /dev/null.
>>
>>     Sorry I don't follow...  What does imply to "cp to /dev/null", and
>> what would be the outcome of that?
>
> /dev/null would be the target file system you referred to. It's a bottomless
> empty pit, so no physically real copy would be made.


    I would be curious to find out how one can use `/dev/null` as a
replacement for an **entire** file-system (i.e. not just a replacement
for a stream sink)?  And how would the `cp` command be used to obtain
such an outcome?