[gentoo-user] Cloning a directory hierarchy, but not the content

[gentoo-user] Cloning a directory hierarchy, but not the content

[Alex Schuster]

Hi there!

I am currently putting extra backups to old hard drives I do no longer need 
for other purposes. After that I send the putput out ls -lR and du -m to my 
log directory so I can check what files are on which drive without having to 
attach the drive.

Works, though a better method would be to clone the drive's root directory, 
but with all file sizes being zero. This way I can easily navigate the 
directory structure, instead of browsing through the ls-lR file. Is there a 
utility that does this? It would be even better if the files would be 
created as sparse files, faking the original size.  

I just wrote a little script that does this, but it does not do the sparse 
file thing yet, and would have problems with newline in file names. And I 
guess someone already wrote such a utility?

	Wonko

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Etaoin Shrdlu]

On Sat, 29 Jan 2011 14:58:13 +0100
Alex Schuster <wonko@wonkology.org> wrote:

> Hi there!
> 
> I am currently putting extra backups to old hard drives I do no longer
> need for other purposes. After that I send the putput out ls -lR and du
> -m to my log directory so I can check what files are on which drive
> without having to attach the drive.
> 
> Works, though a better method would be to clone the drive's root
> directory, but with all file sizes being zero. This way I can easily
> navigate the directory structure, instead of browsing through the ls-lR
> file. Is there a utility that does this? It would be even better if the
> files would be created as sparse files, faking the original size.  
> 
> I just wrote a little script that does this, but it does not do the
> sparse file thing yet, and would have problems with newline in file
> names. And I guess someone already wrote such a utility?

IIUC, try

find / -type d -exec sh 'mkdir -p target"$1"' - {} \;

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Alex Schuster]

Etaoin Shrdlu writes:

> On Sat, 29 Jan 2011 14:58:13 +0100
> 
> Alex Schuster <wonko@wonkology.org> wrote:
> > Hi there!
> > 
> > I am currently putting extra backups to old hard drives I do no longer
> > need for other purposes. After that I send the putput out ls -lR and du
> > -m to my log directory so I can check what files are on which drive
> > without having to attach the drive.
> > 
> > Works, though a better method would be to clone the drive's root
> > directory, but with all file sizes being zero. This way I can easily
> > navigate the directory structure, instead of browsing through the ls-lR
> > file. Is there a utility that does this? It would be even better if the
> > files would be created as sparse files, faking the original size.
> > 
> > I just wrote a little script that does this, but it does not do the
> > sparse file thing yet, and would have problems with newline in file
> > names. And I guess someone already wrote such a utility?
> 
> IIUC, try
> 
> find / -type d -exec sh 'mkdir -p target"$1"' - {} \;

Hmm, that does not really seem to work. It tries to execute the whole stuff 
between single quotes as a command. And I don't really understand what it is 
supposed to do, shouldn't this be something like mkdir -p 
/destination/$1/\{\} ?

Anyway, this is what I already have. It duplicates the hierarchy with empty 
files, but I have to add support for sparse files. That won't be too hard, 
but maybe I'm re-inventing the wheel here.

#!/bin/bash

src=$1
dst=$2

cd "$src" || exit $?
IFS=$'\n'
find . |
	while read file
	do
		if [[ -d $file ]]
		then
			[[ -d "$dst/$file" ]] ||
				mkdir -p "$dst/$file"
		elif [[ -f $file ]]
		then
			[[ -d "$dst/${file%/*}" ]] ||
				mkdir -p "$dst/${file%/*}"
			touch "$dst/$file"
		fi
	done


	Wonko

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Etaoin Shrdlu]

On Sat, 29 Jan 2011 15:27:59 +0100 Alex Schuster <wonko@wonkology.org>
wrote:

> > > I just wrote a little script that does this, but it does not do the
> > > sparse file thing yet, and would have problems with newline in file
> > > names. And I guess someone already wrote such a utility?
> > 
> > IIUC, try
> > 
> > find / -type d -exec sh 'mkdir -p target"$1"' - {} \;
> 
> Hmm, that does not really seem to work. It tries to execute the whole
> stuff between single quotes as a command. And I don't really understand
> what it is supposed to do, shouldn't this be something like mkdir -p 
> /destination/$1/\{\} ?

No. That recreates the full directory hierarchy based at / under /target/,
with no files in it. Just the directory hierarchy. I should have added
that, to do it safely, the target should reside higher than the source in
the hierarchy, or it should be on a different filesystem and in that case
-xdev should be specified to find (otherwise an recursive loop would
result). 

A more sensible approach would probably be

cd /source && find . -type d -exec bash 'mkdir -p "${@/#//target/}"' - {} +

with -xdev if needed. But as I see now, this is not what you wanted, so
ignore it.

> Anyway, this is what I already have. It duplicates the hierarchy with
> empty files, but I have to add support for sparse files. That won't be
> too hard, but maybe I'm re-inventing the wheel here.
> 
> #!/bin/bash
> 
> src=$1
> dst=$2
> 
> cd "$src" || exit $?
> IFS=$'\n'
> find . |
> 	while read file
> 	do
> 		if [[ -d $file ]]
> 		then
> 			[[ -d "$dst/$file" ]] ||
> 				mkdir -p "$dst/$file"
> 		elif [[ -f $file ]]
> 		then
> 			[[ -d "$dst/${file%/*}" ]] ||
> 				mkdir -p "$dst/${file%/*}"
> 			touch "$dst/$file"
> 		fi
> 	done

Ok, I misunderstood. You also want the files but empty. Why do you need
support for sparse files? Do you need to manage other types of file
(symlinks, FIFOs, etc.)

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Alex Schuster]

Etaoin Shrdlu writes:

> On Sat, 29 Jan 2011 15:27:59 +0100 Alex Schuster <wonko@wonkology.org>
> wrote:
> > > > I just wrote a little script that does this, but it does not do the
> > > > sparse file thing yet, and would have problems with newline in file
> > > > names. And I guess someone already wrote such a utility?
> > > 
> > > IIUC, try
> > > 
> > > find / -type d -exec sh 'mkdir -p target"$1"' - {} \;
> > 
> > Hmm, that does not really seem to work. It tries to execute the whole
> > stuff between single quotes as a command. And I don't really understand
> > what it is supposed to do, shouldn't this be something like mkdir -p
> > /destination/$1/\{\} ?
> 
> No. That recreates the full directory hierarchy based at / under
> /target/, with no files in it. Just the directory hierarchy.

Ah, now I get it. There's a -c missing after the sh command. 

> I should
> have added that, to do it safely, the target should reside higher than
> the source in the hierarchy, or it should be on a different filesystem
> and in that case -xdev should be specified to find (otherwise an
> recursive loop would result).

Right, but not important in my case. I want to mount my backup drive to 
/mnt, cd /mnt, and duplicate all stuff soemwhere else, without taking up 
much space. Then I can remove the backup drive and I only have to mount it 
again when I need a file's content, but not for finding out which files 
there are and how much space they take. Well, the space already is in the 
file created by du -m, but I'd like to directly navigate around.


> Ok, I misunderstood. You also want the files but empty. Why do you need
> support for sparse files? Do you need to manage other types of file
> (symlinks, FIFOs, etc.)

Yes, symlinks would ne nice, too, I forgot about them. The rest is 
unimportant, as this would be data only, not root file systems. I backup 
that with rdiff-backup to a 2nd drive, but there's much other stuff that I 
would like to put on one of the old drives that lie around here.

Sparse files would be nice because then I do not only have the same logical 
structure, the files also appear to have the same size as the originals, 
instead of having a size of 0. I could navigate and explore the directory 
structure with mc, and with du --apparent-size I could find out how much 
space a subdirectory takes. Again, my du -m file already has this 
information, but while navigating in the directory tree, being able to use 
du would be nice.

	Wonko

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Etaoin Shrdlu]

On Sat, 29 Jan 2011 17:45:30 +0100 Alex Schuster <wonko@wonkology.org>
wrote:

> Ah, now I get it. There's a -c missing after the sh command. 

Right, thans for spotting it.


> > I should have added that, to do it safely, the target should reside
> > higher than the source in the hierarchy, or it should be on a different
> > filesystem and in that case -xdev should be specified to find
> > (otherwise an recursive loop would result).
> 
> Right, but not important in my case. I want to mount my backup drive to 
> /mnt, cd /mnt, and duplicate all stuff soemwhere else, without taking up 
> much space. Then I can remove the backup drive and I only have to mount
> it again when I need a file's content, but not for finding out which
> files there are and how much space they take. Well, the space already is
> in the file created by du -m, but I'd like to directly navigate around.

Oh, I see now: you want the files to *look like* the real ones (eg when
doing ls -l etc.), but be sparse so they don't take up space?
 
> Sparse files would be nice because then I do not only have the same
> logical structure, the files also appear to have the same size as the
> originals, instead of having a size of 0. I could navigate and explore
> the directory structure with mc, and with du --apparent-size I could find
> out how much space a subdirectory takes. Again, my du -m file already has
> this information, but while navigating in the directory tree, being able
> to use du would be nice.

Ok, one way to create a sparse file of, say, 1 megabyte is using dd:

# dd if=/dev/null of=sparsefile bs=1 seek=1M
0+0 records in
0+0 records out
0 bytes (0 B) copied, 2.5419e-05 s, 0.0 kB/s
# ls -l sparsefile 
-rw-r--r-- 1 root root 1048576 Jan 29 11:57 sparsefile
# du -B1 sparsefile
0	sparsefile

Another way, already suggested, is by using truncate, eg

# truncate -s 1M sparsefile

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Alex Schuster]

Etaoin Shrdlu writes:

> On Sat, 29 Jan 2011 17:45:30 +0100 Alex Schuster <wonko@wonkology.org>
> wrote:

>>> I should have added that, to do it safely, the target should reside
>>> higher than the source in the hierarchy, or it should be on a different
>>> filesystem and in that case -xdev should be specified to find
>>> (otherwise an recursive loop would result).
>>
>> Right, but not important in my case. I want to mount my backup drive to 
>> /mnt, cd /mnt, and duplicate all stuff soemwhere else, without taking up 
>> much space. Then I can remove the backup drive and I only have to mount
>> it again when I need a file's content, but not for finding out which
>> files there are and how much space they take. Well, the space already is
>> in the file created by du -m, but I'd like to directly navigate around.
> 
> Oh, I see now: you want the files to *look like* the real ones (eg when
> doing ls -l etc.), but be sparse so they don't take up space?

Exactly. Sorry I did not make myself clearer.
It's working now, and I like it :) I added some more features, like
clipping files to a maximum size. So the clone can still be very small
compared to the original, with small files being intact and usable.


> Ok, one way to create a sparse file of, say, 1 megabyte is using dd:
> 
> # dd if=/dev/null of=sparsefile bs=1 seek=1M
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 2.5419e-05 s, 0.0 kB/s
> # ls -l sparsefile 
> -rw-r--r-- 1 root root 1048576 Jan 29 11:57 sparsefile
> # du -B1 sparsefile
> 0	sparsefile

That's how I wanted to do it first, too.

> Another way, already suggested, is by using truncate, eg
> 
> # truncate -s 1M sparsefile

I used this, because so I can modify a file that I created empty with cp
--attributes-only. Keeping the attributes would have been a bit complicated.

In case anyone else is interested, the script is here:
http://www.wonkology.org/utils/clone0

wonko@weird ~ $ clone0 -h
clone0 version 2011-01-29
Duplicate a file / directory hierarchy. Files are
created as sparse files, not taking up real space.

Usage: clone0 [-dhSv0] [-s size] src... dst

Options:
  -d      clone directory structure only, not files
  -h      show this help
  -s size copy files up to size as the are, and clip larger files
  -S      do not create sparse files
  -v      show directories being created
  -vv     show files being created
  -vvv    debug output
  -0      clip files larger than size (option -s) to zero size

Arguments:
  src... one or more directories to clone
  dst    destination directory (will be created)


Thanks for the input, guys!

	Wonko

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1514 bytes --]

Am 29.01.2011 14:58, schrieb Alex Schuster:
> Hi there!
> 
> I am currently putting extra backups to old hard drives I do no longer need 
> for other purposes. After that I send the putput out ls -lR and du -m to my 
> log directory so I can check what files are on which drive without having to 
> attach the drive.
> 
> Works, though a better method would be to clone the drive's root directory, 
> but with all file sizes being zero. This way I can easily navigate the 
> directory structure, instead of browsing through the ls-lR file. Is there a 
> utility that does this? It would be even better if the files would be 
> created as sparse files, faking the original size.  
> 
> I just wrote a little script that does this, but it does not do the sparse 
> file thing yet, and would have problems with newline in file names. And I 
> guess someone already wrote such a utility?
> 
> 	Wonko
> 

Use `truncate -s <size> <file>`

It creates a sparse file if the specified file is smaller than the
specified size. It will also create a new file if it does not yet exist.

In order to avoid trouble with line breaks in names, I recommend
something like:
find . -type f -print0 |
while read -d $'\0' file; do
	echo "File=$file"
done

Or use similar commands accepting or outputting 0-byte terminated
strings, for example xargs -0, du -0, grep -z.

For copying file attributes from one file to another you can use `cp
--attributes-only`.

Hope this helps,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Alex Schuster]

Florian Philipp writes:

> Use `truncate -s <size> <file>`
> 
> It creates a sparse file if the specified file is smaller than the
> specified size. It will also create a new file if it does not yet exist.

Nice one. First I did not see an improvement over using dd to create the 
sparse file, but in combination with cp --attributes-only I can now 
duplicate the file with its attributes but zero size, and then fake the 
size.

> In order to avoid trouble with line breaks in names, I recommend
> something like:
> find . -type f -print0 |
> while read -d $'\0' file; do
> 	echo "File=$file"
> done

Thanks!


> For copying file attributes from one file to another you can use `cp
> --attributes-only`.

Oh my, another case of a (german) man page that does not show all the 
possible arguments. Never heard about that, thanks!

	Wonko

Re: [gentoo-user] Cloning a directory hierarchy, but not the content

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 706 bytes --]

Am 29.01.2011 20:31, schrieb Alex Schuster:
> Florian Philipp writes:
[...]
> 
> 
>> For copying file attributes from one file to another you can use `cp
>> --attributes-only`.
> 
> Oh my, another case of a (german) man page that does not show all the 
> possible arguments. Never heard about that, thanks!
> 
> 	Wonko
> 

The German man-pages can be a real pain. Guess there are too few people
who keep them up-to-date. Oh well ... maybe some day when I have more time.

Until then I make it a point to never install them in the first place
when I can avoid it:
`LINGUAS='en' emerge -1 man-pages && emerge -C man-pages-de`

Or I just delete them with `rm -r /usr/share/man/de*` :)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]