[gentoo-user] [OT] fast recursive local copy

[gentoo-user] [OT] fast recursive local copy

[meino.cramer]

Hi,

 I am looking for a faster way to do a

    cp -a r <thisdir> <thatdir>

 locally on one machine with one harddisk inside.

 Is there a neat trick to accomplish this faster than
 good old cp?
 (I have to reorder greater part sof user data on my 
 harddisc...)

 Thank you very much in advance for any help!
 Have a nice weekend!
 Best regards
 mcc



-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

[gentoo-user] Re: [OT] fast recursive local copy

[Nikos Chantziaras]

On 08/14/2009 05:47 AM, meino.cramer@gmx.de wrote:
> Hi,
>
>   I am looking for a faster way to do a
>
>      cp -a r<thisdir>  <thatdir>
>
>   locally on one machine with one harddisk inside.
>
>   Is there a neat trick to accomplish this faster than
>   good old cp?

Nope.  Some people like to use pipes in hope to speed it up, something like:

   tar -c <thisdir> | tar -xC <thatdir>

but this isn't really faster and fscks up sparse files.

But if <thatdir> already contains some files from <thisdir>, then rsync 
would be faster than cp.  If not, stick with cp.

Re: [gentoo-user] [OT] fast recursive local copy

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

On Fri, 14 Aug 2009 04:47:24 +0200
meino.cramer@gmx.de wrote:

>  I am looking for a faster way to do a
> 
>     cp -a r <thisdir> <thatdir>
> 
>  locally on one machine with one harddisk inside.

rsync can parallel some of the tasks, like creating paths while files
are being copied, so I guess it wouldn't hurt to test it against cp:

  rsync -HaAX <thisdir> <thatdir>

Flags are:

 -H - hardlinks
 -a - ids, modes, timestamps
 -A - ACLs
 -X - extended attributes

You might want to throw in '-x' for 'one-file-system', but that
contradicts with behavior of given cp example.

-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Volker Armin Hemmann]

On Freitag 14 August 2009, Nikos Chantziaras wrote:
> On 08/14/2009 05:47 AM, meino.cramer@gmx.de wrote:
> > Hi,
> >
> >   I am looking for a faster way to do a
> >
> >      cp -a r<thisdir>  <thatdir>
> >
> >   locally on one machine with one harddisk inside.
> >
> >   Is there a neat trick to accomplish this faster than
> >   good old cp?
>
> Nope.  Some people like to use pipes in hope to speed it up, something
> like:
>
>    tar -c <thisdir> | tar -xC <thatdir>
>
> but this isn't really faster and fscks up sparse files.
>
> But if <thatdir> already contains some files from <thisdir>, then rsync
> would be faster than cp.  If not, stick with cp.

use cp with -u and files already existing are skipped.

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Joerg Schilling]

Nikos Chantziaras <realnc@arcor.de> wrote:

> On 08/14/2009 05:47 AM, meino.cramer@gmx.de wrote:
> > Hi,
> >
> >   I am looking for a faster way to do a
> >
> >      cp -a r<thisdir>  <thatdir>
> >
> >   locally on one machine with one harddisk inside.
> >
> >   Is there a neat trick to accomplish this faster than
> >   good old cp?
>
> Nope.  Some people like to use pipes in hope to speed it up, something like:
>
>    tar -c <thisdir> | tar -xC <thatdir>
>
> but this isn't really faster and fscks up sparse files.

Pipes definitely do not speed up things, they slow things down as pipes 
introduce a significant system overhead.

The fastest method for copying directory trees (typicalls 30% faster than any 
other known method) is to use star:

star -copy -p -xdot -acl -sparse -C <fromdir> . <todir>

Do not forget the '.' !!!!!

Note that on Linux you may _need_ to add "-no-fsync" because file I/O is slow 
on Linux. On Solaris, not using -no-fsync slows things down by aprox. 10% but 
allows star to grant that everything was really copied to stable storage.
On Linux, ot using -no-fsync slows things down by aprox. 400%, this is why
I recommend to add "-no-fsync". BTW: other programs behave like star 
"-no-fsync" by default.

It may help to speed up things (in case you have enough RAM) to add: fs=128M
Use no more than 1/2 of the physical RAM as FIFO size.

Star implements the most effective way since more than 20 years. The idea is
to use a FIFO made of shared memory and to let star fork by default into two 
processes that work independently. 


Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling:

> The fastest method for copying directory trees (typicalls 30% faster than
> any other known method) is to use star:
>
> star -copy -p -xdot -acl -sparse -C <fromdir> . <todir>

That's a really nice one. However, does it also handle the "update <todir>" 
case mentioned above?

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Joerg Schilling]

Dirk Heinrichs <dirk.heinrichs@online.de> wrote:

> Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling:
>
> > The fastest method for copying directory trees (typicalls 30% faster than
> > any other known method) is to use star:
> >
> > star -copy -p -xdot -acl -sparse -C <fromdir> . <todir>
>
> That's a really nice one. However, does it also handle the "update <todir>" 
> case mentioned above?

star by default only overwrites a file if it is older than the file that is 
going to be extracted.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 233 bytes --]

Am Freitag 14 August 2009 18:11:34 schrieb Joerg Schilling:

> star by default only overwrites a file if it is older than the file that is
> going to be extracted.

Thanks. Will definitely try it out on next occasion.

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling:

> Note that on Linux you may need to add "-no-fsync" because file I/O is slow
> on Linux. On Solaris, not using -no-fsync slows things down by aprox. 10%
> but allows star to grant that everything was really copied to stable
> storage. On Linux, ot using -no-fsync slows things down by aprox. 400%,
> this is why I recommend to add "-no-fsync".

This is also quite interesting. Do you have some (links to) recent benchmarks 
which would second that? Could this even be depending on the filesystem used on 
Linux?

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Joerg Schilling]

Dirk Heinrichs <dirk.heinrichs@online.de> wrote:

> Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling:
>
> > Note that on Linux you may need to add "-no-fsync" because file I/O is slow
> > on Linux. On Solaris, not using -no-fsync slows things down by aprox. 10%
> > but allows star to grant that everything was really copied to stable
> > storage. On Linux, ot using -no-fsync slows things down by aprox. 400%,
> > this is why I recommend to add "-no-fsync".
>
> This is also quite interesting. Do you have some (links to) recent benchmarks 
> which would second that? Could this even be depending on the filesystem used on 
> Linux?

I did this test aprox. 3-4 years ago. You may try to do an own test and report.

I did just rerun a test on a recent ubuntu in a VirtualBox environment and the 
speedup factor with -no-fsync was 8x.

Jörg

-- 
 EMail:joerg@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
       js@cs.tu-berlin.de                (uni)  
       joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]

Am Freitag 14 August 2009 22:47:46 schrieb Joerg Schilling:
> Dirk Heinrichs <dirk.heinrichs@online.de> wrote:
> > Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling:
> > > Note that on Linux you may need to add "-no-fsync" because file I/O is
> > > slow on Linux. On Solaris, not using -no-fsync slows things down by
> > > aprox. 10% but allows star to grant that everything was really copied
> > > to stable storage. On Linux, ot using -no-fsync slows things down by
> > > aprox. 400%, this is why I recommend to add "-no-fsync".
> >
> > This is also quite interesting. Do you have some (links to) recent
> > benchmarks which would second that? Could this even be depending on the
> > filesystem used on Linux?
>
> I did this test aprox. 3-4 years ago. You may try to do an own test and
> report.
>
> I did just rerun a test on a recent ubuntu in a VirtualBox environment and
> the speedup factor with -no-fsync was 8x.

OK, here's mine, then. Please note that the test happened on an encrypted 
logical volume containing an XFS filesystem. Source and destination directory 
are on the same LV.

# time star -copy -p -xdot -acl -sparse -C /gentoo/overlays/portage . 
/gentoo/build/portage
star: 0 blocks + 447532544 bytes (total of 447532544 bytes = 437043.50k).
star -copy -p -xdot -acl -sparse -C /gentoo/overlays/portage .   16,57s user 
60,22s system 11% cpu 11:23,63 total

# time star -copy -p -xdot -acl -sparse -no-fsync -C /gentoo/overlays/portage 
. /gentoo/build/portage
star: 0 blocks + 447532544 bytes (total of 447532544 bytes = 437043.50k).
star -copy -p -xdot -acl -sparse -no-fsync -C /gentoo/overlays/portage .   
12,33s user 35,39s system 10% cpu 7:44,13 total

Unfortunately, I can't compare with OSol :(

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Volker Armin Hemmann]

On Samstag 15 August 2009, Dirk Heinrichs wrote:
> Am Freitag 14 August 2009 22:47:46 schrieb Joerg Schilling:
> > Dirk Heinrichs <dirk.heinrichs@online.de> wrote:
> > > Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling:
> > > > Note that on Linux you may need to add "-no-fsync" because file I/O
> > > > is slow on Linux. On Solaris, not using -no-fsync slows things down
> > > > by aprox. 10% but allows star to grant that everything was really
> > > > copied to stable storage. On Linux, ot using -no-fsync slows things
> > > > down by aprox. 400%, this is why I recommend to add "-no-fsync".
> > >
> > > This is also quite interesting. Do you have some (links to) recent
> > > benchmarks which would second that? Could this even be depending on the
> > > filesystem used on Linux?
> >
> > I did this test aprox. 3-4 years ago. You may try to do an own test and
> > report.
> >
> > I did just rerun a test on a recent ubuntu in a VirtualBox environment
> > and the speedup factor with -no-fsync was 8x.
>
> OK, here's mine, then. Please note that the test happened on an encrypted
> logical volume containing an XFS filesystem. Source and destination
> directory are on the same LV.
>
> # time star -copy -p -xdot -acl -sparse -C /gentoo/overlays/portage .
> /gentoo/build/portage
> star: 0 blocks + 447532544 bytes (total of 447532544 bytes = 437043.50k).
> star -copy -p -xdot -acl -sparse -C /gentoo/overlays/portage .   16,57s
> user 60,22s system 11% cpu 11:23,63 total
>
> # time star -copy -p -xdot -acl -sparse -no-fsync -C
> /gentoo/overlays/portage . /gentoo/build/portage
> star: 0 blocks + 447532544 bytes (total of 447532544 bytes = 437043.50k).
> star -copy -p -xdot -acl -sparse -no-fsync -C /gentoo/overlays/portage .
> 12,33s user 35,39s system 10% cpu 7:44,13 total
>
> Unfortunately, I can't compare with OSol :(
>
> Bye...
>
> 	Dirk

and you dropped caches in between?

Re: [gentoo-user] Re: [OT] fast recursive local copy

[Volker Armin Hemmann]

On Freitag 14 August 2009, Joerg Schilling wrote:
> Dirk Heinrichs <dirk.heinrichs@online.de> wrote:
> > Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling:
> > > Note that on Linux you may need to add "-no-fsync" because file I/O is
> > > slow on Linux. On Solaris, not using -no-fsync slows things down by
> > > aprox. 10% but allows star to grant that everything was really copied
> > > to stable storage. On Linux, ot using -no-fsync slows things down by
> > > aprox. 400%, this is why I recommend to add "-no-fsync".
> >
> > This is also quite interesting. Do you have some (links to) recent
> > benchmarks which would second that? Could this even be depending on the
> > filesystem used on Linux?
>
> I did this test aprox. 3-4 years ago. You may try to do an own test and
> report.
>
> I did just rerun a test on a recent ubuntu in a VirtualBox environment and
> the speedup factor with -no-fsync was 8x.
>
> Jörg

reiser4 on raid5, compression is on, since no barriers, sync mode. Three runs, 
first without, second with, third without -no-fsync. echo 1 > 
/proc/sys/vm/drop_caches in between each run. temp was removed and recreated 
between each run. source and target on different md devices.

star: 0 blocks + 96006656 bytes (total of 96006656 bytes = 93756.50k).                         
star -copy -p -xdot -acl -sparse -C /usr/local/portage . temp/  0,85s user 
16,79s system 6% cpu 4:48,77 total 

star: 0 blocks + 96006656 bytes (total of 96006656 bytes = 93756.50k).
star -copy -p -xdot -acl -sparse -no-fsync -C /usr/local/portage . temp/  
0,43s user 3,14s system 24% cpu 14,389 total


star: 0 blocks + 96006656 bytes (total of 96006656 bytes = 93756.50k).
star -copy -p -xdot -acl -sparse -C /usr/local/portage . temp/  0,88s user 
15,93s system 6% cpu 4:13,76 total
  
but reiser4 is infamous for not loving fsync ;)