[gentoo-user] Copying data efficiently

[gentoo-user] Copying data efficiently

[meino.cramer]

Hi,

is there any tool in the Gentoo portage which may speed up (make it
mopre efficient) the following task:

On my HD there are data I want to copy to two identical external HDs.
These HDs are of the same type/model and each is separately
connectable via USB to my PC (...these two of the typical mobile external
USB-HDs).

Instead of copying the data twice from my PC to eah of the HDs I want
to do it once...like I would be able to give the cp-command two
instead of one target where to copy two.

The result should be two identically populated external HDs, each of
them useable/readable without the need to the other one.

What tool of the portage tree I able to accomplish this task?

Best regards,
mcc

Re: [gentoo-user] Copying data efficiently

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1700 bytes --]

On Saturday 17 May 2014 04:33:57 meino.cramer@gmx.de wrote:
> Hi,
> 
> is there any tool in the Gentoo portage which may speed up (make it
> mopre efficient) the following task:
> 
> On my HD there are data I want to copy to two identical external HDs.
> These HDs are of the same type/model and each is separately
> connectable via USB to my PC (...these two of the typical mobile external
> USB-HDs).
> 
> Instead of copying the data twice from my PC to eah of the HDs I want
> to do it once...like I would be able to give the cp-command two
> instead of one target where to copy two.
> 
> The result should be two identically populated external HDs, each of
> them useable/readable without the need to the other one.
> 
> What tool of the portage tree I able to accomplish this task?
> 
> Best regards,
> mcc


1. You could set up the two ext drives as a mirrored RAID and use your 
copying/tar-ing/dd tool of choice.


2. Or you could use pipe and tee to split the feed into any devices you want, 
e.g.:

  pv /dev/sda1 | tee >(dd of=/dev/sdb1) >(dd of=/dev/sdc1) 


3. Or you could use a sequential copy:

  cp -a /home /dev/sdb1/ && cp -a /home /dev/sdc1


NOTES: 

a) Unlike other commands, pv will give you a progress bar so that you know how 
long your back up is taking.

b) The 3rd example above will copy sequentially, but depending on the size of 
the file(s) the second copy may be read from cache.

c) If you're doing this over the network then you can use nc in listening mode 
at the receiving end, or ssh if the network is untrusted.


I'm interested to see what other ways will be suggested for this task.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Copying data efficiently

[meino.cramer]

Mick <michaelkintzios@gmail.com> [14-05-17 09:48]:
> On Saturday 17 May 2014 04:33:57 meino.cramer@gmx.de wrote:
> > Hi,
> > 
> > is there any tool in the Gentoo portage which may speed up (make it
> > mopre efficient) the following task:
> > 
> > On my HD there are data I want to copy to two identical external HDs.
> > These HDs are of the same type/model and each is separately
> > connectable via USB to my PC (...these two of the typical mobile external
> > USB-HDs).
> > 
> > Instead of copying the data twice from my PC to eah of the HDs I want
> > to do it once...like I would be able to give the cp-command two
> > instead of one target where to copy two.
> > 
> > The result should be two identically populated external HDs, each of
> > them useable/readable without the need to the other one.
> > 
> > What tool of the portage tree I able to accomplish this task?
> > 
> > Best regards,
> > mcc
> 
> 
> 1. You could set up the two ext drives as a mirrored RAID and use your 
> copying/tar-ing/dd tool of choice.
> 
> 
> 2. Or you could use pipe and tee to split the feed into any devices you want, 
> e.g.:
> 
>   pv /dev/sda1 | tee >(dd of=/dev/sdb1) >(dd of=/dev/sdc1) 
> 
> 
> 3. Or you could use a sequential copy:
> 
>   cp -a /home /dev/sdb1/ && cp -a /home /dev/sdc1
> 
> 
> NOTES: 
> 
> a) Unlike other commands, pv will give you a progress bar so that you know how 
> long your back up is taking.
> 
> b) The 3rd example above will copy sequentially, but depending on the size of 
> the file(s) the second copy may be read from cache.
> 
> c) If you're doing this over the network then you can use nc in listening mode 
> at the receiving end, or ssh if the network is untrusted.
> 
> 
> I'm interested to see what other ways will be suggested for this task.
> 
> -- 
> Regards,
> Mick

Hi Mick,

thank your reply! :)

From your numbering of the possibilities...
1.)  ...I am no RAID guru and would try this later with data, which
    are not valuable...
2.) That looks interesting! Unfortunately it seems to copy device
    contents on low level instead of files. The source are directory
    structure -- not whole devices...
    Or did I overlook an option mentioned in the manpage...?
3.) The files I want to copy are in the size of some GB each. So the
    cache isnt big enough to hold ALL files for the second part.


Best regards,
mcc

Re: [gentoo-user] Copying data efficiently

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

On Sat, 17 May 2014 09:59:08 +0200, meino.cramer@gmx.de wrote:

> > 3. Or you could use a sequential copy:
> > 
> >   cp -a /home /dev/sdb1/ && cp -a /home /dev/sdc1

> 3.) The files I want to copy are in the size of some GB each. So the
>     cache isnt big enough to hold ALL files for the second part.

Run the two copies simultaneously, start the first, switch to another
tab, start the second. That way the data for the second copy is always
the most recently cached.

However, I expect the speed limit here may be the USB bus unless you are
using USB 3.0 drives on different buses.


-- 
Neil Bothwick

I'll never forget the 1st time I ran Windows, but I'm trying...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Copying data efficiently

[Daniel Troeder]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

Am 17.05.2014 11:58, schrieb Neil Bothwick:
> On Sat, 17 May 2014 09:59:08 +0200, meino.cramer@gmx.de wrote:
> 
>>> 3. Or you could use a sequential copy:
>>>
>>>   cp -a /home /dev/sdb1/ && cp -a /home /dev/sdc1
> 
>> 3.) The files I want to copy are in the size of some GB each. So the
>>     cache isnt big enough to hold ALL files for the second part.
> 
> Run the two copies simultaneously, start the first, switch to another
> tab, start the second. That way the data for the second copy is always
> the most recently cached.
> 
> However, I expect the speed limit here may be the USB bus unless you are
> using USB 3.0 drives on different buses.
I was thinking about "how to make sure cache is used", and that two
simultanious cp won't work, because the progress for the two cp will
quickly diverge. But then I realized: there is no need to think about
the read cache - the limiting factor is always the writing side,
especially with USB! So IMO it doesn't matter at all how you do it!

I guess two simultaneous cp will be the same as two sequential cp,
except if you have two separate USB-buses. Usually you have just one
externally connectible, use "lsusb -t" to check.

If you have less that 2 times the size of your files, IMO simultaneous
cp will be worse, because Linux (don't know if USB-subsystem or cp)
creates big buffers when cp'ing (check with "free -m"), and you'll
probably get into memory trouble.


Greetings,
Daniel

PS: Quickest way is always to open USB-case and plug SATA cable from
motherboard into drive. With >80GB it's always worth the trouble.

-- 
Get my PGP key at:
*
http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x837FB8B5BB9D4887
* $ gpg --recv-keys --keyserver keyserver.ubuntu.com 0xBB9D4887


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

Re: [gentoo-user] Copying data efficiently

[meino.cramer]

Daniel Troeder <daniel@admin-box.com> [14-05-17 16:36]:
> Am 17.05.2014 11:58, schrieb Neil Bothwick:
> > On Sat, 17 May 2014 09:59:08 +0200, meino.cramer@gmx.de wrote:
> > 
> >>> 3. Or you could use a sequential copy:
> >>>
> >>>   cp -a /home /dev/sdb1/ && cp -a /home /dev/sdc1
> > 
> >> 3.) The files I want to copy are in the size of some GB each. So the
> >>     cache isnt big enough to hold ALL files for the second part.
> > 
> > Run the two copies simultaneously, start the first, switch to another
> > tab, start the second. That way the data for the second copy is always
> > the most recently cached.
> > 
> > However, I expect the speed limit here may be the USB bus unless you are
> > using USB 3.0 drives on different buses.
> I was thinking about "how to make sure cache is used", and that two
> simultanious cp won't work, because the progress for the two cp will
> quickly diverge. But then I realized: there is no need to think about
> the read cache - the limiting factor is always the writing side,
> especially with USB! So IMO it doesn't matter at all how you do it!
> 
> I guess two simultaneous cp will be the same as two sequential cp,
> except if you have two separate USB-buses. Usually you have just one
> externally connectible, use "lsusb -t" to check.
> 
> If you have less that 2 times the size of your files, IMO simultaneous
> cp will be worse, because Linux (don't know if USB-subsystem or cp)
> creates big buffers when cp'ing (check with "free -m"), and you'll
> probably get into memory trouble.
> 
> 
> Greetings,
> Daniel
> 
> PS: Quickest way is always to open USB-case and plug SATA cable from
> motherboard into drive. With >80GB it's always worth the trouble.
> 
> -- 
> Get my PGP key at:
> *
> http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0x837FB8B5BB9D4887
> * $ gpg --recv-keys --keyserver keyserver.ubuntu.com 0xBB9D4887
> 

Hi,

thank you /.*/  for /.*/ replies ! :)

...one aspect is missing: The load (that is the I/O on the 
source hd).
If this hd is busy spitting out the data twice, it cannot
serve outhe jobs twice as long...

Hmmmm....

Best regards,
mcc

PS:
On the source system
#> lsusb -t 
/:  Bus 09.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/4p, 12M
/:  Bus 08.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/2p, 12M
/:  Bus 07.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/5p, 12M
    |__ Port 2: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 2: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=ohci-pci/5p, 12M
    |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 3: Dev 3, If 0, Class=Audio, Driver=snd-usb-audio, 12M
    |__ Port 3: Dev 3, If 1, Class=Audio, Driver=snd-usb-audio, 12M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/4p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/5p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/5p, 480M
    |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/6p, 480M

Re: [gentoo-user] Copying data efficiently

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 3052 bytes --]

On Saturday 17 May 2014 08:59:08 meino.cramer@gmx.de wrote:
> Mick <michaelkintzios@gmail.com> [14-05-17 09:48]:
> > On Saturday 17 May 2014 04:33:57 meino.cramer@gmx.de wrote:
> > > Hi,
> > > 
> > > is there any tool in the Gentoo portage which may speed up (make it
> > > mopre efficient) the following task:
> > > 
> > > On my HD there are data I want to copy to two identical external HDs.
> > > These HDs are of the same type/model and each is separately
> > > connectable via USB to my PC (...these two of the typical mobile
> > > external USB-HDs).
> > > 
> > > Instead of copying the data twice from my PC to eah of the HDs I want
> > > to do it once...like I would be able to give the cp-command two
> > > instead of one target where to copy two.
> > > 
> > > The result should be two identically populated external HDs, each of
> > > them useable/readable without the need to the other one.
> > > 
> > > What tool of the portage tree I able to accomplish this task?
> > > 
> > > Best regards,
> > > mcc
> > 
> > 1. You could set up the two ext drives as a mirrored RAID and use your
> > copying/tar-ing/dd tool of choice.
> > 
> > 
> > 2. Or you could use pipe and tee to split the feed into any devices you
> > want,
> > 
> > e.g.:
> >   pv /dev/sda1 | tee >(dd of=/dev/sdb1) >(dd of=/dev/sdc1)
> > 
> > 3. Or you could use a sequential copy:
> >   cp -a /home /dev/sdb1/ && cp -a /home /dev/sdc1
> > 
> > NOTES:
> > 
> > a) Unlike other commands, pv will give you a progress bar so that you
> > know how long your back up is taking.
> > 
> > b) The 3rd example above will copy sequentially, but depending on the
> > size of the file(s) the second copy may be read from cache.
> > 
> > c) If you're doing this over the network then you can use nc in listening
> > mode at the receiving end, or ssh if the network is untrusted.
> > 
> > 
> > I'm interested to see what other ways will be suggested for this task.
> 
> Hi Mick,
> 
> thank your reply! :)
> 
> From your numbering of the possibilities...
> 1.)  ...I am no RAID guru and would try this later with data, which
>     are not valuable...

If the drives are not permanently connected, then you will have to assemble 
the RAID each time you want to copy something, after you plug them in.  
Perhaps not the most efficient method.


> 2.) That looks interesting! Unfortunately it seems to copy device
>     contents on low level instead of files. The source are directory
>     structure -- not whole devices...

Well, I just offered an example for a whole drive.  You will need to point it 
to the file(s) you want to copy over.  The pv command works like cat; 

e.g. cat myfile | tee >(blah ...) 

is the same like:

pv myfile | tee >(blah ...)

>     Or did I overlook an option mentioned in the manpage...?
> 3.) The files I want to copy are in the size of some GB each. So the
>     cache isnt big enough to hold ALL files for the second part.
> 
> 
> Best regards,
> mcc

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Copying data efficiently

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

On Sat, 17 May 2014 20:41:15 +0100, Mick wrote:

> > From your numbering of the possibilities...
> > 1.)  ...I am no RAID guru and would try this later with data, which
> >     are not valuable...  
> 
> If the drives are not permanently connected, then you will have to
> assemble the RAID each time you want to copy something, after you plug
> them in. Perhaps not the most efficient method.

Whatever method is used will likely consist of multiple commands, and
probably a good few arguments, lending itself best to scripting. In that
case, the RAID approach becomes no more difficult to use once set up.


-- 
Neil Bothwick

The Computer is the logical advancement of humankind:
intelligence without morality.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Copying data efficiently

[Stroller]

On Sat, 17 May 2014, at 8:46 am, Mick <michaelkintzios@gmail.com> wrote:
> ...
> a) Unlike other commands, pv will give you a progress bar so that you know how 
> long your back up is taking.

I've been using ddrescue recently - by default it seems to be faster than plain old regular dd.

I assume it must be possible to adjust the parameters (blocksize?) to speed dd up, but I just want to run the command and not have to bother tinkering.

On dumb blind copies, I'm finding ddrescue about 3x or 4x faster than dd, I would estimate.

Stroller.

Re: [gentoo-user] Copying data efficiently

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

On Sat, 17 May 2014 14:44:04 +0100, Stroller wrote:

> > a) Unlike other commands, pv will give you a progress bar so that you
> > know how long your back up is taking.  
> 
> I've been using ddrescue recently - by default it seems to be faster
> than plain old regular dd.
> 
> I assume it must be possible to adjust the parameters (blocksize?) to
> speed dd up, but I just want to run the command and not have to bother
> tinkering.

Use the bs option. The default is 512, which is incredibly slow, bs=4k
makes a substantial difference. However, I usually use dcfldd these days,
whch works out the best block size to use and adds progress reporting.


-- 
Neil Bothwick

Never underestimate the bandwidth of a station wagon full of tapes!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]