[gentoo-user] is "scp" reliable?

[gentoo-user] is "scp" reliable?

[thelma]

I use "scp" to copy large file over local network, windows-7_pro.ova (about 28GB) but I it failed to import the appliance with VirtualBox.
Checking the md5sum of the source and destination it turn out they are different.

Is "scp" reliable for large files? 

Re: [gentoo-user] is "scp" reliable?

[thelma]

On 5/29/21 1:53 PM, thelma@sys-concept.com wrote:
> I use "scp" to copy large file over local network, windows-7_pro.ova (about 28GB) but I it failed to import the appliance with VirtualBox.
> Checking the md5sum of the source and destination it turn out they are different.
> 
> Is "scp" reliable for large files? 

Even with "rsync" I get a different result.

Source: md5sum windows-7_pro.ova
55073f564101274bc7cd609a25c21e5b

destination:  md5sum windows-7_pro.ova
f5f3404cf1b0f654e873abe7e21e4328

Re: [gentoo-user] is "scp" reliable?

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

On Sat, May 29, 2021 at 12:54 PM <thelma@sys-concept.com> wrote:
>
> I use "scp" to copy large file over local network, windows-7_pro.ova
(about 28GB) but I it failed to import the appliance with VirtualBox.
> Checking the md5sum of the source and destination it turn out they are
different.
>
> Is "scp" reliable for large files?

In my experience scp is reliable but for large files I'd probably opt for
some version of rsync, possibly the -P option. That way if the transfer
stops for some reason you can generally pick up where it left off.

HTH,
Mark

[-- Attachment #2: Type: text/html, Size: 748 bytes --]

Re: [gentoo-user] is "scp" reliable?

[thelma]

On 5/29/21 2:00 PM, Mark Knecht wrote:
> 
> 
> On Sat, May 29, 2021 at 12:54 PM <thelma@sys-concept.com <mailto:thelma@sys-concept.com>> wrote:
>>
>> I use "scp" to copy large file over local network, windows-7_pro.ova (about 28GB) but I it failed to import the appliance with VirtualBox.
>> Checking the md5sum of the source and destination it turn out they are different.
>>
>> Is "scp" reliable for large files?
> 
> In my experience scp is reliable but for large files I'd probably opt for some version of rsync, possibly the -P option. That way if the transfer stops for some reason you can generally pick up where it left off.
> 
> HTH,
> Mark

Another mystery.
I copied the file to USB 1TB sandisk.  
md5sum check OK same as my computer 


md5sum /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
6f3348f1fb915af9c45806d947558a37  /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova

I mount the same USB 1TB sandisk on another computer and running md5sum on same file gives me different number,  why???

md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
c478cb48e2f7961cb0e3eb452df6e642  /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova

Re: [gentoo-user] is "scp" reliable?

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

On Sat, May 29, 2021 at 1:33 PM <thelma@sys-concept.com> wrote:
<SNIP>
> Another mystery.
> I copied the file to USB 1TB sandisk.
> md5sum check OK same as my computer
>
>
> md5sum
/run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> 6f3348f1fb915af9c45806d947558a37
 /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>
> I mount the same USB 1TB sandisk on another computer and running md5sum
on same file gives me different number,  why???
>
> md5sum
/run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> c478cb48e2f7961cb0e3eb452df6e642
 /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova

Different revisions of md5sum possibly?

- Mark

[-- Attachment #2: Type: text/html, Size: 938 bytes --]

Re: [gentoo-user] is "scp" reliable?

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On Sat, May 29, 2021 at 2:04 PM Mark Knecht <markknecht@gmail.com> wrote:
>
>
>
> On Sat, May 29, 2021 at 1:33 PM <thelma@sys-concept.com> wrote:
> <SNIP>
> > Another mystery.
> > I copied the file to USB 1TB sandisk.
> > md5sum check OK same as my computer
> >
> >
> > md5sum
/run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> > 6f3348f1fb915af9c45806d947558a37
 /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> >
> > I mount the same USB 1TB sandisk on another computer and running md5sum
on same file gives me different number,  why???
> >
> > md5sum
/run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> > c478cb48e2f7961cb0e3eb452df6e642
 /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>
> Different revisions of md5sum possibly?
>
> - Mark

>
On the first machine did the md5sum of the file on the internal hard drive
match the version on the USB device?

I don't know if the underlying file system type effects this calculation.
I'd hope not but I don't know.

[-- Attachment #2: Type: text/html, Size: 1616 bytes --]

Re: [gentoo-user] is "scp" reliable?

[cal]

On 5/29/21 1:32 PM, thelma@sys-concept.com wrote:
> On 5/29/21 2:00 PM, Mark Knecht wrote:
>>
>>
>> On Sat, May 29, 2021 at 12:54 PM <thelma@sys-concept.com <mailto:thelma@sys-concept.com>> wrote:
>>>
>>> I use "scp" to copy large file over local network, windows-7_pro.ova (about 28GB) but I it failed to import the appliance with VirtualBox.
>>> Checking the md5sum of the source and destination it turn out they are different.
>>>
>>> Is "scp" reliable for large files?
>>
>> In my experience scp is reliable but for large files I'd probably opt for some version of rsync, possibly the -P option. That way if the transfer stops for some reason you can generally pick up where it left off.
>>
>> HTH,
>> Mark
> 
> Another mystery.
> I copied the file to USB 1TB sandisk.  
> md5sum check OK same as my computer 
> 
> 
> md5sum /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
> 6f3348f1fb915af9c45806d947558a37  /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> 
> I mount the same USB 1TB sandisk on another computer and running md5sum on same file gives me different number,  why???
> 
> md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
> c478cb48e2f7961cb0e3eb452df6e642  /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
> 
Did you sync and unmount the partition before ejecting the drive from
the first computer?  With a file this large being copied, it is likely
that a large amount of data remains buffered/cached and will not be
fully written to the flash memory even after the copy command completes.

On the first machine, you would still see the correct md5sum because the
kernel abstracts this fact away from you.  But if you rip out the drive
and take it somewhere else without flushing those caches, you're going
to get an incomplete file.

Check if the file on the drive still md5sums the same if you plug it
back into the first machine.  Check what size it is, and whether there
are a lot of 0s at the end indicating an unfinished write.

cal

Re: [gentoo-user] is "scp" reliable?

[thelma]

On 5/29/21 3:32 PM, Mark Knecht wrote:
> 
> 
> On Sat, May 29, 2021 at 2:04 PM Mark Knecht <markknecht@gmail.com <mailto:markknecht@gmail.com>> wrote:
>>
>>
>>
>> On Sat, May 29, 2021 at 1:33 PM <thelma@sys-concept.com <mailto:thelma@sys-concept.com>> wrote:
>> <SNIP>
>> > Another mystery.
>> > I copied the file to USB 1TB sandisk.
>> > md5sum check OK same as my computer
>> >
>> >
>> > md5sum /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>> > 6f3348f1fb915af9c45806d947558a37  /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>> >
>> > I mount the same USB 1TB sandisk on another computer and running md5sum on same file gives me different number,  why???
>> >
>> > md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>> > c478cb48e2f7961cb0e3eb452df6e642  /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>
>> Different revisions of md5sum possibly?
>>
>> - Mark
> 
> 
> On the first machine did the md5sum of the file on the internal hard drive match the version on the USB device? 

Yes, the md5sum output on the  internal HD is the same as on the USB device.
 

Re: [gentoo-user] is "scp" reliable?

[thelma]

>> Another mystery.
>> I copied the file to USB 1TB sandisk.  
>> md5sum check OK same as my computer 
>>
>>
>> md5sum /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>> 6f3348f1fb915af9c45806d947558a37  /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>
>> I mount the same USB 1TB sandisk on another computer and running md5sum on same file gives me different number,  why???
>>
>> md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>> c478cb48e2f7961cb0e3eb452df6e642  /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>
> Did you sync and unmount the partition before ejecting the drive from
> the first computer?  With a file this large being copied, it is likely
> that a large amount of data remains buffered/cached and will not be
> fully written to the flash memory even after the copy command completes.
> 
> On the first machine, you would still see the correct md5sum because the
> kernel abstracts this fact away from you.  But if you rip out the drive
> and take it somewhere else without flushing those caches, you're going
> to get an incomplete file.
> 
> Check if the file on the drive still md5sums the same if you plug it
> back into the first machine.  Check what size it is, and whether there
> are a lot of 0s at the end indicating an unfinished write.
> 
> cal

Yes, I unmounted the USB device every time. 
And yes, I plug the USB device back to original machine and md5sum is correct, same as the original. 

I copied the large file over network to another box and md5sum of:  windows-7_pro_May-23-21.ova is correct same as on the original box.

I run this:   "rsync -avh [source] [destination] && rsync -avhc [source] [destination]"

above code rsync files folder on first run and if complete without issue, will run rsync again immediately while performing same file name comparison by using hash of entire file.

This i what I got:

rsync -avh windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/ && rsync -avhc windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/
sending incremental file list
windows-7_pro_May-29-21.ova

sent 30.29G bytes  received 35 bytes  115.81M bytes/sec
total size is 30.28G  speedup is 1.00
sending incremental file list
windows-7_pro_May-29-21.ova
WARNING: windows-7_pro_May-29-21.ova failed verification -- update discarded (will try again).
windows-7_pro_May-29-21.ova
ERROR: windows-7_pro_May-29-21.ova failed verification -- update discarded.

sent 33.44M bytes  received 6.47M bytes  123.38K bytes/sec
total size is 30.28G  speedup is 758.62
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]

 

Re: [gentoo-user] is "scp" reliable?

[cal]

On 5/29/21 5:42 PM, thelma@sys-concept.com wrote:
>>> Another mystery.
>>> I copied the file to USB 1TB sandisk.  
>>> md5sum check OK same as my computer 
>>>
>>>
>>> md5sum /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>>> 6f3348f1fb915af9c45806d947558a37  /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>>
>>> I mount the same USB 1TB sandisk on another computer and running md5sum on same file gives me different number,  why???
>>>
>>> md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>>> c478cb48e2f7961cb0e3eb452df6e642  /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>>
>> Did you sync and unmount the partition before ejecting the drive from
>> the first computer?  With a file this large being copied, it is likely
>> that a large amount of data remains buffered/cached and will not be
>> fully written to the flash memory even after the copy command completes.
>>
>> On the first machine, you would still see the correct md5sum because the
>> kernel abstracts this fact away from you.  But if you rip out the drive
>> and take it somewhere else without flushing those caches, you're going
>> to get an incomplete file.
>>
>> Check if the file on the drive still md5sums the same if you plug it
>> back into the first machine.  Check what size it is, and whether there
>> are a lot of 0s at the end indicating an unfinished write.
>>
>> cal
> 
> Yes, I unmounted the USB device every time. 
> And yes, I plug the USB device back to original machine and md5sum is correct, same as the original. 
> 
> I copied the large file over network to another box and md5sum of:  windows-7_pro_May-23-21.ova is correct same as on the original box.
> 
> I run this:   "rsync -avh [source] [destination] && rsync -avhc [source] [destination]"
> 
> above code rsync files folder on first run and if complete without issue, will run rsync again immediately while performing same file name comparison by using hash of entire file.
> 
> This i what I got:
> 
> rsync -avh windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/ && rsync -avhc windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/
> sending incremental file list
> windows-7_pro_May-29-21.ova
> 
> sent 30.29G bytes  received 35 bytes  115.81M bytes/sec
> total size is 30.28G  speedup is 1.00
> sending incremental file list
> windows-7_pro_May-29-21.ova
> WARNING: windows-7_pro_May-29-21.ova failed verification -- update discarded (will try again).
> windows-7_pro_May-29-21.ova
> ERROR: windows-7_pro_May-29-21.ova failed verification -- update discarded.
> 
> sent 33.44M bytes  received 6.47M bytes  123.38K bytes/sec
> total size is 30.28G  speedup is 758.62
> rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]
> 
>  
> 
rsync is emitting errors indicating the file was not transferred
correctly.  At this point I would call into question whether your second
machine is the problem rather than any of the tools you're using.  If
you have a third machine that is easy to test.  Otherwise I would run
memtest86+ and smartctl.

cal

Re: [SOLVED] [gentoo-user] is "scp" reliable?

[thelma]

On 5/29/21 7:09 PM, cal wrote:
> On 5/29/21 5:42 PM, thelma@sys-concept.com wrote:
>>>> Another mystery.
>>>> I copied the file to USB 1TB sandisk.  
>>>> md5sum check OK same as my computer 
>>>>
>>>>
>>>> md5sum /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>>>> 6f3348f1fb915af9c45806d947558a37  /run/media/joseph/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>>>
>>>> I mount the same USB 1TB sandisk on another computer and running md5sum on same file gives me different number,  why???
>>>>
>>>> md5sum /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova 
>>>> c478cb48e2f7961cb0e3eb452df6e642  /run/media/fd/SSD-1TB/business/backup/VDI/windows-7_pro_May-23-21.ova
>>>>
>>> Did you sync and unmount the partition before ejecting the drive from
>>> the first computer?  With a file this large being copied, it is likely
>>> that a large amount of data remains buffered/cached and will not be
>>> fully written to the flash memory even after the copy command completes.
>>>
>>> On the first machine, you would still see the correct md5sum because the
>>> kernel abstracts this fact away from you.  But if you rip out the drive
>>> and take it somewhere else without flushing those caches, you're going
>>> to get an incomplete file.
>>>
>>> Check if the file on the drive still md5sums the same if you plug it
>>> back into the first machine.  Check what size it is, and whether there
>>> are a lot of 0s at the end indicating an unfinished write.
>>>
>>> cal
>>
>> Yes, I unmounted the USB device every time. 
>> And yes, I plug the USB device back to original machine and md5sum is correct, same as the original. 
>>
>> I copied the large file over network to another box and md5sum of:  windows-7_pro_May-23-21.ova is correct same as on the original box.
>>
>> I run this:   "rsync -avh [source] [destination] && rsync -avhc [source] [destination]"
>>
>> above code rsync files folder on first run and if complete without issue, will run rsync again immediately while performing same file name comparison by using hash of entire file.
>>
>> This i what I got:
>>
>> rsync -avh windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/ && rsync -avhc windows-7_pro_May-29-21.ova fd@10.0.0.138:/home/fd/business/VDI/
>> sending incremental file list
>> windows-7_pro_May-29-21.ova
>>
>> sent 30.29G bytes  received 35 bytes  115.81M bytes/sec
>> total size is 30.28G  speedup is 1.00
>> sending incremental file list
>> windows-7_pro_May-29-21.ova
>> WARNING: windows-7_pro_May-29-21.ova failed verification -- update discarded (will try again).
>> windows-7_pro_May-29-21.ova
>> ERROR: windows-7_pro_May-29-21.ova failed verification -- update discarded.
>>
>> sent 33.44M bytes  received 6.47M bytes  123.38K bytes/sec
>> total size is 30.28G  speedup is 758.62
>> rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]
>>
>>  
>>
> rsync is emitting errors indicating the file was not transferred
> correctly.  At this point I would call into question whether your second
> machine is the problem rather than any of the tools you're using.  If
> you have a third machine that is easy to test.  Otherwise I would run
> memtest86+ and smartctl.
> 
> cal

Right On Cal, running memtest86+ gave me nothing but errors.  I'm surprised it compiled all the packages recently without any errors.
Putting two good stick in it and md5sum worked without a problem.   
Was able to import OVA into virtualbox without any errors. 
 

[gentoo-user] Re: [SOLVED] is "scp" reliable?

[Grant Edwards]

On 2021-05-30, thelma@sys-concept.com <thelma@sys-concept.com> wrote:

> Right On Cal, running memtest86+ gave me nothing but errors.  I'm
> surprised it compiled all the packages recently without any errors.

It might not have.

> Putting two good stick in it and md5sum worked without a problem.
> Was able to import OVA into virtualbox without any errors.

--
Grant

Re: [gentoo-user] is "scp" reliable?

[J. Roeleveld]

On Saturday, May 29, 2021 11:04:44 PM CEST Mark Knecht wrote:
> On Sat, May 29, 2021 at 1:33 PM <thelma@sys-concept.com> wrote:
> <SNIP>
> 
> > Another mystery.
> > I copied the file to USB 1TB sandisk.
> > md5sum check OK same as my computer
> > 
> > 

> Different revisions of md5sum possibly?

I have never had issues with different md5sum tools.
I often use md5sum along with sha1sum to check file-integrity of downloaded 
files. The checksums being provided by the source.

If there are differences, I would definitely suspect memory and CPU.

--
Joost

Re: [gentoo-user] is "scp" reliable?

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

On Tue, Jun 1, 2021 at 2:46 PM J. Roeleveld <joost@antarean.org> wrote:

> On Saturday, May 29, 2021 11:04:44 PM CEST Mark Knecht wrote:
> > On Sat, May 29, 2021 at 1:33 PM <thelma@sys-concept.com> wrote:
> > <SNIP>
> >
> > > Another mystery.
> > > I copied the file to USB 1TB sandisk.
> > > md5sum check OK same as my computer
> > >
> > >
>
> > Different revisions of md5sum possibly?
>
> I have never had issues with different md5sum tools.
>

Yes - it hard to imagine a bug of that seriousness not being detected.
Getting it right is the tool's raison d'etre.

[-- Attachment #2: Type: text/html, Size: 1030 bytes --]

[gentoo-user] Re: is "scp" reliable?

[Grant Edwards]

On 2021-05-29, Mark Knecht <markknecht@gmail.com> wrote:

> Different revisions of md5sum possibly?

No, not unless md5sum is seriosly and fatally broken.

MD5 is MD5.

The entire point of a hash algorithm is that it's precisely and
completely defined and the answer never varies from one implementation
to another.

-- 
Grant Edwards               grant.b.edwards        Yow! HELLO KITTY gang
                                  at               terrorizes town, family
                              gmail.com            STICKERED to death!

Re: [gentoo-user] is "scp" reliable?

[n952162]

On 6/1/21 6:42 AM, J. Roeleveld wrote:
>
> If there are differences, I would definitely suspect memory and CPU.
>
> --
> Joost
>
>
>
>
CPU?  USB was mentioned which set off alarm bells for me.  In general
though, I would suspect the media  - either source of destination.

Re: [gentoo-user] Re: is "scp" reliable?

[thelma]

On 6/1/21 9:57 AM, Grant Edwards wrote:
> On 2021-05-29, Mark Knecht <markknecht@gmail.com> wrote:
> 
>> Different revisions of md5sum possibly?
> 
> No, not unless md5sum is seriosly and fatally broken.
> 
> MD5 is MD5.
> 
> The entire point of a hash algorithm is that it's precisely and
> completely defined and the answer never varies from one implementation
> to another.

It was a problem with  memory module, I did not run memory test after assembling the PC (should have), so the Gentoo install OK, transferring standard (small files) md5sum check OK; but on larger files eg. 28GB scp, rysnc, md5sum all fail.
Running memory test give me nothing but errors, swapping the memory stick from a good PC solved the problem. 

Re: [gentoo-user] is "scp" reliable?

[J. Roeleveld]

On Tuesday, June 1, 2021 6:21:52 PM CEST n952162 wrote:
> On 6/1/21 6:42 AM, J. Roeleveld wrote:
> > If there are differences, I would definitely suspect memory and CPU.
> > 
> > --
> > Joost
> 
> CPU?  USB was mentioned which set off alarm bells for me.  In general
> though, I would suspect the media  - either source of destination.

The issue was first seen using SCP-copies. Eg, over network.
USB was only mentioned as being used as an alternative, which also didn't 
work.

--
Joost