Re: [gentoo-user] Use split to break up a 10GB file binary?

[Mark Knecht <markknecht@gmail.com>]

On Mon, Jun 20, 2011 at 9:21 AM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@gmail.com> wrote:
>> Hi,
>>   Is split an appropriate program to use to break a single 10GB file
>> into 100 100MB files to transfer over the net using rsync, and then
>> use cat to reassemble?
>
> I think it should work just fine. I've split huge files into huge
> chunks and never had any issues.
>
>>   Is there some better way to do this?
>
> I wonder if splitting is even necessary; rsync will analyze the file
> and only transmit the differences, right?. So I'd think that even if
> the transfer fails, a retry would pick up where it left off (assuming
> rsync keeps the failed copy).
>
> Also check out net-misc/unison. It seems to be designed for just this
> sort of thing.

I'll check them out Paul. Thanks for the extra ideas.

I just tried it as an experiment between two machines here. Using
split I broke the 10GB file into 100MB pieces, used rsync to get the
pieces to my laptop, and then used cat to reassemble. The size of the
results compares to the byte so that looks good.

I was unsure whether rsync would restart where it left off or whether
it would start over from the beginning. It's one huge file so it would
be painful if it did the latter. This way I know my risk is at most
100MB, or maybe 20MB if I break the original up into smaller pieces.

Cheers,
Mark