[gentoo-user] Use split to break up a 10GB file binary?

[gentoo-user] Use split to break up a 10GB file binary?

[Mark Knecht]

Hi,
   Is split an appropriate program to use to break a single 10GB file
into 100 100MB files to transfer over the net using rsync, and then
use cat to reassemble?

   Is there some better way to do this?

Thanks,
Mark

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Paul Hartman]

On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@gmail.com> wrote:
> Hi,
>   Is split an appropriate program to use to break a single 10GB file
> into 100 100MB files to transfer over the net using rsync, and then
> use cat to reassemble?

I think it should work just fine. I've split huge files into huge
chunks and never had any issues.

>   Is there some better way to do this?

I wonder if splitting is even necessary; rsync will analyze the file
and only transmit the differences, right?. So I'd think that even if
the transfer fails, a retry would pick up where it left off (assuming
rsync keeps the failed copy).

Also check out net-misc/unison. It seems to be designed for just this
sort of thing.

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Mark Knecht]

On Mon, Jun 20, 2011 at 9:21 AM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@gmail.com> wrote:
>> Hi,
>>   Is split an appropriate program to use to break a single 10GB file
>> into 100 100MB files to transfer over the net using rsync, and then
>> use cat to reassemble?
>
> I think it should work just fine. I've split huge files into huge
> chunks and never had any issues.
>
>>   Is there some better way to do this?
>
> I wonder if splitting is even necessary; rsync will analyze the file
> and only transmit the differences, right?. So I'd think that even if
> the transfer fails, a retry would pick up where it left off (assuming
> rsync keeps the failed copy).
>
> Also check out net-misc/unison. It seems to be designed for just this
> sort of thing.

I'll check them out Paul. Thanks for the extra ideas.

I just tried it as an experiment between two machines here. Using
split I broke the 10GB file into 100MB pieces, used rsync to get the
pieces to my laptop, and then used cat to reassemble. The size of the
results compares to the byte so that looks good.

I was unsure whether rsync would restart where it left off or whether
it would start over from the beginning. It's one huge file so it would
be painful if it did the latter. This way I know my risk is at most
100MB, or maybe 20MB if I break the original up into smaller pieces.

Cheers,
Mark

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Michael Hampicke]

> I was unsure whether rsync would restart where it left off or whether
> it would start over from the beginning. It's one huge file so it would
> be painful if it did the latter. This way I know my risk is at most
> 100MB, or maybe 20MB if I break the original up into smaller pieces.

check out the --partial option of rsync, this is what you need :)

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Allan Gottlieb]

On Mon, Jun 20 2011, Paul Hartman wrote:

> On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@gmail.com> wrote:
>> Hi,
>>   Is split an appropriate program to use to break a single 10GB file
>> into 100 100MB files to transfer over the net using rsync, and then
>> use cat to reassemble?
>
> I think it should work just fine. I've split huge files into huge
> chunks and never had any issues.
>
>>   Is there some better way to do this?
>
> I wonder if splitting is even necessary; rsync will analyze the file
> and only transmit the differences, right?. So I'd think that even if
> the transfer fails, a retry would pick up where it left off (assuming
> rsync keeps the failed copy).

I believe that is the --partial option.

allan

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Mark Knecht]

On Mon, Jun 20, 2011 at 10:48 AM, Allan Gottlieb <gottlieb@nyu.edu> wrote:
> On Mon, Jun 20 2011, Paul Hartman wrote:
>
>> On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@gmail.com> wrote:
>>> Hi,
>>>   Is split an appropriate program to use to break a single 10GB file
>>> into 100 100MB files to transfer over the net using rsync, and then
>>> use cat to reassemble?
>>
>> I think it should work just fine. I've split huge files into huge
>> chunks and never had any issues.
>>
>>>   Is there some better way to do this?
>>
>> I wonder if splitting is even necessary; rsync will analyze the file
>> and only transmit the differences, right?. So I'd think that even if
>> the transfer fails, a retry would pick up where it left off (assuming
>> rsync keeps the failed copy).
>
> I believe that is the --partial option.
>
> allan

Yes, that looks like what I want.

Is there an option to have rsync keep trying if the other end goes
down for a while or would I need to put the rsync command into a cron
job so that it restarts every hour until it's completed the transfers?
I don't see one scanning through the many, many options to rsync.

Thanks,
Mark

[gentoo-user] Re: Use split to break up a 10GB file binary?

[Francesco Talamona]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]

On Monday 20 June 2011, Paul Hartman wrote:
> On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht <markknecht@gmail.com> 
wrote:
> > Hi,
> >   Is split an appropriate program to use to break a single 10GB
> > file into 100 100MB files to transfer over the net using rsync,
> > and then use cat to reassemble?
> 
> I think it should work just fine. I've split huge files into huge
> chunks and never had any issues.
> 
> >   Is there some better way to do this?
> 
> I wonder if splitting is even necessary; rsync will analyze the file
> and only transmit the differences, right?. So I'd think that even if
> the transfer fails, a retry would pick up where it left off (assuming
> rsync keeps the failed copy).
> 
> Also check out net-misc/unison. It seems to be designed for just this
> sort of thing.

Unison is wonderful for more complex tasks but is very inefficient with 
large files. As a matter of fact it uses rsync algorithm in order to get 
good performance, still isn't the best choice in this scenario, see: 
http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-
manual.html#speeding

I keep in sync the user home across two computer, and I use both of them 
daily. It would be impossible without unison, but the 20GB virtual 
machine is excluded from the sync.

Cheers
	Francesco

-- 
Linux Version 2.6.39-gentoo-r1, Compiled #1 SMP PREEMPT Thu Jun 9 
11:20:57 CEST 2011
Two 1GHz AMD Athlon 64 X2 Processors, 4GB RAM, 4021.84 Bogomips Total
aemaeth

[-- Attachment #2: Type: text/html, Size: 6573 bytes --]

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Nils Larsson]

On Monday 20 June 2011 08:25:57 Mark Knecht wrote:
> Hi,
>    Is split an appropriate program to use to break a single 10GB file
> into 100 100MB files to transfer over the net using rsync, and then
> use cat to reassemble?
> 
>    Is there some better way to do this?

Maybe not better, but you could use a torrent application that support magnet 
links to send files across the net. No need to send chop it up in chunks then.

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Mark Knecht]

On Mon, Jun 20, 2011 at 12:17 PM, Nils Larsson <ni1s@nerdshack.com> wrote:
> On Monday 20 June 2011 08:25:57 Mark Knecht wrote:
>> Hi,
>>    Is split an appropriate program to use to break a single 10GB file
>> into 100 100MB files to transfer over the net using rsync, and then
>> use cat to reassemble?
>>
>>    Is there some better way to do this?
>
> Maybe not better, but you could use a torrent application that support magnet
> links to send files across the net. No need to send chop it up in chunks then.

Interesting idea. I'll keep in in mind.

I've already tested that chopping the file with split and reassembling
it locally on another machine works. One advantage to chopping it up
is that because I have two ISPs in my house I can send half over each
ISP and speed things up considerably. The transfer to the remote
machine is now in progress and I've transferred about 20 of the 402
files I need to send so things are progressing splendidly at this
time.

I loaded unison on one of my local machines and will play with that.
Maybe I'll add some form of bittorrent into the mix also.

Cheers,
Mark

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Walter Dnes]

On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote
> Hi,
>    Is split an appropriate program to use to break a single 10GB file
> into 100 100MB files to transfer over the net using rsync, and then
> use cat to reassemble?
> 
>    Is there some better way to do this?

  That's what split was written for.  I can't think of anything better.
BTW, what type of data is the 10 gig file?  If it's text, then consider
using zip or bzip2 on each of the fragments before transferring.  If
it's an already compressed binary format, then don't waste time
attempting further compression.

-- 
Walter Dnes <waltdnes@waltdnes.org>

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Mark Knecht]

On Mon, Jun 20, 2011 at 1:51 PM, Walter Dnes <waltdnes@waltdnes.org> wrote:
> On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote
>> Hi,
>>    Is split an appropriate program to use to break a single 10GB file
>> into 100 100MB files to transfer over the net using rsync, and then
>> use cat to reassemble?
>>
>>    Is there some better way to do this?
>
>  That's what split was written for.  I can't think of anything better.
> BTW, what type of data is the 10 gig file?  If it's text, then consider
> using zip or bzip2 on each of the fragments before transferring.  If
> it's an already compressed binary format, then don't waste time
> attempting further compression.
>
> --
> Walter Dnes <waltdnes@waltdnes.org>

Hi Walter.

It's a Virtualbox VM exported from virtualbox-4.0.8. It consumes about
40GB on disk, compressed by Virtualbox to about 10GB by their 'Export
Appliance' feature.

I actually did try compressing the exported 10GB file with gzip &
bzip2 before splitting it. None of those provided any compression. I
didn't try the spilt outputs as I figured they are just binary chunks
and wouldn't compress either.

Thanks for the info.

Cheers,
Mark

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1471 bytes --]

On Monday 20 Jun 2011 22:06:59 Mark Knecht wrote:
> On Mon, Jun 20, 2011 at 1:51 PM, Walter Dnes <waltdnes@waltdnes.org> wrote:
> > On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote
> > 
> >> Hi,
> >>    Is split an appropriate program to use to break a single 10GB file
> >> into 100 100MB files to transfer over the net using rsync, and then
> >> use cat to reassemble?
> >> 
> >>    Is there some better way to do this?
> > 
> >  That's what split was written for.  I can't think of anything better.
> > BTW, what type of data is the 10 gig file?  If it's text, then consider
> > using zip or bzip2 on each of the fragments before transferring.  If
> > it's an already compressed binary format, then don't waste time
> > attempting further compression.
> > 
> > --
> > Walter Dnes <waltdnes@waltdnes.org>
> 
> Hi Walter.
> 
> It's a Virtualbox VM exported from virtualbox-4.0.8. It consumes about
> 40GB on disk, compressed by Virtualbox to about 10GB by their 'Export
> Appliance' feature.
> 
> I actually did try compressing the exported 10GB file with gzip &
> bzip2 before splitting it. None of those provided any compression. I
> didn't try the spilt outputs as I figured they are just binary chunks
> and wouldn't compress either.
> 
> Thanks for the info.

You could try:

 wget -c -t 0

to retry (forever) should the transfer fail for some reason and not bother 
with splitting the file.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Indi]

On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote:
> Hi,
>    Is split an appropriate program to use to break a single 10GB file
> into 100 100MB files to transfer over the net using rsync, and then
> use cat to reassemble?
> 
>    Is there some better way to do this?
> 

Just using rsync by itself would probably be a great deal faster, 
unless you have some undisclosed reason for wanting to split it up.

-- 
caveat utilitor 
♫ ❤ ♫ ❤ ♫ ❤ ♫ 

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Mark Knecht]

On Mon, Jun 20, 2011 at 2:46 PM, Indi <thebeelzebubtrigger@gmail.com> wrote:
> On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote:
>> Hi,
>>    Is split an appropriate program to use to break a single 10GB file
>> into 100 100MB files to transfer over the net using rsync, and then
>> use cat to reassemble?
>>
>>    Is there some better way to do this?
>>
>
> Just using rsync by itself would probably be a great deal faster,
> unless you have some undisclosed reason for wanting to split it up.

Hi,
   Nothing technical that's undisclosed. My original reason was not
knowing what rsync did in the case of errors I simply didn't want to
start over on such a big file. I figured there was little to lose by
stitching it back together are the other end and I could always figure
out exactly what file had failed.

   That said I don't think there's much difference in the speed. In my
case (and I think others will have a similar case) my uploads speeds
are far lower than download. I get about 8MB/S download but only about
250KB/S upload. It's that low speed that's dominating everything else.
When I first tried transferring the big file the intermediate speeds
rsync was reporting were very similar.

Cheers,
Mark

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Indi]

On Mon, Jun 20, 2011 at 05:28:05PM -0700, Mark Knecht wrote:
> On Mon, Jun 20, 2011 at 2:46 PM, Indi <thebeelzebubtrigger@gmail.com> wrote:
> > On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote:
> >> Hi,
> >>    Is split an appropriate program to use to break a single 10GB file
> >> into 100 100MB files to transfer over the net using rsync, and then
> >> use cat to reassemble?
> >>
> >>    Is there some better way to do this?
> >>
> >
> > Just using rsync by itself would probably be a great deal faster,
> > unless you have some undisclosed reason for wanting to split it up.
> 
> Hi,
>    Nothing technical that's undisclosed. My original reason was not
> knowing what rsync did in the case of errors I simply didn't want to
> start over on such a big file. I figured there was little to lose by
> stitching it back together are the other end and I could always figure
> out exactly what file had failed.
> 
>    That said I don't think there's much difference in the speed. In my
> case (and I think others will have a similar case) my uploads speeds
> are far lower than download. I get about 8MB/S download but only about
> 250KB/S upload. It's that low speed that's dominating everything else.
> When I first tried transferring the big file the intermediate speeds
> rsync was reporting were very similar.
> 

Of course I was referring to the time taken by the extra steps, not the 
transfer speed. :)

You might check man rsync though, it does what you need without
splitting and reassembling files..

-- 
caveat utilitor 
♫ ❤ ♫ ❤ ♫ ❤ ♫ 

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Peter Humphrey]

On Tuesday 21 June 2011 01:28:05 Mark Knecht wrote:

> I get about 8MB/S download but only about 250KB/S upload.

Should those B's be b's? In other words, bits instead of bytes? Or are you 
on something much faster than DSL? In either case the S's ought to be s's.


On ADSL I get about 900KB/s down and 60KB/s up.

-- 
Rgds
Peter

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Dale]

Peter Humphrey wrote:
> On Tuesday 21 June 2011 01:28:05 Mark Knecht wrote:
>
>    
>> I get about 8MB/S download but only about 250KB/S upload.
>>      
> Should those B's be b's? In other words, bits instead of bytes? Or are you
> on something much faster than DSL? In either case the S's ought to be s's.
>
>
> On ADSL I get about 900KB/s down and 60KB/s up.
>
>    

My neighbor has DSL and they get about that.  I noticed it the other day 
when they needed help downloading and installing something.  It was 
~30Mbs or so and it took only a few seconds to get it.  Made my lowly 
768KBs look really bad.  I just thought back to the old dial-up days and 
3Kbs.  lol   Yea, OOo and KDE updates were lots of fun.

I get those case sizes mixed up sometimes too.  Maybe it is the pinky 
that hits the shift key that gets confused. < sighs >

Dale

:-)  :-)

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Mark Knecht]

On Tue, Jun 21, 2011 at 8:11 AM, Peter Humphrey
<peter@humphrey.ukfsn.org> wrote:
> On Tuesday 21 June 2011 01:28:05 Mark Knecht wrote:
>
>> I get about 8MB/S download but only about 250KB/S upload.
>
> Should those B's be b's? In other words, bits instead of bytes? Or are you
> on something much faster than DSL? In either case the S's ought to be s's.
>
>
> On ADSL I get about 900KB/s down and 60KB/s up.
>
> --
> Rgds
> Peter

Bytes, not bits. I have two ISPs, Comcast & Verizon. I needed the
redundancy as Comcast for a long time was very unreliable. It's been
better in the last year. That said Comcast speeds vary a lot based on
what I suppose my neighbors are doing as cable modem is a shared
interface. The Verizon 3Mbps service speeds are pretty constant.

Measured this morning using Speakeasy's page with San Francisco as the
other end:

http://www.speakeasy.net/speedtest/

Comcast cable modem - 22.38Mbps download, 4.27Mbps upload
Verizon DSL - 2.89Mbps download, 0.74Mbps upload

In my home I'm 54Mbps wireless to the Comcast router which might
effect things a bit but directly connected to the DSL router. If one
goes down I just change a couple of Ethernet cables and start using
the other. I suspect that if I went to my wife's machine right now I'd
get close to 40Mbps.

In reality I find that _real_ transfers to other Gentoo computers
using rsync typically run around 50% of those numbers averaged over
time.

Point taken about s/S. I didn't know there was an accepted standard to use 's'.

Cheers,
Mark

Re: [gentoo-user] Use split to break up a 10GB file binary?

[Peter Humphrey]

On Tuesday 21 June 2011 16:36:32 Mark Knecht wrote:

> Point taken about s/S. I didn't know there was an accepted standard to
> use 's'.

Come to think of it, it may not apply to you (in which case apologies): it's 
an SI unit.

-- 
Rgds
Peter