From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1QYhvV-0007MM-Ik for garchives@archives.gentoo.org; Mon, 20 Jun 2011 17:06:17 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 054831C0BC for ; Mon, 20 Jun 2011 17:06:16 +0000 (UTC) Received: from mail-qw0-f53.google.com (mail-qw0-f53.google.com [209.85.216.53]) by pigeon.gentoo.org (Postfix) with ESMTP id 90B031C050 for ; Mon, 20 Jun 2011 16:30:02 +0000 (UTC) Received: by qwb7 with SMTP id 7so1747960qwb.40 for ; Mon, 20 Jun 2011 09:30:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=AKHebD5crqVmemFi0cPEWEe5a1ncvVnFYeaOeboJPUs=; b=t5ri4VlCLXRcBletOXYl8Mp3kv1LFXtX36RSTqc7ruqEMjjq9M8jQ8EebTgIy27iJl Ag0DBKvlDyB/8tiepTnUxT/BCt8JZBqlSdptrckdP7/HATzmCIIQQT8UEVGAOWZiahCQ EfyTorNxgI5DQcDW5t1Dd9am47YAgdKwgBzr8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=R8Jg36pDac8NrWOx+TlBIKl4TROLKKgkya/0gk4jPIg4t0eOcfPKZROp7enQ7lJPjE v/gLYM/AhfD9gkrSWWIJt2EsR8IbaDpEa881lZPiUn3Bk/B9KahBH61m5GvjrOqZz6z9 QmWMN/aLt1E6/+O/+QlM0X9Mu6juy+a2/jlqs= Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Received: by 10.224.67.72 with SMTP id q8mr3422412qai.278.1308587402077; Mon, 20 Jun 2011 09:30:02 -0700 (PDT) Received: by 10.224.61.14 with HTTP; Mon, 20 Jun 2011 09:30:02 -0700 (PDT) In-Reply-To: References: Date: Mon, 20 Jun 2011 09:30:02 -0700 Message-ID: Subject: Re: [gentoo-user] Use split to break up a 10GB file binary? From: Mark Knecht To: gentoo-user@lists.gentoo.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Archives-Salt: X-Archives-Hash: 78869f19e1360bd64c205098142e4f76 On Mon, Jun 20, 2011 at 9:21 AM, Paul Hartman wrote: > On Mon, Jun 20, 2011 at 10:25 AM, Mark Knecht wrot= e: >> Hi, >> =C2=A0 Is split an appropriate program to use to break a single 10GB fil= e >> into 100 100MB files to transfer over the net using rsync, and then >> use cat to reassemble? > > I think it should work just fine. I've split huge files into huge > chunks and never had any issues. > >> =C2=A0 Is there some better way to do this? > > I wonder if splitting is even necessary; rsync will analyze the file > and only transmit the differences, right?. So I'd think that even if > the transfer fails, a retry would pick up where it left off (assuming > rsync keeps the failed copy). > > Also check out net-misc/unison. It seems to be designed for just this > sort of thing. I'll check them out Paul. Thanks for the extra ideas. I just tried it as an experiment between two machines here. Using split I broke the 10GB file into 100MB pieces, used rsync to get the pieces to my laptop, and then used cat to reassemble. The size of the results compares to the byte so that looks good. I was unsure whether rsync would restart where it left off or whether it would start over from the beginning. It's one huge file so it would be painful if it did the latter. This way I know my risk is at most 100MB, or maybe 20MB if I break the original up into smaller pieces. Cheers, Mark