From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1QYlwM-0003e6-5O for garchives@archives.gentoo.org; Mon, 20 Jun 2011 21:23:26 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 9E1BD1C0FA for ; Mon, 20 Jun 2011 21:23:25 +0000 (UTC) Received: from mail-qw0-f53.google.com (mail-qw0-f53.google.com [209.85.216.53]) by pigeon.gentoo.org (Postfix) with ESMTP id BCE841C196 for ; Mon, 20 Jun 2011 21:07:05 +0000 (UTC) Received: by qwb7 with SMTP id 7so1894325qwb.40 for ; Mon, 20 Jun 2011 14:06:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=Gw48AcAJonc99GZPDmq8M7v0iVKIlj3iuewogoLJP8A=; b=AObu7tbeUYEYwToewf5QP/g5xucCAUhCpm6WQuvSAbJhRb1NZx9uHA7VJ8YBN4fWpy Omu/mN8De/794/x27l8PE/TbLwqe4HJSUt2soQDoUhHcty1a6JUZPU5RqfUpOJst9eEj WUU7M4/wRxB9JN7W00YLfhI2SCLNcfmUS8pf8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=yFn2BGDkm35z162KaS26Lsg62NMZiQb5oZUuF16rw9o1cDBQEMMM+FTWdl2FyqE4p6 mXvR/cDRXMTECwUHxjqpa5YQPbILoBYxX36MPTtnEkqMrJ2g8il/lFTzjEEoDM61gwHk DGeSdhxCXEKBekeqUKIwYXLEVP39NdgBm/DHs= Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Received: by 10.224.137.133 with SMTP id w5mr4621768qat.178.1308604019682; Mon, 20 Jun 2011 14:06:59 -0700 (PDT) Received: by 10.224.61.14 with HTTP; Mon, 20 Jun 2011 14:06:59 -0700 (PDT) In-Reply-To: <20110620205106.GB14933@waltdnes.org> References: <20110620205106.GB14933@waltdnes.org> Date: Mon, 20 Jun 2011 14:06:59 -0700 Message-ID: Subject: Re: [gentoo-user] Use split to break up a 10GB file binary? From: Mark Knecht To: gentoo-user@lists.gentoo.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Archives-Salt: X-Archives-Hash: 0927c4d7a268dbd9175611e46acc652b On Mon, Jun 20, 2011 at 1:51 PM, Walter Dnes wrote: > On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote >> Hi, >> =C2=A0 =C2=A0Is split an appropriate program to use to break a single 10= GB file >> into 100 100MB files to transfer over the net using rsync, and then >> use cat to reassemble? >> >> =C2=A0 =C2=A0Is there some better way to do this? > > =C2=A0That's what split was written for. =C2=A0I can't think of anything = better. > BTW, what type of data is the 10 gig file? =C2=A0If it's text, then consi= der > using zip or bzip2 on each of the fragments before transferring. =C2=A0If > it's an already compressed binary format, then don't waste time > attempting further compression. > > -- > Walter Dnes Hi Walter. It's a Virtualbox VM exported from virtualbox-4.0.8. It consumes about 40GB on disk, compressed by Virtualbox to about 10GB by their 'Export Appliance' feature. I actually did try compressing the exported 10GB file with gzip & bzip2 before splitting it. None of those provided any compression. I didn't try the spilt outputs as I figured they are just binary chunks and wouldn't compress either. Thanks for the info. Cheers, Mark