From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1QYmbq-0007Ei-Gz for garchives@archives.gentoo.org; Mon, 20 Jun 2011 22:06:18 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 940031C180 for ; Mon, 20 Jun 2011 22:06:17 +0000 (UTC) Received: from mail-wy0-f181.google.com (mail-wy0-f181.google.com [74.125.82.181]) by pigeon.gentoo.org (Postfix) with ESMTP id 888F31C021 for ; Mon, 20 Jun 2011 21:57:50 +0000 (UTC) Received: by wyb28 with SMTP id 28so3433508wyb.40 for ; Mon, 20 Jun 2011 14:57:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:reply-to:to:subject:date:user-agent :references:in-reply-to:mime-version:content-type :content-transfer-encoding:message-id; bh=f1UsPTas/BFMecxETMYr1q1sPiugw2o3unH958yhNeA=; b=Aq1swmKesWxGI6j1eex1tOdugxveBx4BqlpISZOpX82uguafacOBFUFgMUDPSYnAKq +9GIcn9oP6puROyBO2UbJ5DuL47uBf0VNIaSM58loJ7+pR8v3JcHuVLzNM+01Tzy7AS5 nQqIjFWcgFbUvt0ulqjDhbrSE1FFM8TOIvFT8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:reply-to:to:subject:date:user-agent:references:in-reply-to :mime-version:content-type:content-transfer-encoding:message-id; b=UN8sCxL1PyMq10gOYPhuliF1PtyOv6JZACNYd7cj81gTJBU8Vvyav7d08l3pJEhKEe CEhtV9bYdNtzbpA0WzihJsOpkuuLLnoLyrdvENEuscDj4PYrfIAXEJyDNzt3lyl88Tc2 tKHqLL+ewp4nNsM7lCVI5lBAt014MAQFMb+Oo= Received: by 10.227.174.73 with SMTP id s9mr745678wbz.75.1308607069754; Mon, 20 Jun 2011 14:57:49 -0700 (PDT) Received: from dell_xps.localnet (230.3.169.217.in-addr.arpa [217.169.3.230]) by mx.google.com with ESMTPS id gb6sm3519795wbb.0.2011.06.20.14.57.47 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 20 Jun 2011 14:57:48 -0700 (PDT) From: Mick To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] Use split to break up a 10GB file binary? Date: Mon, 20 Jun 2011 22:58:20 +0100 User-Agent: KMail/1.13.7 (Linux/2.6.38-gentoo-r6; KDE/4.6.3; x86_64; ; ) References: <20110620205106.GB14933@waltdnes.org> In-Reply-To: Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2818281.iaICGhxCT1"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <201106202258.29616.michaelkintzios@gmail.com> X-Archives-Salt: X-Archives-Hash: 4d83f2f1ef4ec69c1daa9faf68e5be1b --nextPart2818281.iaICGhxCT1 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On Monday 20 Jun 2011 22:06:59 Mark Knecht wrote: > On Mon, Jun 20, 2011 at 1:51 PM, Walter Dnes wrot= e: > > On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote > >=20 > >> Hi, > >> Is split an appropriate program to use to break a single 10GB file > >> into 100 100MB files to transfer over the net using rsync, and then > >> use cat to reassemble? > >>=20 > >> Is there some better way to do this? > >=20 > > That's what split was written for. I can't think of anything better. > > BTW, what type of data is the 10 gig file? If it's text, then consider > > using zip or bzip2 on each of the fragments before transferring. If > > it's an already compressed binary format, then don't waste time > > attempting further compression. > >=20 > > -- > > Walter Dnes >=20 > Hi Walter. >=20 > It's a Virtualbox VM exported from virtualbox-4.0.8. It consumes about > 40GB on disk, compressed by Virtualbox to about 10GB by their 'Export > Appliance' feature. >=20 > I actually did try compressing the exported 10GB file with gzip & > bzip2 before splitting it. None of those provided any compression. I > didn't try the spilt outputs as I figured they are just binary chunks > and wouldn't compress either. >=20 > Thanks for the info. You could try: wget -c -t 0 to retry (forever) should the transfer fail for some reason and not bother= =20 with splitting the file. =2D-=20 Regards, Mick --nextPart2818281.iaICGhxCT1 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iEYEABECAAYFAk3/woUACgkQVTDTR3kpaLb74QCfbcQgZ+JuRIr74putwNlx7mVj E58AoN4xWC11MeSFksCVm4NcsUP26ayU =Xzb3 -----END PGP SIGNATURE----- --nextPart2818281.iaICGhxCT1--