From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1QYlpj-0002yJ-Hp for garchives@archives.gentoo.org; Mon, 20 Jun 2011 21:16:36 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id EE66C1C101 for ; Mon, 20 Jun 2011 21:16:33 +0000 (UTC) Received: from ironport2-out.pppoe.ca (ironport2-out.teksavvy.com [206.248.154.181]) by pigeon.gentoo.org (Postfix) with ESMTP id 3F58C1C05A for ; Mon, 20 Jun 2011 20:50:18 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhwHAG2x/01FpZ+O/2dsb2JhbABTmBuOSXjIdYYqBJZMhw2EIg X-IronPort-AV: E=Sophos;i="4.65,396,1304308800"; d="scan'208";a="116721982" Received: from 69-165-159-142.dsl.teksavvy.com (HELO waltdnes.org) ([69.165.159.142]) by ironport2-out.pppoe.ca with SMTP; 20 Jun 2011 16:50:16 -0400 Received: by waltdnes.org (sSMTP sendmail emulation); Mon, 20 Jun 2011 16:51:06 -0400 From: "Walter Dnes" Date: Mon, 20 Jun 2011 16:51:06 -0400 To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] Use split to break up a 10GB file binary? Message-ID: <20110620205106.GB14933@waltdnes.org> References: Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Archives-Salt: X-Archives-Hash: 8ee13fb8e2435888933d1738fcc1e86c On Mon, Jun 20, 2011 at 08:25:57AM -0700, Mark Knecht wrote > Hi, > Is split an appropriate program to use to break a single 10GB file > into 100 100MB files to transfer over the net using rsync, and then > use cat to reassemble? > > Is there some better way to do this? That's what split was written for. I can't think of anything better. BTW, what type of data is the 10 gig file? If it's text, then consider using zip or bzip2 on each of the fragments before transferring. If it's an already compressed binary format, then don't waste time attempting further compression. -- Walter Dnes