From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 615C8158013 for ; Mon, 27 Sep 2021 14:13:43 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id BFAC7E0930; Mon, 27 Sep 2021 14:13:37 +0000 (UTC) Received: from smarthost01a.ixn.mail.zen.net.uk (smarthost01a.ixn.mail.zen.net.uk [212.23.1.20]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 2A020E0907 for ; Mon, 27 Sep 2021 14:13:35 +0000 (UTC) Received: from [82.69.80.10] (helo=wstn.localnet) by smarthost01a.ixn.mail.zen.net.uk with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUrNu-0004Fl-TB for gentoo-user@lists.gentoo.org; Mon, 27 Sep 2021 14:13:34 +0000 From: Peter Humphrey To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] How to compress lots of tarballs Date: Mon, 27 Sep 2021 15:13:34 +0100 Message-ID: <3138453.44csPzL39Z@wstn> In-Reply-To: <4346990.LvFx2qVVIh@wstn> References: <5513654.DvuYhMxLoT@wstn> <4346990.LvFx2qVVIh@wstn> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Originating-smarthost01a-IP: [82.69.80.10] Feedback-ID: 82.69.80.10 X-Archives-Salt: 0ee162e3-937d-4643-8eba-dd26e0367ba8 X-Archives-Hash: 2e6d3f24ae1b4c0b5fd5089c81041f59 On Monday, 27 September 2021 14:30:36 BST Peter Humphrey wrote: > On Monday, 27 September 2021 02:39:19 BST Adam Carter wrote: > > On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey > > > > wrote: > > > Hello list, > > > > > > I have an external USB-3 drive with various system backups. There are > > > 350 > > > .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't > > > need to compress them, so I didn't, but now I think I'm going to have > > > to. > > > Is there a reasonably efficient way to do this? > > > > find -name \*tar -exec zstd -TN {} \; > > > > Where N is the number of cores you want to allocate. zstd -T0 (or just > > zstdmt) if you want to use all the available cores. I use zstd for > > everything now as it's as good as or better than all the others in the > > general case. > > > > Parallel means it uses more than one core, so on a modern machine it is > > much faster. > > Thanks to all who've helped. I can't avoid feeling, though, that the main > bottleneck has been missed: that I have to read and write on a USB-3 drive. > It's just taken 23 minutes to copy the current system backup from USB-3 to > SATA SSD: 108GB in 8 .tar files. I was premature. In contrast to the 23 minutes to copy the files from USB-3 to internal SSD, zstd -T0 took 3:22 to compress them onto another internal SSD. I watched /bin/top and didn't see more than 250% CPU (this is a 24-CPU box) with next-to-nothing else running. The result was 65G of .tar.zst files. So, at negligible cost in CPU load*, I can achieve a 40% saving in space. Of course, I'll have to manage the process myself, and I still have to copy the compressed files back to USB-3 - but then I am retired, so what else do I have to do? :) Thanks again, all who've helped. * ...so I can continue running my 5 BOINC projects at the same time. -- Regards, Peter.