[gentoo-user] How to compress lots of tarballs

[gentoo-user] How to compress lots of tarballs

[Peter Humphrey]

Hello list,

I have an external USB-3 drive with various system backups. There are 350 .tar 
files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't need to 
compress them, so I didn't, but now I think I'm going to have to. Is there a 
reasonably efficient way to do this? I have 500GB spare space on /dev/sda, and 
the machine runs constantly.

-- 
Regards,
Peter.

Re: [gentoo-user] How to compress lots of tarballs

[Simon Thelen]

[2021-09-26 11:57] Peter Humphrey <peter@prh.myzen.co.uk>
> part       text/plain                 382
> Hello list,
Hi,

> I have an external USB-3 drive with various system backups. There are 350 .tar
> files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't need to
> compress them, so I didn't, but now I think I'm going to have to. Is there a
> reasonably efficient way to do this? I have 500GB spare space on /dev/sda, and
> the machine runs constantly.
Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
then:
mount USB-3 /mnt; cd /mnt; lzip *

The archiver you chose will compress the file and add the appropriate
extension all on its own and tar will use that (and the file magic) to
find the appropriate decompresser when you want to extract files later
(you can use `tar tf' to test if you want).

--
Simon Thelen

Re: [gentoo-user] How to compress lots of tarballs

[Ramon Fischer]

[-- Attachment #1.1: Type: text/plain, Size: 1605 bytes --]

In addition to this, you may want to use the parallel implementations of 
"gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are 
"pigz"[1], "pixz"[2], "pbzip2"[3], or "zstmt" (within package 
"app-arch/zstd")[4] in order to increase performance:

    $ cd <path_to_mounted_backup_partition>
    $ for tar_archive in *.tar; do pixz "${tar_archive}"; done

-Ramon

[1]
* https://www.zlib.net/pigz/

[2]
* https://github.com/vasi/pixz

[3]
* https://launchpad.net/pbzip2
* http://compression.ca/pbzip2/

[4]
* https://facebook.github.io/zstd/


On 26/09/2021 13:36, Simon Thelen wrote:
> [2021-09-26 11:57] Peter Humphrey <peter@prh.myzen.co.uk>
>> part       text/plain                 382
>> Hello list,
> Hi,
>
>> I have an external USB-3 drive with various system backups. There are 350 .tar
>> files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't need to
>> compress them, so I didn't, but now I think I'm going to have to. Is there a
>> reasonably efficient way to do this? I have 500GB spare space on /dev/sda, and
>> the machine runs constantly.
> Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
> then:
> mount USB-3 /mnt; cd /mnt; lzip *
>
> The archiver you chose will compress the file and add the appropriate
> extension all on its own and tar will use that (and the file magic) to
> find the appropriate decompresser when you want to extract files later
> (you can use `tar tf' to test if you want).
>
> --
> Simon Thelen
>

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-user] How to compress lots of tarballs

[Ramon Fischer]

[-- Attachment #1.1: Type: text/plain, Size: 1879 bytes --]

Addendum:

To complete the list. Here the parallel implementation of "lzip":

    "plzip": https://www.nongnu.org/lzip/plzip.html

-Ramon

On 26/09/2021 14:23, Ramon Fischer wrote:
> In addition to this, you may want to use the parallel implementations 
> of "gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are 
> "pigz"[1], "pixz"[2], "pbzip2"[3], or "zstmt" (within package 
> "app-arch/zstd")[4] in order to increase performance:
>
>    $ cd <path_to_mounted_backup_partition>
>    $ for tar_archive in *.tar; do pixz "${tar_archive}"; done
>
> -Ramon
>
> [1]
> * https://www.zlib.net/pigz/
>
> [2]
> * https://github.com/vasi/pixz
>
> [3]
> * https://launchpad.net/pbzip2
> * http://compression.ca/pbzip2/
>
> [4]
> * https://facebook.github.io/zstd/
>
>
> On 26/09/2021 13:36, Simon Thelen wrote:
>> [2021-09-26 11:57] Peter Humphrey <peter@prh.myzen.co.uk>
>>> part       text/plain 382
>>> Hello list,
>> Hi,
>>
>>> I have an external USB-3 drive with various system backups. There 
>>> are 350 .tar
>>> files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't 
>>> need to
>>> compress them, so I didn't, but now I think I'm going to have to. Is 
>>> there a
>>> reasonably efficient way to do this? I have 500GB spare space on 
>>> /dev/sda, and
>>> the machine runs constantly.
>> Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
>> then:
>> mount USB-3 /mnt; cd /mnt; lzip *
>>
>> The archiver you chose will compress the file and add the appropriate
>> extension all on its own and tar will use that (and the file magic) to
>> find the appropriate decompresser when you want to extract files later
>> (you can use `tar tf' to test if you want).
>>
>> -- 
>> Simon Thelen
>>
>

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-user] How to compress lots of tarballs

[Peter Humphrey]

On Sunday, 26 September 2021 13:25:24 BST Ramon Fischer wrote:
> Addendum:
> 
> To complete the list. Here the parallel implementation of "lzip":
> 
>     "plzip": https://www.nongnu.org/lzip/plzip.html
> 
> -Ramon
> 
> On 26/09/2021 14:23, Ramon Fischer wrote:
> > In addition to this, you may want to use the parallel implementations
> > of "gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are
> > "pigz"[1], "pixz"[2], "pbzip2"[3], or "zstmt" (within package
> > "app-arch/zstd")[4] in order to increase performance:
> > 
> >    $ cd <path_to_mounted_backup_partition>
> >    $ for tar_archive in *.tar; do pixz "${tar_archive}"; done
> > 
> > -Ramon
> > 
> > [1]
> > * https://www.zlib.net/pigz/
> > 
> > [2]
> > * https://github.com/vasi/pixz
> > 
> > [3]
> > * https://launchpad.net/pbzip2
> > * http://compression.ca/pbzip2/
> > 
> > [4]
> > * https://facebook.github.io/zstd/
> > 
> > On 26/09/2021 13:36, Simon Thelen wrote:
> >> [2021-09-26 11:57] Peter Humphrey <peter@prh.myzen.co.uk>
> >> 
> >>> part       text/plain 382
> >>> Hello list,
> >> 
> >> Hi,
> >> 
> >>> I have an external USB-3 drive with various system backups. There
> >>> are 350 .tar
> >>> files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't
> >>> need to
> >>> compress them, so I didn't, but now I think I'm going to have to. Is
> >>> there a
> >>> reasonably efficient way to do this? I have 500GB spare space on
> >>> /dev/sda, and
> >>> the machine runs constantly.
> >> 
> >> Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
> >> then:
> >> mount USB-3 /mnt; cd /mnt; lzip *
> >> 
> >> The archiver you chose will compress the file and add the appropriate
> >> extension all on its own and tar will use that (and the file magic) to
> >> find the appropriate decompresser when you want to extract files later
> >> (you can use `tar tf' to test if you want).

Thank you both. Now, as it's a single USB-3 drive, what advantage would a 
parallel implementation confer? I assume I'd be better compressing from 
external to SATA, then writing back, or is that wrong?

Or, I could connect a second USB-3 drive to a different interface, then read 
from one and write to the other, with or without the SATA between.

-- 
Regards,
Peter.

Re: [gentoo-user] How to compress lots of tarballs

[antlists]

On 26/09/2021 16:38, Peter Humphrey wrote:
> Or, I could connect a second USB-3 drive to a different interface, then read
> from one and write to the other, with or without the SATA between.

If you've got a second drive, consider changing your strategy ...

First of all, you want eSATA or USB3 for the speed ...

Format the drive with lvm, and create an lv-partition big enough to hold 
your backup, but not much more.

Work out the syntax for an in-place rsync backup (sorry I haven't done 
it, I can't help.

Every time you make a backup, snapshot the lv before you do it.

That way, the inplace rsync will only actually write data that has 
changed. Your backup volume will grow at an incremental rate, but you'll 
actually have full backups.

The only downside is if the backup gets damaged, it will corrupt every 
copy of the files affected at one stroke, bit if you are using said 
second drive, you can repurpose your first drive if you can back up 
those tar files to dvd or whatever (or throw them away if they've served 
their purpose, but I guess they haven't ...). And by alternating the 
backup drives, you've got two distinct copies.

Cheers,
Wol

RE: [gentoo-user] How to compress lots of tarballs

[Laurence Perkins]

Regular xzutils now does multiple threads with the -T option.

> -----Original Message-----
> From: Ramon Fischer <Ramon_Fischer@hotmail.de> 
> Sent: Sunday, September 26, 2021 5:23 AM
> To: gentoo-user@lists.gentoo.org
> Subject: Re: [gentoo-user] How to compress lots of tarballs
> 
> In addition to this, you may want to use the parallel implementations of "gzip", "xz", "bzip2" or the new "zstd" (zstandard), which are "pigz"[1], "pixz"[2], "pbzip2"[3], or "zstmt" (within package "app-arch/zstd")[4] in order to increase performance:
> 
>     $ cd <path_to_mounted_backup_partition>
>     $ for tar_archive in *.tar; do pixz "${tar_archive}"; done
> 
> -Ramon
> 
> [1]
> * https://www.zlib.net/pigz/
> 
> [2]
> * https://github.com/vasi/pixz
> 
> [3]
> * https://launchpad.net/pbzip2
> * http://compression.ca/pbzip2/
> 
> [4]
> * https://facebook.github.io/zstd/
> 
> 
> On 26/09/2021 13:36, Simon Thelen wrote:
> > [2021-09-26 11:57] Peter Humphrey <peter@prh.myzen.co.uk>
> >> part       text/plain                 382
> >> Hello list,
> > Hi,
> >
> >> I have an external USB-3 drive with various system backups. There are 
> >> 350 .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I 
> >> wouldn't need to compress them, so I didn't, but now I think I'm 
> >> going to have to. Is there a reasonably efficient way to do this? I 
> >> have 500GB spare space on /dev/sda, and the machine runs constantly.
> > Pick your favorite of gzip, bzip2, xz or lzip (I recommend lzip) and
> > then:
> > mount USB-3 /mnt; cd /mnt; lzip *
> >
> > The archiver you chose will compress the file and add the appropriate 
> > extension all on its own and tar will use that (and the file magic) to 
> > find the appropriate decompresser when you want to extract files later 
> > (you can use `tar tf' to test if you want).
> >
> > --
> > Simon Thelen
> >
> 
> --
> GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
> 

Re: [gentoo-user] How to compress lots of tarballs

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey <peter@prh.myzen.co.uk>
wrote:

> Hello list,
>
> I have an external USB-3 drive with various system backups. There are 350
> .tar
> files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't need
> to
> compress them, so I didn't, but now I think I'm going to have to. Is there
> a
> reasonably efficient way to do this?
>

find <mountpoint> -name \*tar -exec zstd -TN {} \;

Where N is the number of cores you want to allocate. zstd -T0 (or just
zstdmt) if you want to use all the available cores. I use zstd for
everything now as it's as good as or better than all the others in the
general case.

Parallel means it uses more than one core, so on a modern machine it is
much faster.

[-- Attachment #2: Type: text/html, Size: 1137 bytes --]

Re: [gentoo-user] How to compress lots of tarballs

[Peter Humphrey]

On Monday, 27 September 2021 02:39:19 BST Adam Carter wrote:
> On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey <peter@prh.myzen.co.uk>
> 
> wrote:
> > Hello list,
> > 
> > I have an external USB-3 drive with various system backups. There are 350
> > .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't
> > need to compress them, so I didn't, but now I think I'm going to have to.
> > Is there a reasonably efficient way to do this?
> 
> find <mountpoint> -name \*tar -exec zstd -TN {} \;
> 
> Where N is the number of cores you want to allocate. zstd -T0 (or just
> zstdmt) if you want to use all the available cores. I use zstd for
> everything now as it's as good as or better than all the others in the
> general case.
> 
> Parallel means it uses more than one core, so on a modern machine it is
> much faster.

Thanks to all who've helped. I can't avoid feeling, though, that the main 
bottleneck has been missed: that I have to read and write on a USB-3 drive. 
It's just taken 23 minutes to copy the current system backup from USB-3 to 
SATA SSD: 108GB in 8 .tar files.

Perhaps I have things out of proportion.

-- 
Regards,
Peter.

Re: [gentoo-user] How to compress lots of tarballs

[Peter Humphrey]

On Monday, 27 September 2021 14:30:36 BST Peter Humphrey wrote:
> On Monday, 27 September 2021 02:39:19 BST Adam Carter wrote:
> > On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey 
<peter@prh.myzen.co.uk>
> > 
> > wrote:
> > > Hello list,
> > > 
> > > I have an external USB-3 drive with various system backups. There are
> > > 350
> > > .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't
> > > need to compress them, so I didn't, but now I think I'm going to have
> > > to.
> > > Is there a reasonably efficient way to do this?
> > 
> > find <mountpoint> -name \*tar -exec zstd -TN {} \;
> > 
> > Where N is the number of cores you want to allocate. zstd -T0 (or just
> > zstdmt) if you want to use all the available cores. I use zstd for
> > everything now as it's as good as or better than all the others in the
> > general case.
> > 
> > Parallel means it uses more than one core, so on a modern machine it is
> > much faster.
> 
> Thanks to all who've helped. I can't avoid feeling, though, that the main
> bottleneck has been missed: that I have to read and write on a USB-3 drive.
> It's just taken 23 minutes to copy the current system backup from USB-3 to
> SATA SSD: 108GB in 8 .tar files.

I was premature. In contrast to the 23 minutes to copy the files from USB-3 to 
internal SSD, zstd -T0 took 3:22 to compress them onto another internal SSD. I 
watched /bin/top and didn't see more than 250% CPU (this is a 24-CPU box) with 
next-to-nothing else running. The result was 65G of .tar.zst files.

So, at negligible cost in CPU load*, I can achieve a 40% saving in space. Of 
course, I'll have to manage the process myself, and I still have to copy the 
compressed files back to USB-3 - but then I am retired, so what else do I have 
to do? :)

Thanks again, all who've helped.

*  ...so I can continue running my 5 BOINC projects at the same time.

-- 
Regards,
Peter.

RE: [gentoo-user] How to compress lots of tarballs

[Laurence Perkins]

>On Monday, 27 September 2021 14:30:36 BST Peter Humphrey wrote:
>> On Monday, 27 September 2021 02:39:19 BST Adam Carter wrote:
>> > On Sun, Sep 26, 2021 at 8:57 PM Peter Humphrey
><peter@prh.myzen.co.uk>
>> > 
>> > wrote:
>> > > Hello list,
>> > > 
>> > > I have an external USB-3 drive with various system backups. There 
>> > > are
>> > > 350
>> > > .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I 
>> > > wouldn't need to compress them, so I didn't, but now I think I'm 
>> > > going to have to.
>> > > Is there a reasonably efficient way to do this?
>> > 
>> > find <mountpoint> -name \*tar -exec zstd -TN {} \;
>> > 
>> > Where N is the number of cores you want to allocate. zstd -T0 (or 
>> > just
>> > zstdmt) if you want to use all the available cores. I use zstd for 
>> > everything now as it's as good as or better than all the others in 
>> > the general case.
>> > 
>> > Parallel means it uses more than one core, so on a modern machine it 
>> > is much faster.
>> 
>> Thanks to all who've helped. I can't avoid feeling, though, that the 
>> main bottleneck has been missed: that I have to read and write on a USB-3 drive.
>> It's just taken 23 minutes to copy the current system backup from 
>> USB-3 to SATA SSD: 108GB in 8 .tar files.
>
>I was premature. In contrast to the 23 minutes to copy the files from USB-3 to internal SSD, zstd -T0 took 3:22 to compress them onto another internal SSD. I watched /bin/top and didn't see more than 250% CPU (this is a 24-CPU box) with next-to-nothing else running. The result was 65G of .tar.zst files.
>
>So, at negligible cost in CPU load*, I can achieve a 40% saving in space. Of course, I'll have to manage the process myself, and I still have to copy the compressed files back to USB-3 - but then I am retired, so what else do I have to do? :)
>
>Thanks again, all who've helped.
>
>*  ...so I can continue running my 5 BOINC projects at the same time.
>
>--
>Regards,
>Peter.

There are also backup tools which will handle the compression step for you.

app-backup/duplicity uses a similar tar file and index system with periodic full and then incremental chains.  Plus it keeps a condensed list of file hashes from previous runs so it doesn't have to re-read the entire archive to determine what changed the way rsync does.

app-backup/borgbackup is more complex, but is very, very good at deduplicating file data, which saves even more space.  Furthermore, it can store backups for multiple systems and deduplicate between them, so if you have any other machines you can have backups there as well, potentially at negligble space cost if you have a lot of redundancy.

LMP

Re: [gentoo-user] How to compress lots of tarballs

[Peter Humphrey]

On Tuesday, 28 September 2021 18:43:06 BST Laurence Perkins wrote:

> There are also backup tools which will handle the compression step for you.
> 
> app-backup/duplicity uses a similar tar file and index system with periodic
> full and then incremental chains.  Plus it keeps a condensed list of file
> hashes from previous runs so it doesn't have to re-read the entire archive
> to determine what changed the way rsync does.
> 
> app-backup/borgbackup is more complex, but is very, very good at
> deduplicating file data, which saves even more space.  Furthermore, it can
> store backups for multiple systems and deduplicate between them, so if you
> have any other machines you can have backups there as well, potentially at
> negligble space cost if you have a lot of redundancy.

Thanks Laurence. I've looked at borg before, wondering whether I needed a 
more sophisticated tool than just tar, but it looked like too much work for 
little gain. I didn't know about duplicity, but I'm used to my weekly routine 
and it seems reliable, so I'll stick with it pro tem. I've been keeping a 
daily KMail archive since the bad old days, and five weekly backups of the 
whole system, together with 12 monthly backups and, recently an annual 
backup. That last may be overkill, I dare say.

-- 
Regards,
Peter.

Re: [gentoo-user] How to compress lots of tarballs

[Rich Freeman]

On Wed, Sep 29, 2021 at 4:27 AM Peter Humphrey <peter@prh.myzen.co.uk> wrote:
>
> Thanks Laurence. I've looked at borg before, wondering whether I needed a
> more sophisticated tool than just tar, but it looked like too much work for
> little gain. I didn't know about duplicity, but I'm used to my weekly routine
> and it seems reliable, so I'll stick with it pro tem. I've been keeping a
> daily KMail archive since the bad old days, and five weekly backups of the
> whole system, together with 12 monthly backups and, recently an annual
> backup. That last may be overkill, I dare say.

I think Restic might be gaining some ground on duplicity.  I use
duplicity and it is fine, so I haven't had much need to look at
anything else.  Big advantages of duplicity over tar are:

1. It will do all the compression/encryption/etc stuff for you - all
controlled via options.
2. It uses librsync, which means if one byte in the middle of a 10GB
file changes, you end up with a few bytes in your archive and not 10GB
(pre-compression).
3. It has a ton of cloud/remote backends, so it is real easy to store
the data on AWS/Google/whatever.  When operating this way it can keep
local copies of the metadata, and if for some reason those are lost it
can just pull that only down from the cloud to resync without a huge
bill.
4. It can do all the backup rotation logic (fulls, incrementals,
retention, etc).
5. It can prefix files so that on something like AWS you can have the
big data archive files go to glacier (cheap to store, expensive to
restore), and the small metadata stays in a data class that is cheap
to access.
6. By default local metadata is kept unencrypted, and anything on the
cloud is encrypted.  This means that you can just keep a public key in
your keyring for completely unattended backups, without fear of access
to the private key.  Obviously if you need to restore your metadata
from the cloud you'll need the private key for that.

If you like the more tar-like process another tool you might want to
look at is dar.  It basically is a near-drop-in replacement for tar
but it stores indexes at the end of every file, which means that you
can view archive contents/etc or restore individual files without
scanning the whole archive.  tar was really designed for tape where
random access is not possible.

-- 
Rich

Re: [gentoo-user] How to compress lots of tarballs

[Dale]

Rich Freeman wrote:
> On Wed, Sep 29, 2021 at 4:27 AM Peter Humphrey <peter@prh.myzen.co.uk> wrote:
>> Thanks Laurence. I've looked at borg before, wondering whether I needed a
>> more sophisticated tool than just tar, but it looked like too much work for
>> little gain. I didn't know about duplicity, but I'm used to my weekly routine
>> and it seems reliable, so I'll stick with it pro tem. I've been keeping a
>> daily KMail archive since the bad old days, and five weekly backups of the
>> whole system, together with 12 monthly backups and, recently an annual
>> backup. That last may be overkill, I dare say.
> I think Restic might be gaining some ground on duplicity.  I use
> duplicity and it is fine, so I haven't had much need to look at
> anything else.  Big advantages of duplicity over tar are:
>
> 1. It will do all the compression/encryption/etc stuff for you - all
> controlled via options.
> 2. It uses librsync, which means if one byte in the middle of a 10GB
> file changes, you end up with a few bytes in your archive and not 10GB
> (pre-compression).
> 3. It has a ton of cloud/remote backends, so it is real easy to store
> the data on AWS/Google/whatever.  When operating this way it can keep
> local copies of the metadata, and if for some reason those are lost it
> can just pull that only down from the cloud to resync without a huge
> bill.
> 4. It can do all the backup rotation logic (fulls, incrementals,
> retention, etc).
> 5. It can prefix files so that on something like AWS you can have the
> big data archive files go to glacier (cheap to store, expensive to
> restore), and the small metadata stays in a data class that is cheap
> to access.
> 6. By default local metadata is kept unencrypted, and anything on the
> cloud is encrypted.  This means that you can just keep a public key in
> your keyring for completely unattended backups, without fear of access
> to the private key.  Obviously if you need to restore your metadata
> from the cloud you'll need the private key for that.
>
> If you like the more tar-like process another tool you might want to
> look at is dar.  It basically is a near-drop-in replacement for tar
> but it stores indexes at the end of every file, which means that you
> can view archive contents/etc or restore individual files without
> scanning the whole archive.  tar was really designed for tape where
> random access is not possible.
>


Curious question here.  As you may recall, I backup to a external hard
drive.  Would it make sense to use that software for a external hard
drive?  Right now, I'm just doing file updates with rsync and the drive
is encrypted.  Thing is, I'm going to have to split into three drives
soon.  So, compressing may help.  Since it is video files, it may not
help much but I'm not sure about that.  Just curious. 

Dale

:-)  :-) 

RE: [gentoo-user] How to compress lots of tarballs

[Laurence Perkins]

> > On Wed, Sep 29, 2021 at 4:27 AM Peter Humphrey <peter@prh.myzen.co.uk> wrote:
> >> Thanks Laurence. I've looked at borg before, wondering whether I 
> >> needed a more sophisticated tool than just tar, but it looked like 
> >> too much work for little gain. I didn't know about duplicity, but I'm 
> >> used to my weekly routine and it seems reliable, so I'll stick with 
> >> it pro tem. I've been keeping a daily KMail archive since the bad old 
> >> days, and five weekly backups of the whole system, together with 12 
> >> monthly backups and, recently an annual backup. That last may be overkill, I dare say.
> > I think Restic might be gaining some ground on duplicity.  I use 
> > duplicity and it is fine, so I haven't had much need to look at 
> > anything else.  Big advantages of duplicity over tar are:
> >
> > 1. It will do all the compression/encryption/etc stuff for you - all 
> > controlled via options.
> > 2. It uses librsync, which means if one byte in the middle of a 10GB 
> > file changes, you end up with a few bytes in your archive and not 10GB 
> > (pre-compression).
> > 3. It has a ton of cloud/remote backends, so it is real easy to store 
> > the data on AWS/Google/whatever.  When operating this way it can keep 
> > local copies of the metadata, and if for some reason those are lost it 
> > can just pull that only down from the cloud to resync without a huge 
> > bill.
> > 4. It can do all the backup rotation logic (fulls, incrementals, 
> > retention, etc).
> > 5. It can prefix files so that on something like AWS you can have the 
> > big data archive files go to glacier (cheap to store, expensive to 
> > restore), and the small metadata stays in a data class that is cheap 
> > to access.
> > 6. By default local metadata is kept unencrypted, and anything on the 
> > cloud is encrypted.  This means that you can just keep a public key in 
> > your keyring for completely unattended backups, without fear of access 
> > to the private key.  Obviously if you need to restore your metadata 
> > from the cloud you'll need the private key for that.
> >
> > If you like the more tar-like process another tool you might want to 
> > look at is dar.  It basically is a near-drop-in replacement for tar 
> > but it stores indexes at the end of every file, which means that you 
> > can view archive contents/etc or restore individual files without 
> > scanning the whole archive.  tar was really designed for tape where 
> > random access is not possible.
> >
> 
> 
> Curious question here.  As you may recall, I backup to a external hard drive.  Would it make sense to use that software for a external hard drive?  Right now, I'm just doing file updates with rsync and the drive is encrypted.  Thing is, I'm going to have to split into three drives soon.  So, compressing may help.  Since it is video files, it may not help much but I'm not sure about that.  Just curious. 
> 
> Dale
> 
> :-)  :-) 
> 
> 

If I understand correctly you're using rsync+tar and then keeping a set of copies of various ages.

If you lose a single file that you want to restore and have to go hunting for it, with tar you can only list the files in the archive by reading the entire thing into memory and only extract by reading from the beginning until you stumble across the matching filename.  So with large archives to hunt through, that could take...  a while...

dar is compatible with tar (Pretty sure, would have to look again, but I remember that being one of its main selling-points) but adds an index at the end of the file allowing listing of the contents and jumping to particular files without having to read the entire thing.  Won't help with your space shortage, but will make searching and single-file restores much faster.

Duplicity and similar has the indices, and additionally a full+incremental scheme.  So searching is reasonably quick, and restoring likewise doesn't have to grovel over all the data.  It can be slower than tar or dar for restore though because it has to restore first from the full, and then walk through however many incrementals are necessary to get the version you want.  This comes with a substantial space savings though as each set of archive files after the full contains only the pieces which actually changed.  Coupled with compression, that might solve your space issues for a while longer.

Borg and similar break the files into variable-size chunks and store each chunk indexed by its content hash.  So each chunk gets stored exactly once regardless of how many times it may occur in the data set.  Backups then become simply lists of file attributes and what chunks they contain.  This results both in storing only changes between backup runs and in deduplication of commonly-occurring data chunks across the entire backup.  The database-like structure also means that all backups can be searched and restored from in roughly equal amounts of time and that backup sets can be deleted in any order.  Many of them (Borg included) also allow mounting backup sets via FUSE.  The disadvantage is that restore requires a compatible version of the backup tool rather than just a generic utility.

LMP

Re: [gentoo-user] How to compress lots of tarballs

[Dale]

Laurence Perkins wrote:
>>
>> Curious question here.  As you may recall, I backup to a external hard drive.  Would it make sense to use that software for a external hard drive?  Right now, I'm just doing file updates with rsync and the drive is encrypted.  Thing is, I'm going to have to split into three drives soon.  So, compressing may help.  Since it is video files, it may not help much but I'm not sure about that.  Just curious. 
>>
>> Dale
>>
>> :-)  :-) 
>>
>>
> If I understand correctly you're using rsync+tar and then keeping a set of copies of various ages.

Actually, it is uncompressed and just stores one version and one copy. 


>
> If you lose a single file that you want to restore and have to go hunting for it, with tar you can only list the files in the archive by reading the entire thing into memory and only extract by reading from the beginning until you stumble across the matching filename.  So with large archives to hunt through, that could take...  a while...
>
> dar is compatible with tar (Pretty sure, would have to look again, but I remember that being one of its main selling-points) but adds an index at the end of the file allowing listing of the contents and jumping to particular files without having to read the entire thing.  Won't help with your space shortage, but will make searching and single-file restores much faster.
>
> Duplicity and similar has the indices, and additionally a full+incremental scheme.  So searching is reasonably quick, and restoring likewise doesn't have to grovel over all the data.  It can be slower than tar or dar for restore though because it has to restore first from the full, and then walk through however many incrementals are necessary to get the version you want.  This comes with a substantial space savings though as each set of archive files after the full contains only the pieces which actually changed.  Coupled with compression, that might solve your space issues for a while longer.
>
> Borg and similar break the files into variable-size chunks and store each chunk indexed by its content hash.  So each chunk gets stored exactly once regardless of how many times it may occur in the data set.  Backups then become simply lists of file attributes and what chunks they contain.  This results both in storing only changes between backup runs and in deduplication of commonly-occurring data chunks across the entire backup.  The database-like structure also means that all backups can be searched and restored from in roughly equal amounts of time and that backup sets can be deleted in any order.  Many of them (Borg included) also allow mounting backup sets via FUSE.  The disadvantage is that restore requires a compatible version of the backup tool rather than just a generic utility.
>
> LMP


I guess that is the downside of not having just plain uncompressed
files.  Thing is, so far, I've never needed to restore a single file or
even several files.  So it's not a big deal for me.  If I accidentally
delete something tho, that could be a problem, if it has left the trash
already. 

Since the drive also uses LVM, someone mentioned using snapshots.  Still
not real clear on those even tho I've read a bit about them.  Some of
the backup technics are confusing to me.  I get plain files, even
incremental to a extent but some of the new stuff just muddies the water. 

I really need to just build a file server, RAID or something.  :/

Dale

:-)  :-)

Re: [gentoo-user] How to compress lots of tarballs

[Wols Lists]

On 29/09/2021 21:58, Dale wrote:
> Since the drive also uses LVM, someone mentioned using snapshots. 

Me?

> Still
> not real clear on those even tho I've read a bit about them.  Some of
> the backup technics are confusing to me.  I get plain files, even
> incremental to a extent but some of the new stuff just muddies the water.

An LVM snapshot creates a "copy on write" image. I'm just beginning to 
dig into it myself, but I agree it's a bit confusing.
> 
> I really need to just build a file server, RAID or something.  :/

Once you've got your logical volume, what I think you do is ask yourself 
how much has changed since the last backup. Then create a snapshot with 
enough space to hold that, before you do an "in place" rsync.

That updates only the stuff that has changed, moving the original data 
into the backup snapshot you've just made.

If you need to restore, you can just mount the backup, and copy stuff 
out of it.

That way, you've now got two full backups, but the old backup is 
actually just storing a diff from the new one - a bit like git actually 
only stores the latest and diffs, recreating older versions on demand.

 From what I gather, you can also revert, which just merges the snapshot 
back in deleting all the new stuff.

The one worry, as far as I can tell, is that if a snapshot overflows it 
just gets dropped and lost, so sizing is crucial. Although if you're 
backing up, you may be able to over-provision, and then shrink it once 
you've done the backup.

For what I want it for - snapshotting before I run an emerge - guessing 
the size is far more important because I don't want the snapshot to be 
dropped half-way through the emerge!

Cheers,
Wol

Re: [gentoo-user] How to compress lots of tarballs

[Rich Freeman]

On Wed, Sep 29, 2021 at 5:48 PM Wols Lists <antlists@youngman.org.uk> wrote:
>
> An LVM snapshot creates a "copy on write" image. I'm just beginning to
> dig into it myself, but I agree it's a bit confusing.

So, snapshots in general are a solution for making backups atomic.
That is, they allow a backup to look as if the entire backup was taken
in an instant.

The simplest way to accomplish that is via offline backups.  Unmount
the drive, mount it read-only, then perform a backup.  That guarantees
that nothing changes between the time the backup starts/stops.  Of
course, it also can mean that for many hours you can't really use the
drive.

Snapshots let you cheat.  They create two views of the drive - one
that can be used normally, and one which is a moment-in-time snapshot
of what the drive looked like.  You backup the snapshot, and you can
use the regular drive.

With something like LVM you probably want to unmount the filesystem
before snapshotting it.  Otherwise it is a bit like just pulling the
plug on the PC before doing an offline backup - the snapshot will be
in a dirty state with things in various state of modification.  This
is because LVM knows nothing about your filesystem - when you run the
snapshot it just captures the state of every block on the disk in that
instant, which means dirty caches/etc (the filesystem caches or file
buffers wouldn't be covered, because those are above the LVM layer,
but caches below LVM would be covered, like the physical disk cache).

Some filesystems like btrfs and zfs also support snapshotting.  Those
are much safer to snapshot while online, since the filesystem is aware
of everything going on.  Of course even this will miss file buffers.
The cleanest way to snapshot anything is to ensure the filesystem is
quiescent.  This need only be for a few seconds while snapshotting
though, and then you can go right back to using it.

If you do a full backup on a snapshot, then if you restore that backup
you'll get the contents of the filesystem at the moment the snapshot
was taken.

You can also use snapshots as a sort-of backup as well.  Of course
those are on the same physical disk so it only protects you against
some types of failures.  Often there is a way to serialize a snapshot,
perhaps incrementally - I know zfs and btrfs both allow this.  This
can be a super-efficient way to create backup files since the
filesystem metadata can be used to determine exactly what blocks to
back up with perfect reliability.  The downside of this is that you
can only restore to the same filesystem - it isn't a simple file-based
solution like with tar/etc.

So as to not do a second reply I want to address an earlier question,
regarding backups that span multiple volumes.  This is a need I have,
and I haven't found a simple tool that does it well.  I am VERY
interested in suggestions here.  Tar I think can support something
like this, since it is common for tape backups to span multiple
volumes.  I have no idea how clean that is.  Most tools want you to
use LVM or a union filesystem or whatever to combine all the drives
into a single mountpoint and let linux manage the spanning.  I'm using
bacula which can sort-of handle multiple volumes, but not very
cleanly, and it is an overly complex tool for backing up one system.
I wouldn't really endorse it wholeheartedly, but it can be made to
work.

-- 
Rich

Re: [gentoo-user] How to compress lots of tarballs

[antlists]

On 30/09/2021 00:17, Rich Freeman wrote:
> On Wed, Sep 29, 2021 at 5:48 PM Wols Lists<antlists@youngman.org.uk>  wrote:
>> An LVM snapshot creates a "copy on write" image. I'm just beginning to
>> dig into it myself, but I agree it's a bit confusing.

> So, snapshots in general are a solution for making backups atomic.
> That is, they allow a backup to look as if the entire backup was taken
> in an instant.
> 
> The simplest way to accomplish that is via offline backups.  Unmount
> the drive, mount it read-only, then perform a backup.  That guarantees
> that nothing changes between the time the backup starts/stops.  Of
> course, it also can mean that for many hours you can't really use the
> drive.
> 
> Snapshots let you cheat.  They create two views of the drive - one
> that can be used normally, and one which is a moment-in-time snapshot
> of what the drive looked like.  You backup the snapshot, and you can
> use the regular drive.

Yup. I'm planning to configure systemd to do most of this for me. As a 
desktop system it goes up and down, so the plan is a trigger will fire 
midnight fri/sat, and the first time it gets booted after that, a 
snapshot will be taken before fstab is run.

Then I'll have backups of /home and /. I won't keep many root backups, 
but I'll keep /home until I run out of space.

And that's why I suggested if you want a separate backup rather than a 
collection of snapshots, you snapshot the backup and use in-place rsync. 
Of course that still means you need to quiesce the bit you're copying, 
but you could back it up piecemeal.

Cheers,
Wol

Re: [gentoo-user] How to compress lots of tarballs

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 4508 bytes --]

Am Wed, Sep 29, 2021 at 03:04:41PM -0500 schrieb Dale:

> Curious question here.  As you may recall, I backup to a external hard
> drive.  Would it make sense to use that software for a external hard
> drive?

Since you are using LVM for everything IIRC, it would be a very efficient
way for you to make incremental backups with snapshots. But I have no
knowledge in that area to give you hints.


But I do use Borg. It’s been my primary backup tool for my systems for
almost two years now. Before that I used rsnapshot (i.e. rsync with
versioning through hard links) for my home partion and simple rsync for the
data partition. Rsnapshot is quite slow, because it has to compare at least
the inodes of all files on the source and destination. Borg uses a cache,
which speeds things up drastically.

I have one Borg repo for the root fs, one for ~ and one for the data
partition, and each repo receives the partition from two different hosts,
but which have most of their data mirrored daily with Unison. A tool like
Borg can deduplicate all of that and create snapshots of it. This saves
oogles of space, but also allows me to restore an entire host with a simple
rsync from a mounted Borg repo. (only downside: no hardlink support, AFAIK).

Borg saves its data in 500 MB files, which makes it very SMR-friendly.
Rsnapshot will create little holes in the backup FS over time with the
deletion of old snapshots. And as we all know, this will bring SMR drives
down to a crawl. If you back-up only big video files, then this may not be a
huge problem. But it will be with the ~ partition, with its thousands of
little files. In Borg, little changes do not trickle down to many random
writes. If a data file becomes partially obsolete, it is rewritten into a
new file and the old one deleted as a whole. Thanks to that, I have no worry
using 2.5″ 4 TB drives as main backup drive (as we all know, everything 2.5″
above 1 TB is SMR).

Those big data files also make it very efficient to copy a Borg repo (for
example to mirror the backup drive to another drive for off-site storage),
because it uses a very small number of files itself:

$ borg info data
...
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:               18.09 TB             17.60 TB              1.23 TB

                       Unique chunks         Total chunks
Chunk index:                  722096             10888890

$ find data -type f | wc -l
2498

I have 21 snapshots in that repo, which amount to 18 TB of backed-up data,
deduped down to 1.23 TB, spread over only 2498 files including metadata.

> Right now, I'm just doing file updates with rsync and the drive
> is encrypted.

While Borg has an encryption feature, I chose not to use it and rely on the
underlying LUKS. Because then I can use KDE GUI stuff to mount the drive and
run my Borg wrapper script without ever having to enter a passphrase.

> Thing is, I'm going to have to split into three drives soon.  So,
> compressing may help. Since it is video files, it may not help much but
> I'm not sure about that.

Of my PC’s data partition, almost 50 % is music, 20 % is my JPEG pictures
library, 15 % is video files and the rest is misc stuff like Kerbal Space
Program, compressed archives of OpenStreetMap files and VM images.
This is the statistics of my last snapshot:

Original size   Compressed size   Deduplicated size
    730.80 GB         698.76 GB             1.95 MB

Compression gain is around 4 %. Much of which probably comes from empty
areas in VM images and 4 GB of pdf and html files. On my laptop, whose data
partition has fewer VM stuff, but a lot more videos, it looks thus:

Original size   Compressed size   Deduplicated size
      1.01 TB           1.00 TB             1.67 MB

So only around 1 % of savings. However, compression is done using lz4 (by
default, you can choose other algos), which is extremely fast but not very
strong. In fact, Borg tries to compress all chunks, but if it finds that
compressing a chunk doesn’t yield enough benefit, it actually discards it
and uses the uncompressed data to save on CPU load later on.

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Some people are so tired, they can’t even stay awake until falling asleep.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] How to compress lots of tarballs

[Rich Freeman]

On Mon, Sep 27, 2021 at 9:30 AM Peter Humphrey <peter@prh.myzen.co.uk> wrote:
>
> Thanks to all who've helped. I can't avoid feeling, though, that the main
> bottleneck has been missed: that I have to read and write on a USB-3 drive.
> It's just taken 23 minutes to copy the current system backup from USB-3 to
> SATA SSD: 108GB in 8 .tar files.

You keep mentioning USB3, but I think the main factor here is that the
external drive is probably a spinning hard drive (I don't think you
explicitly mentioned this but it seems likely esp with the volume of
data).  That math works out to 78MB/s.  Hard drive transfer speeds
depend on the drive itself and especially whether there is more than
one IO task to be performed, so I can't be entirely sure, but I'm
guessing that the USB3 interface itself is having almost no adverse
impact on the transfer rate.

The main thing to avoid is doing other sustained read/writes from the
drive at the same time.

It looks like you ended up doing the bulk of the compression on an
SSD, and obviously those don't care nearly as much about IOPS.

I've been playing around with lizardfs for bulk storage and found that
USB3 hard drives actually work very well, as long as you're mindful
about what physical ports are on what USB hosts and so on.  A USB3
host can basically handle two hard drives with no loss of performance.
I'm not dealing with a ton of IO though so I can probably stack more
drives with pretty minimal impact unless there is a rebuild (in which
case the gigabit ethernet is probably still the larger bottleneck).
Even a Raspberry Pi 4 has two USB3 hosts, which means you could stack
4 hard drives on one and get basically the same performance as SATA.
When you couple that with the tendency of manufacturers to charge less
for USB3 drives than SATA drives of the same performance it just
becomes a much simpler solution than messing with HBAs and so on and
limiting yourself to hardware that can actually work with an HBA.

-- 
Rich

Re: [gentoo-user] How to compress lots of tarballs

[Peter Humphrey]

On Tuesday, 28 September 2021 12:38:42 BST Rich Freeman wrote:

> You keep mentioning USB3, but I think the main factor here is that the
> external drive is probably a spinning hard drive (I don't think you
> explicitly mentioned this but it seems likely esp with the volume of
> data).  That math works out to 78MB/s.  Hard drive transfer speeds
> depend on the drive itself and especially whether there is more than
> one IO task to be performed, so I can't be entirely sure, but I'm
> guessing that the USB3 interface itself is having almost no adverse
> impact on the transfer rate.

I'm sure you're right Rich, and yes, this is 2.5" drive. I'm seeing about 
110MB/s reading from USB feeding into zstd.

> The main thing to avoid is doing other sustained read/writes from the
> drive at the same time.

Quite so.

> It looks like you ended up doing the bulk of the compression on an
> SSD, and obviously those don't care nearly as much about IOPS.

Yes, input from USB and output to SSD.

> I've been playing around with lizardfs for bulk storage and found that
> USB3 hard drives actually work very well, as long as you're mindful
> about what physical ports are on what USB hosts and so on.  A USB3
> host can basically handle two hard drives with no loss of performance.
> I'm not dealing with a ton of IO though so I can probably stack more
> drives with pretty minimal impact unless there is a rebuild (in which
> case the gigabit ethernet is probably still the larger bottleneck).
> Even a Raspberry Pi 4 has two USB3 hosts, which means you could stack
> 4 hard drives on one and get basically the same performance as SATA.
> When you couple that with the tendency of manufacturers to charge less
> for USB3 drives than SATA drives of the same performance it just
> becomes a much simpler solution than messing with HBAs and so on and
> limiting yourself to hardware that can actually work with an HBA.

-- 
Regards,
Peter.

Re: [gentoo-user] How to compress lots of tarballs

[Peter Humphrey]

On Sunday, 26 September 2021 11:57:43 BST Peter Humphrey wrote:
> Hello list,
> 
> I have an external USB-3 drive with various system backups. There are 350
> .tar files (not .tar.gz etc.), amounting to 2.5TB. I was sure I wouldn't
> need to compress them, so I didn't, but now I think I'm going to have to.
> Is there a reasonably efficient way to do this? I have 500GB spare space on
> /dev/sda, and the machine runs constantly.

To complete the topic, and in case anyone's interested, I've settled on the 
following method, which could have been made more general, but this way I 
have a complete set of compressed tarballs in two places in case I need to 
recover something.

This is the command to compress one week's backups:

# (cd /mnt/sdc/wstn/main/vvv.old && time (for tar_archive in *.tar; do zstd -
T0 --rm "${tar_archive}" -o /mnt/bu-space/vvv.old/"${tar_archive}.zst"; done 
&& cp /mnt/bu-space/vvv.old/* /mnt/sdc/wstn/main/vvv.old/ && sync))

This was 36GB and took 20 minutes.

-- 
Regards,
Peter.