From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 1B46215815E for ; Thu, 8 Feb 2024 05:27:20 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 60CBDE2A43; Thu, 8 Feb 2024 05:27:13 +0000 (UTC) Received: from m4.out1.mxs.au (m4.out1.mxs.au [110.232.143.182]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id A0CF8E29D3 for ; Thu, 8 Feb 2024 05:27:12 +0000 (UTC) Received: from fe.s4.ax.email (fe.s4.ax.email [110.232.143.220]) by out1.mxs.au (Halon) with ESMTPS (TLSv1.3) tls TLS_AES_256_GCM_SHA384 id b060e4df-c642-11ee-a33c-00163c39b365 for ; Thu, 08 Feb 2024 16:27:02 +1100 (AEDT) Authentication-Results: fe.s4.ax.email; dkim=pass header.d=kenworthy.id.au header.b="VQCGzITx" Received: from be4.s3.ax.email (10.40.10.219) by fe.s4.ax.email (Axigen) with (TLS_AES_256_GCM_SHA384 encrypted) ESMTPSA id 1C6951; Thu, 8 Feb 2024 16:27:02 +1100 DKIM-Signature: v=1; a=rsa-sha256; d=kenworthy.id.au; s=axigen; c=relaxed/relaxed; q=dns/txt; h=from:date:to:cc:message-id:subject; bh=pykED0J0VINJ6+uZeJO17qEAD0D9eMTDn3/8pu/q30A=; b=VQCGzITxhR8pDpPGpXl81Dr/C9NfGLapIlITcCclP3aF+nRJezEK6xnSksigo32l8x4ch8xcgHWs+VunPm9sTNysBfL5XWOTIYkDu1sZHQKocTIRn62KELWkbJdfrCsbO6ZiFzFcnQkTrroouKaXtzaEpj78xbhvwetH8hVu11z3prTepntSVCvGV8+OSZH0Oocni6XoWhNNovtUuw5a0qH/PFFdUZiPiVPOjGe8Yl3iJMSFmIT+vZkv+uMIAeDM/QGapAKZ3w+NON16revVAu4U6vmMXFLhHDUX0ks9z5/2yykR5d/F3z8YhuUEQbScmjFFZJAQSF+7aWOKUHQuTw==; Authentication-Results: be4.s3.ax.email; dkim=none Authentication-Results: be4.s3.ax.email; spf=pass smtp.mailfrom=bill@kenworthy.id.au Received: from fe.s4.ax.email (10.40.10.220) by be4.s3.ax.email (Axigen) with (TLS_AES_256_GCM_SHA384 encrypted) ESMTPSA id 3C0386; Thu, 8 Feb 2024 16:27:02 +1100 Authentication-Results: fe.s4.ax.email; dkim=none Received: from mail.infra.localdomain (10.40.10.201) by fe.s4.ax.email (Axigen) with (ECDHE-RSA-AES128-GCM-SHA256 encrypted) ESMTPSA id 17B755; Thu, 8 Feb 2024 16:27:02 +1100 Received: from localhost (mail.infra.localdomain [127.0.0.1]) by mail.infra.localdomain (Postfix) with ESMTP id 57BA518B6E81CC for ; Thu, 8 Feb 2024 13:27:01 +0800 (AWST) X-Virus-Scanned: amavisd-new at localdomain Received: from mail.infra.localdomain ([127.0.0.1]) by localhost (mail.infra.localdomain [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BRVLKk0zIa77 for ; Thu, 8 Feb 2024 13:26:55 +0800 (AWST) Message-ID: <1d98122f-9edc-42da-bff3-8962db489de8@kenworthy.id.au> Date: Thu, 8 Feb 2024 13:26:50 +0800 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [gentoo-user] Suggestions for backup scheme? Content-Language: en-AU, en-US To: gentoo-user@lists.gentoo.org References: From: William Kenworthy Organization: me@home In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit DomainKey-Status: no signature DKIM-Status: good Received-SPF: X-AXIGEN-DK-Result: No records X-AXIGEN-DKIM-Result: Ok X-Archives-Salt: a94a4220-0f1e-4902-bc1f-c2cea1047e97 X-Archives-Hash: 95178204012787895636a73ba0c3fcf3 On 8/2/24 06:36, Frank Steinmetzger wrote: > Am Tue, Jan 30, 2024 at 06:15:09PM -0000 schrieb Grant Edwards: >> I need to set up some sort of automated backup on a couple Gentoo >> machines (typical desktop software development and home use). One of >> them used rsnapshot in the past but the crontab entries that drove >> that have vanished :/ (presumably during a reinstall or upgrade -- >> IIRC, it took a fair bit of trial and error to get the crontab entries >> figured out). >> >> I believe rsnapshot ran nightly and kept daily snapshots for a week, >> weekly snapshots for a month, and monthly snapshots for a couple >> years. >> >> Are there other backup solutions that people would like to suggest I >> look at to replace rsnapshot? I was happy enough with rsnapshot (when >> it was running), but perhaps there's something else I should consider? > In my early backup times I, too, used rsnapshot to back up my ~ and rsync > for my big media files. But that only included my PC. My laptop was wholly > un-backed-up. I only syncronised much of my home and my audio collection > between the two with unison. At some point my external 3 TB drive became > free and then I started using borg to finally do proper backups. > > Borg is very similar to restic, I actually used the two in parallel for a > while to compare them, but stayed with borg. One pain point was that I > couln’t switch off restic’s own password protection. Since all my backup > disks are LUKSed anyway, I don’t need that. > > Since borg works block-based, it does deduplication without extra cost and > it is suitable for big image files which don’t change much. I do full > filesystem backups of /, ~ and my media partition of my main PC and my > laptop. I have one repository for each of those three filesystems, and each > repo receives the data from both machines, so they are deduped. Since both > machines run Arch, their roots are binary identical. The same goes for my > unison-synced homes. > > Borg has retention logic built-in. You can say I want to keep the latest > archive of each of the last 6 days/weeks/months/years, and it even goes down > to seconds. And of course you can combine those rules. The only thing is > they don’t overlap, meaning if you want to keep the last 14 days and the > last four weeks, those weekly retentions start after the last daily > snapshots. > > In summary, advantages: > + fast dedup, built-in compression (different algos and levels configurable) > + big data files allow for quick mirroring of repositories. > I simply rsync my primary backup disk to two other external HDDs. > + Incremental backups are quite fast because borg uses a cache to detect > changed files quickly. > Disadvantages: > - you need borg to mount the backups it > - it is not as fast as native disk access, especially during restore and > when getting a total file listing due to lots of random I/O on the HDD. > > > As example, I currently have 63 snapshots in my data partition repository: > > # borg list data/ > tp_2021-06-07 Mon, 2021-06-07 16:27:44 [5f9ebd9f24353c340691b2a71f5228985a41699d2e23473ae4e9e795669c8440] > kern_2021-06-07 Mon, 2021-06-07 23:58:56 [19c76211a9c35432e6a66ac1892ee19a08368af28d2d621f509af3d45f203d43] > [... 55 more lines ...] > kern_2024-01-14 Sun, 2024-01-14 20:53:23 [499ce7629e64cffb7ec6ec9ffbf0c595e4ede3d93f131a9a4b424b165647f645] > tp_2024-01-14 Sun, 2024-01-14 20:57:42 [ea2baef3e4bb49c5aec7cf8536f7b00b55fb27ecae3a80ef9f5a5686a1da30d5] > kern_2024-01-21 Sun, 2024-01-21 23:42:46 [71aa2ce6cf4021712f949af068498bfda7797b5d1c5ddc0f0ce8862b89e48961] > tp_2024-01-21 Sun, 2024-01-21 23:48:24 [45e35ed9206078667fa62d0e4a1ac213e77f52415f196101d14ee21e79fc393d] > kern_2024-02-04 Sun, 2024-02-04 23:16:43 [e1b015117143fad6b89cea66329faa888cffc990644e157b1d25846220c62448] > tp_2024-02-04 Sun, 2024-02-04 23:23:15 [e9b167ceec1ab9a80cbdb1acf4ff31cd3935fc23e81674cad1b8694d98547aeb] > > The last “tp” (Thinkpad) snapshot contains 1 TB, “kern” (my PC) 809 GB. > And here you see how much space this actually takes on disk: > > # borg info data/ > [ ... ] > Original size Compressed size Deduplicated size > All archives: 56.16 TB 54.69 TB 1.35 TB > > Obviously, compression doesn’t do much for media files. But it is very > effective in the repository for the root partitions: > > # borg info arch-root/ > [ ... ] > Original size Compressed size Deduplicated size > All archives: 1.38 TB 577.58 GB 79.41 GB > I would also like to add my +1 to borgbackup ... I long ago lost the ability to use snapshots and full size backups due to the sheer amount of data involved.  Currently I use borg to backup multiple hosts to individual backups on a  dedicated machine (low power arm based, 6TB drive).  I also backup from the top level of the directory all those repos are stored in to another arm system (2TB drive) again using borg.  As each 1st level backup only adds/changes a few chunks for each iteration, the second level only takes minutes to run as against 30minutes or so for some of the individual hosts.  The second level adds redundancy if I lose the 1st level backups, and the second can be recreated at any time from the 1st level.  This is working for ~15 hosts and VM's of various types involving hundreds of terabytes of original data. Downside for VM's is that even a slight change to the image requires the whole image to be read and check-summed to identify the changes to be stored.  For images hundreds of gigabytes in size (on my hardware/network) its actually quicker to mount and backup the internal files (camera images in my case) than the VM image. It is more complex than simple schemas, but I regularly restore from both the first and second level backups for disaster recovery/testing/rollbacks etc.  There is a management package (borgmatic) but I have not tried it as use my own scripts. BillK