public inbox for gentoo-user@lists.gentoo.org
 help / color / mirror / Atom feed
From: William Kenworthy <billk@iinet.net.au>
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] [OT] SMR drives (WAS: cryptsetup close and device in use when it is not)
Date: Sun, 1 Aug 2021 11:36:36 +0800	[thread overview]
Message-ID: <faf7bf15-ec9e-ba6f-a840-fa062faccf18@iinet.net.au> (raw)
In-Reply-To: <YQXvvdkbgC8HYeQ6@kern>


On 1/8/21 8:50 am, Frank Steinmetzger wrote:
> Am Sat, Jul 31, 2021 at 12:58:29PM +0800 schrieb William Kenworthy:
>
>> Its not raid, just a btrfs single on disk (no partition).  Contains a
>> single borgbackup repo for an offline backup of all the online
>> borgbackup repo's I have for a 3 times a day backup rota of individual
>> machines/data stores
> So you are borg’ing a repo into a repo? I am planning on simply rsync’ing
> the borg directory from one external HDD to another. Hopefully SMR can cope
> with this adequatly.
>
> And you are storing several machines into a single repo? The docs say this
> is not supported officially. But I have one repo each for /, /home and data
> for both my PC and laptop. Using a wrapper script, I create snapshots that
> are named $HOSTNAME_$DATE in each repo.

Basicly yes: I use a once per hour snapshot of approximately 500Gib of
data on moosefs, plus borgbackups 3 times a day to individual repos on
moosefs for each host.  3 times a day, the latest snapshot is stuffed
into a borg repo on moosefs and the old  snapshots are deleted.  I
currently manually push all the repos into a borg repo on the USB3 SMR
drive once a day or so.

1. rsync (and cp etc.) are dismally slow on SMR - use where you have to,
avoid otherwise.

2. borgbackup with small updates goes very quick

3. borgbackup often to keep changes between updates small - time to
backup will stay short.

4. borg'ing a repo into a repo works extreemly well - however there are
catches based around backup set names and the file change tests used.
(ping me if you want the details)

5. Yes, I have had disasters (i.e., a poorly thought out rm -rf in a
moosefs directory, unstable power that took awhile to cure, ...)
requiring underfire restoration of both large and small datasets - it works!

6. Be careful of snapshot resources on moosefs - moosefs has a defined
amount of memory for each file stored.  Even with the lazy snapshot
method, taking a snapshot will about double the memory usage on the
master for that portion of the filesystem.  Also taking too many
snapshots multiplies the effect.  Once you go into swap, it becomes a
recovery effort.  Also keep in mind that trashtime is carried into the
snapshot so the data may exist in trash even after deletion - its
actually easy to create a DOS condition by not paying attention to this.

BillK


>
>> - I get an insane amount of de-duplication that way for a slight decrease
>> in conveniance!
> And thanks to the cache, a new snapshots usually is done very fast. But for
> a yet unknown reason, sometimes Borg re-hashes all files, even though I
> didn’t touch the cache. In that case it takes 2½ hours to go through my
> video directory.
>


  reply	other threads:[~2021-08-01  3:37 UTC|newest]

Thread overview: 64+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-14  4:50 [gentoo-user] cryptsetup close and device in use when it is not Dale
2021-06-15 13:48 ` Ramon Fischer
2021-06-15 14:21   ` Dale
2021-06-15 14:52     ` Jack
2021-06-15 15:26       ` Dale
2021-06-15 19:04         ` Ramon Fischer
2021-06-21  4:18         ` Dale
2021-06-21  4:49           ` Dale
2021-06-21  5:41             ` Dale
2021-06-21  5:59               ` Dale
2021-06-28  3:35                 ` Dale
2021-07-05  3:19                   ` Dale
2021-07-06 18:40                     ` Ramon Fischer
2021-07-06 19:43                       ` Dale
2021-07-07 14:48                       ` Dr Rainer Woitok
2021-07-07 18:08                         ` Dale
2021-07-08  8:20                           ` Ramon Fischer
2021-07-12  8:31                             ` Dale
2021-07-12 13:14                               ` Ramon Fischer
2021-08-02 13:33                                 ` Dale
2021-08-09 13:38                                   ` Ramon Fischer
2021-09-19 11:55                                     ` Dale
2021-07-25 20:29                           ` Frank Steinmetzger
2021-07-25 23:10                             ` Dale
2021-07-26 21:00                               ` Frank Steinmetzger
2021-07-26 22:48                                 ` Dale
2021-07-29 16:46                                 ` Wols Lists
2021-07-29 20:55                                   ` [gentoo-user] [OT] SMR drives (WAS: cryptsetup close and device in use when it is not) Frank Steinmetzger
2021-07-29 21:31                                     ` Frank Steinmetzger
2021-07-30 12:48                                       ` Frank Steinmetzger
2021-07-30  5:14                                     ` William Kenworthy
2021-07-30 14:29                                       ` Rich Freeman
2021-07-30 16:50                                         ` antlists
2021-07-30 18:38                                           ` Rich Freeman
2021-07-31  3:14                                         ` William Kenworthy
2021-07-31  3:50                                           ` Wols Lists
2021-07-31  4:58                                             ` William Kenworthy
2021-07-31 12:12                                               ` Rich Freeman
2021-08-01  0:41                                                 ` Frank Steinmetzger
2021-08-01  0:56                                                   ` Rich Freeman
2021-07-31 16:38                                               ` antlists
2021-08-01  0:50                                               ` Frank Steinmetzger
2021-08-01  3:36                                                 ` William Kenworthy [this message]
2021-08-01  3:46                                                   ` William Kenworthy
2021-08-01 21:38                                                     ` Frank Steinmetzger
2021-08-02  5:38                                                       ` William Kenworthy
2021-08-02 21:52                                                         ` Frank Steinmetzger
2021-08-02 23:10                                                           ` William Kenworthy
2021-08-03  8:18                                                             ` Frank Steinmetzger
2021-08-05 20:40                                                               ` Frank Steinmetzger
2021-08-06  7:22                                                                 ` William Kenworthy
2021-08-01 21:55                                                   ` Frank Steinmetzger
2021-08-02  6:12                                                     ` William Kenworthy
2021-08-02 22:03                                                       ` Frank Steinmetzger
2021-08-02 23:35                                                         ` William Kenworthy
2021-08-01  3:41                                                 ` William Kenworthy
2021-08-01 21:41                                                   ` Frank Steinmetzger
2021-07-31 12:21                                             ` Rich Freeman
2021-07-31 12:59                                               ` William Kenworthy
2021-07-31 13:30                                                 ` Rich Freeman
2021-08-01  3:05                                                   ` William Kenworthy
2021-08-01 11:37                                                     ` Rich Freeman
2021-07-31  5:23                                           ` William Kenworthy
2021-06-15 17:48     ` [gentoo-user] Re: cryptsetup close and device in use when it is not Remy Blank

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=faf7bf15-ec9e-ba6f-a840-fa062faccf18@iinet.net.au \
    --to=billk@iinet.net.au \
    --cc=gentoo-user@lists.gentoo.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox