[gentoo-user] NAS suggestions for home user

[gentoo-user] NAS suggestions for home user

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]

I'm in the study phase on some sort of NAS backup system for my home. I'll
be building (or buying) a new desktop/server machine in the next few months
- my i980 machine doesn't have the right instruction set for running
Tensorflow anymore - so I want to figure out backups before I put together
the new machine.

I have about $400 credit at NewEgg and would like to keep my additional
costs on the NAS down to about $100-$200. I expect the new machine to
probably be 4TB RAID but it would be quite a while before that gets filled
up.

What do I need to be thinking about? Do I need 8TB in the NAS box? Are 2 or
4 bay NAS boxes generally RAID? I do backups today about once a week. I do
not currently keep any snapshots. I just back up files so over time the
backup carries a lot of stuff that I don't need anymore and I have to go
clean it up if I run out of space.

The NAS would be turned off most of the time. If I need to use it I'll just
power it up.

I've been looking at a few software solutions based on another thread here
but so far nothing has excited me so recommendations for what makes sense
for high reliability home backup is of great interest, especially if it
helps me somehow in cleaning up the backups after deleting stuff on my main
machine on purpose and therefore not needing it on the backup.

Thanks in advance for any ideas.

Cheers,
Mark

[-- Attachment #2: Type: text/html, Size: 1565 bytes --]

Re: [gentoo-user] NAS suggestions for home user

[Rich Freeman]

On Thu, Sep 30, 2021 at 5:50 PM Mark Knecht <markknecht@gmail.com> wrote:
>
> I've been looking at a few software solutions based on another thread here but so far nothing has excited me so recommendations for what makes sense for high reliability home backup is of great interest, especially if it helps me somehow in cleaning up the backups after deleting stuff on my main machine on purpose and therefore not needing it on the backup.

It is probably going to stretch your budget, but you should look into
distributed filesystems like CephFS/MooseFS/LizardFS.  The latter two
at least should run fine on hardware like a Pi4 if you aren't doing
too much IOPS.  You could actually run them on as little as a single
host, which would probably be cheaper than a commercial NAS though
really no better.  The big advantages is that you aren't limited by
the drive capacity of a single host, and you have redundancy at the
host level.  That is, you can pull a plug on any host and the whole
thing just keeps running.

Again, I realize this isn't exactly what you asked for.  IMO this is
the long-term direction storage is trending towards though.  I can't
vouch for the hardware requirements for Ceph, but that can scale
incredibly well and is pretty-much the future.  I've heard it isn't so
great on just a few hosts though.

-- 
Rich

Re: [gentoo-user] NAS suggestions for home user

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 2267 bytes --]

On Thu, Sep 30, 2021 at 3:35 PM Rich Freeman <rich0@gentoo.org> wrote:
>
> On Thu, Sep 30, 2021 at 5:50 PM Mark Knecht <markknecht@gmail.com> wrote:
> >
> > I've been looking at a few software solutions based on another thread
here but so far nothing has excited me so recommendations for what makes
sense for high reliability home backup is of great interest, especially if
it helps me somehow in cleaning up the backups after deleting stuff on my
main machine on purpose and therefore not needing it on the backup.
>
> It is probably going to stretch your budget, but you should look into
> distributed filesystems like CephFS/MooseFS/LizardFS.  The latter two
> at least should run fine on hardware like a Pi4 if you aren't doing
> too much IOPS.  You could actually run them on as little as a single
> host, which would probably be cheaper than a commercial NAS though
> really no better.  The big advantages is that you aren't limited by
> the drive capacity of a single host, and you have redundancy at the
> host level.  That is, you can pull a plug on any host and the whole
> thing just keeps running.
>
> Again, I realize this isn't exactly what you asked for.  IMO this is
> the long-term direction storage is trending towards though.  I can't
> vouch for the hardware requirements for Ceph, but that can scale
> incredibly well and is pretty-much the future.  I've heard it isn't so
> great on just a few hosts though.
>
> --
> Rich

Rich,
   The idea of using a Raspberry Pi hadn't entered my mind. I took
a very quick look at CepfFS and immediately felt a bit swamped
but I will look into all the file systems and get back to you if that's ok.

   If the new machine just needs a standard backup plan then IOPS
should be pretty low. If the backup system is low power then I could
just run it Friday afternoons and incremental backups wouldn't be
a problem at all.

   Physically I am sort of looking for a stand along chassis, or that's
the picture in my head anyway. If I can find a Pi4 in a chassis with
power supply and 1-2 drive bays at the right cost that could possibly
make sense for home. The other issue going that way is how much
time do I spend managing the OS on this backup box? Best answer
is zero.

   Thanks for the ideas.

Cheers,
Mark

[-- Attachment #2: Type: text/html, Size: 2838 bytes --]

Re: [gentoo-user] NAS suggestions for home user

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]

On Fri, Oct 1, 2021 at 4:00 AM Wols Lists <antlists@youngman.org.uk> wrote:
>
> On 30/09/21 22:50, Mark Knecht wrote:
> > I'm in the study phase on some sort of NAS backup system for my home.
> > I'll be building (or buying) a new desktop/server machine in the next
> > few months - my i980 machine doesn't have the right instruction set for
> > running Tensorflow anymore - so I want to figure out backups before I
> > put together the new machine.
>
> Okay, your NAS box is going to be shut down most of the time? Why not
> re-purpose your old box as the NAS? If it was running a lot, I'd be
> concerned about the power consumption,but here it sounds okay.

The number 1 reason would be I want the backup system running
_before_ I decommission this old box. I'd like to spend some time
using the old box, running backups, learning the software, etc, before
the new machine comes online.

Still it's a possibility. I could live on my old laptop for a few months,
take the current server machine, decommission, rebuild it, bring up
the NAS/backup solution/whatever it is. It's possible.

This old machine is now about 10 years old. It's a big Cooler Master
case, 6 or 8 removable drive bays, heavy. It collects dust and
sometimes the fans are quite noisy. If I was going this direction
I think I'd have to tear the whole thing down, redo the case fans at
least. If I did all that then I think I'd use it for the new machine,
but you do have a point.


> >
> > I have about $400 credit at NewEgg and would like to keep my additional
> > costs on the NAS down to about $100-$200. I expect the new machine to
> > probably be 4TB RAID but it would be quite a while before that gets
> > filled up.
>
> If that includes buying new drives, you're pushing your luck here ... a
> new "suitable for raid" 4TB drive will probably blow that budget in one
hit!

Currently my thinking about hardware was pretty modest:

https://www.newegg.com/synology-ds220/p/N82E16822108743

and a couple of 4TB drives at $100+ would be in the $500-$500 range.

I know NOTHING about boxes like that in terms of OS or maintaining
them.

Will respond to the rest of the post when I get back from lunch in
Phoenix. 2 hour drive to meet up with the bass player in one of
my college progressive rock bands. (It's tough being in your
60's...)

Thanks much,
Mark

[-- Attachment #2: Type: text/html, Size: 3076 bytes --]

Re: [gentoo-user] NAS suggestions for home user

[Wols Lists]

On 01/10/2021 17:08, Mark Knecht wrote:
> This old machine is now about 10 years old. It's a big Cooler Master
> case, 6 or 8 removable drive bays, heavy. It collects dust and
> sometimes the fans are quite noisy. If I was going this direction
> I think I'd have to tear the whole thing down, redo the case fans at
> least. If I did all that then I think I'd use it for the new machine,
> but you do have a point.

Okay, get your new disk drive, stick it in your old server, put btrfs on 
it, learn to play with the backups etc. You can hoover out the inside at 
the same time, and possibly replace the fans - they might be noisy 
because the bearings are shot.

There's no reason why your backup drive has to be in a different machine 
(other than the physical safety of it being separate), so play with it 
as part of your current machine. Learn btrfs, learn rsync, learn all 
that stuff.

(Your case sounds a bit like the N300 I've just bought. I want to put a 
whole load of 1TB drives in it as a raid testbed - you might have 
noticed my name on the raid wiki :-)

The other thing, if you are interested and happy with just one disk not 
raid, look at getting one of these HOST MANAGED shingled drives, and use 
a log-structured file system. Again, I don't know anything about these 
other than what they are, but for backups it should be a good and 
reasonably cheap solution.

If you want to go down the pi route, I think you can get little cases, 
and I've got a USB thingy into which you can plug two drives. But at 
about £30-40 each, that's $100 for hardware over and above your drive. 
I'd recycle the old machine :-)

Cheers,
Wol

Re: [gentoo-user] NAS suggestions for home user

[Wols Lists]

On 30/09/21 22:50, Mark Knecht wrote:
> I'm in the study phase on some sort of NAS backup system for my home.
> I'll be building (or buying) a new desktop/server machine in the next
> few months - my i980 machine doesn't have the right instruction set for
> running Tensorflow anymore - so I want to figure out backups before I
> put together the new machine.

Okay, your NAS box is going to be shut down most of the time? Why not
re-purpose your old box as the NAS? If it was running a lot, I'd be
concerned about the power consumption,but here it sounds okay.
> 
> I have about $400 credit at NewEgg and would like to keep my additional
> costs on the NAS down to about $100-$200. I expect the new machine to
> probably be 4TB RAID but it would be quite a while before that gets
> filled up.

If that includes buying new drives, you're pushing your luck here ... a
new "suitable for raid" 4TB drive will probably blow that budget in one hit!
> 
> What do I need to be thinking about? Do I need 8TB in the NAS box?

Ummm... I'd be inclined to buy a single 8TB drive. It's not much more
than the 4TB - I'd suggest suggest Seagate Ironwolf. The Tosh N300s
aren't bad afaik, but I've no personal experience. You could get a WD
Red Pro, but do NOT get a plain Red, and given the way they've mucked
people about I personally wouldn't get a WD.

> Are 2 or 4 bay NAS boxes generally RAID? I do backups today about once a week.
> I do not currently keep any snapshots.

2-bay will be mirrored, 4-bay is normally raid-5

> I just back up files so over time
> the backup carries a lot of stuff that I don't need anymore and I have
> to go clean it up if I run out of space.

Get that 8TB Ironwolf/N300, format it btrfs (it pains me to say that
:-), and set up your backup to do an "in place" rsync then snapshot the
volume.

That way, each backup will be an incremental, but the snapshot will give
you a full backup. Think of it like git, it will store the current
state, with diffs so you can checkout any previous state if you want.

Just MAKE SURE the drive never actually fills up - btrfs has an
appalling rep for surviving a disk full with snapshots.

If you need more space you can then think about getting a second (and
third, and fourth) 8TB drive and going to raid or just adding them to
the btrfs. Again, just remember that, at the moment, btrfs parity raid
also has a pretty appalling rap. Mirroring is fine.
> 
> The NAS would be turned off most of the time. If I need to use it I'll
> just power it up.
> 
> I've been looking at a few software solutions based on another thread
> here but so far nothing has excited me so recommendations for what makes
> sense for high reliability home backup is of great interest, especially
> if it helps me somehow in cleaning up the backups after deleting stuff
> on my main machine on purpose and therefore not needing it on the backup.
> 
> Thanks in advance for any ideas.

Take a read of the raid wiki -
https://raid.wiki.kernel.org/index.php/Linux_Raid

In particular the overview section. My current (new) setup is ext4, over
lvm, over mirror raid, over dm-integrity, over two IronWolves. I'm
planning to add a Barracuda and go raid-5, but not without a solid
backup first !!!

Make sure you read the warnings, and the timeout mismatch section! It
talks about Barracudas, and WD Reds!

On a different topic, I don't know anything about it but some file
systems do block level de-duplication - zfs I believe for one. That way,
you can create your backups in dated directories, and the filesystem
will store each duplicated file only once, without you having to worry
apart from maybe having to trigger a manual de-duplicate once in a
while. But with 8TB that won't be often.

Cheers,
Wol

Re: [gentoo-user] NAS suggestions for home user

[Mark Knecht]

On Fri, Oct 1, 2021 at 9:39 AM Wols Lists <antlists@youngman.org.uk> wrote:
>
> On 01/10/2021 17:08, Mark Knecht wrote:
> > This old machine is now about 10 years old. It's a big Cooler Master
> > case, 6 or 8 removable drive bays, heavy. It collects dust and
> > sometimes the fans are quite noisy. If I was going this direction
> > I think I'd have to tear the whole thing down, redo the case fans at
> > least. If I did all that then I think I'd use it for the new machine,
> > but you do have a point.
>
> Okay, get your new disk drive, stick it in your old server, put btrfs on
> it, learn to play with the backups etc. You can hoover out the inside at
> the same time, and possibly replace the fans - they might be noisy
> because the bearings are shot.
>
> There's no reason why your backup drive has to be in a different machine
> (other than the physical safety of it being separate), so play with it
> as part of your current machine. Learn btrfs, learn rsync, learn all
> that stuff.
>
> (Your case sounds a bit like the N300 I've just bought. I want to put a
> whole load of 1TB drives in it as a raid testbed - you might have
> noticed my name on the raid wiki :-)
>
> The other thing, if you are interested and happy with just one disk not
> raid, look at getting one of these HOST MANAGED shingled drives, and use
> a log-structured file system. Again, I don't know anything about these
> other than what they are, but for backups it should be a good and
> reasonably cheap solution.
>
> If you want to go down the pi route, I think you can get little cases,
> and I've got a USB thingy into which you can plug two drives. But at
> about £30-40 each, that's $100 for hardware over and above your drive.
> I'd recycle the old machine :-)
>
> Cheers,
> Wol
>

So here I am reporting back after a couple of months of not working on
this task.

I dug around in the garage and found an old i5 Clarksdale machine that
literally hadn't been turned on since we sold a house back in about
2013. Unboxed it, cleaned it up a bit, took out all the old hard
drives, the CD and the floppy and put in 2 500GB WD Enterprise drives
I had sitting here from a previous upgrade. Darned if the machine
didn't boot right up from a FreeNAS (now TrueNAS Core) flash drive. I
installed the OS to a second USB thumb drive, booted the machine,
created a 500GB ZFS mirrored pool, created a user directory and 3
hours later I'm doing backups. So far it's done 30GB of about 450GB
and just seems to be humming along nicely. CPU usage is only about 5%
most of the time. The processor is only 2 cores, 4 threads, but most
of the time it's only using about 5% CPU.

No idea how stable it will be but the computer itself was always a
good machine 10 years ago so I'll keep my fingers crossed and see how
it goes.

I'll be adding a SSD front end cache to the storage pool later this
week and will likely move the OS to something internal (SSD or maybe
an old HDD) as I don't like the idea of depending on a USB boot.

Cheers,
Mark

Anyone looking for some similar solution so far I really couldn't be
happier with how easy it was to get this up and running.