From: "J. Roeleveld" <joost@antarean.org>
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] Hard drive storage questions
Date: Fri, 09 Nov 2018 10:02:37 +0100 [thread overview]
Message-ID: <44053795.EIAGmacMho@eve> (raw)
In-Reply-To: <CAGfcS_nJCc3L7v7zupYD_214gUGNev6Gmm170c8MQ0PxApuqVA@mail.gmail.com>
On Friday, November 9, 2018 3:29:52 AM CET Rich Freeman wrote:
> On Thu, Nov 8, 2018 at 8:16 PM Dale <rdalek1967@gmail.com> wrote:
> > I'm trying to come up with a
> > plan that allows me to grow easier and without having to worry about
> > running out of motherboard based ports.
>
> So, this is an issue I've been changing my mind on over the years.
> There are a few common approaches:
>
> * Find ways to cram a lot of drives on one host
> * Use a patchwork of NAS devices or improvised hosts sharing over
> samba/nfs/etc and end up with a mess of mount points.
> * Use a distributed FS
>
> Right now I'm mainly using the first approach, and I'm trying to move
> to the last. The middle option has never appealed to me.
I'm actually in the middle, but have a single large NAS.
> So, to do more of what you're doing in the most efficient way
> possible, I recommend finding used LSI HBA cards. These have mini-SAS
> ports on them, and one of these can be attached to a breakout cable
> that gets you 4 SATA ports. I just picked up two of these for $20
> each on ebay (used) and they have 4 mini-SAS ports each, which is
> capacity for 16 SATA drives per card. Typically these have 4x or
> larger PCIe interfaces, so you'll need a large slot, or one with a
> cutout. You'd have to do the math but I suspect that if the card+MB
> supports PCIe 3.0 you're not losing much if you cram it into a smaller
> slot. If most of the drives are idle most of the time then that also
> demands less bandwidth. 16 fully busy hard drives obviously can put
> out a lot of data if reading sequentially.
I also recommend LSI HBA cards, they work really well and are really well
supported by Linux.
> You can of course get more consumer-oriented SATA cards, but you're
> lucky to get 2-4 SATA ports on a card that runs you $30. The mini-SAS
> HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are
> you main limiting factor assuming you have power and case space.
>
> Oh, and those HBA cards need to be flashed into "IT" mode - they're
> often sold this way, but if they support RAID you want to flash the IT
> firmware that just makes them into a bunch of standalone SATA slots.
> This is usually a PITA that involves DOS or whatever, but I have
> noticed some of the software needed in the Gentoo repo.
Even with Raid-firmware, they can be configured for JBOD.
> If you go that route it is just like having a ton of SATA ports in
> your system - they just show up as sda...sdz and so on (no idea where
> it goes after that).
I tested this once, ended up getting sdaa, sdab,...
> Software-wise you just keep doing what you're
> already doing (though you should be seriously considering
> mdadm/zfs/btrfs/whatever at that point).
I would suggest ZFS or BTRFS over mdadm. Gives you more flexibility and is a
logical follow-up to LVM.
> That is the more traditional route.
>
> Now let me talk about distributed filesystems, which is the more
> scalable approach. I'm getting tired of being limited by SATA ports,
> and cases, and such. I'm also frustrated with some of zfs's
> inflexibility around removing drives.
IMHO, ZFS is nice for large storage devices, not so much for regular desktops.
This is why I am hoping BTRFS will solve the resilver issues. (not kept up, is
this still not working?)
--
Joost
next prev parent reply other threads:[~2018-11-09 9:02 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-09 1:16 [gentoo-user] Hard drive storage questions Dale
2018-11-09 1:31 ` Jack
2018-11-09 1:43 ` Dale
2018-11-09 2:04 ` Andrew Lowe
2018-11-09 2:07 ` Bill Kenworthy
2018-11-09 8:39 ` Neil Bothwick
2018-11-09 2:29 ` Rich Freeman
2018-11-09 8:17 ` Bill Kenworthy
2018-11-09 13:25 ` Rich Freeman
2018-11-09 9:02 ` J. Roeleveld [this message]
2018-11-11 0:45 ` Dale
2018-11-11 21:41 ` Wol's lists
2018-11-11 22:17 ` Dale
2018-11-09 9:24 ` Wols Lists
-- strict thread matches above, loose matches on Subject: below --
2015-04-28 8:39 Dale
2015-04-28 14:49 ` Francisco Ares
2015-04-28 15:01 ` Alan McKinnon
2015-04-28 15:24 ` Neil Bothwick
2015-04-28 17:38 ` Rich Freeman
2015-04-28 18:11 ` Neil Bothwick
2015-04-28 18:31 ` Rich Freeman
2015-04-28 18:41 ` Neil Bothwick
2015-04-29 6:13 ` Alan McKinnon
2015-04-29 7:52 ` Neil Bothwick
2015-05-04 7:39 ` Dale
2015-05-04 7:46 ` Neil Bothwick
2015-05-04 8:13 ` Mick
2015-05-04 8:26 ` Dale
2015-05-04 8:23 ` Dale
2015-05-04 10:31 ` Neil Bothwick
2015-05-04 10:40 ` Dale
2015-05-04 11:26 ` Neil Bothwick
2015-05-09 10:56 ` Dale
2015-05-09 12:59 ` Rich Freeman
2015-05-09 14:46 ` Todd Goodman
2015-05-09 18:16 ` Rich Freeman
2015-05-04 11:35 ` Rich Freeman
2015-05-04 18:42 ` Nuno Magalhães
2015-05-05 6:41 ` Alan McKinnon
2015-05-05 10:56 ` Rich Freeman
2015-05-05 11:33 ` Neil Bothwick
2015-05-05 12:05 ` Mick
2015-05-05 12:21 ` Neil Bothwick
2015-05-05 12:39 ` Mick
2015-05-05 12:53 ` Rich Freeman
2015-05-05 21:50 ` Neil Bothwick
2015-05-05 22:21 ` Bill Kenworthy
2015-05-05 22:33 ` Bill Kenworthy
2015-05-04 10:57 ` Alan Mackenzie
2015-04-28 15:02 ` Rich Freeman
2015-05-04 7:23 ` Dale
2015-05-05 3:01 ` Walter Dnes
2015-04-27 7:41 Dale
2015-04-28 18:25 ` Daniel Frey
2015-04-28 21:23 ` Dale
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=44053795.EIAGmacMho@eve \
--to=joost@antarean.org \
--cc=gentoo-user@lists.gentoo.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox