From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 383B1138334 for ; Sun, 11 Nov 2018 00:45:49 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 2D3E3E0CED; Sun, 11 Nov 2018 00:45:42 +0000 (UTC) Received: from mail-yw1-xc2a.google.com (mail-yw1-xc2a.google.com [IPv6:2607:f8b0:4864:20::c2a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 95B7FE0C4B for ; Sun, 11 Nov 2018 00:45:41 +0000 (UTC) Received: by mail-yw1-xc2a.google.com with SMTP id c126-v6so3089267ywd.8 for ; Sat, 10 Nov 2018 16:45:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:openpgp:autocrypt:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=bW3dnIoUAatMVfHiNWR2erwe7zCa6Midcxjv+gRimPs=; b=lcijB1Qv2UAHAS1QwCtJHuTeG+VjRjfc7ZbC70efoHthsHroKagY2ZZBBmIz57LhUk zdFYX9ur4sXQS1dKDpSD8koxw9sjxpZEKSMldlGULiQlz7rxKfe1GV4d6ciY+mTFlp4L BzIVEq+K1hGy3q7v3TvfmooMVSWBm/KbjlCJAjlmU6X2BRVq/pg3ECtytg6FuOb9asjL og0+MrpN9iTHEwy2sfPPS/eeshlRWkHXczZ7Ntl4dk9GtsKeJnkYhNQJ6tV6Xd1jvX4r QmUDwlNbIhr+hNQZ4G10stuI4H9Pnet2C+1X9t2StzJM6DAAfumXXy+9iYXVwmkuWEI2 ngSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding; bh=bW3dnIoUAatMVfHiNWR2erwe7zCa6Midcxjv+gRimPs=; b=mWx/d/w37wW0htaGh8iLPSWSOdsjQlHrRNjp0vgX7Om+T0Hh7to+jv7ZP6RESOOpcV 1xJOkyQ18IMAEl2yZ/qZd/udrYWbmfYeW4Z/9EMoHEvN3AbUvvganFO5x0oRRzgbyIIe jO9ZyKjWA9/lNZ8znZJqmRUee5RdCAcADaaAQ8M260UrIOjMdDrHAJjibU8g8tkTIZTf 3bTAcNQ+H3AN2Fx8EaGDRWC6yMMB2pg8aSKBOASDT/MsxZyKYpNKES65yIM5ZIvsdm97 +21P6aU92x5spUhcGCMYRR9W8iPs8x+ZjoL9IHRaWdGec16RIC4sTYb8ovII7EYnC6Vr FdnQ== X-Gm-Message-State: AGRZ1gK/1aYxxtrNmz+e0GRCJfVRYNfGDDcpjmMHcgxgJBjpAzZYWJTf 9d1w1+I8d2c2vcFi6jvHjSg= X-Google-Smtp-Source: AJdET5eE64dyDF/5IsWbMuUgzN6qST4a55Mp/IOwzkwryjKkjLQ0gYHt37cpYXfmEqHkyXfYIMq84w== X-Received: by 2002:a0d:c186:: with SMTP id c128-v6mr13545497ywd.12.1541897140476; Sat, 10 Nov 2018 16:45:40 -0800 (PST) Received: from [192.168.2.5] (adsl-68-19-248-97.bna.bellsouth.net. [68.19.248.97]) by smtp.gmail.com with ESMTPSA id i189-v6sm2729945ywb.23.2018.11.10.16.45.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Nov 2018 16:45:39 -0800 (PST) Subject: Re: [gentoo-user] Hard drive storage questions To: gentoo-user@lists.gentoo.org References: <9b5365ed-5cca-54b3-0da7-bbbe697b4c40@gmail.com> From: Dale Openpgp: preference=signencrypt Autocrypt: addr=rdalek1967@gmail.com; prefer-encrypt=mutual; keydata= xsFNBFpEtdQBEADI51WaryP3FJlDfmCQx2aPQpSppEKxqWhCTA8KFEcOVFmIIfiFAeekqMMD mhUxgZTtlQh7dsNqha6ioaYDqGKTv7oeJlPJw4hmIMJX3WYVSOHlsJUNM2jpDIAFeEKfup/T zDzFpuU2Qtr/Y0ji35wHyOAZLRckeNk705oRvE9wqi6noTP15Gxmw/U6aMzEfvu+wGEfCjgs 9bERmu+CS75PZEaFAv8RnsXUv1UcvQ45jmk/8ni/ogxE2h53OIp6c/hOlgJkSVRQWPZZyKZw lDiSUKCtMXPMdZ9w0X6RltQxtIQXO0KxAKaAp+tnL8z+0piafF5uW4RIglhT922RXKxxdZyx SjRgtE4V1IPtUcwPAeqVUZw2P1b4pjfPv7tNtMoFsIiY0ZnT+ua4ps6KOUeocRPKAX14mZkL jt/sZM7aIKiwyoteshRgWNNkxh4OiSxGCRUKNQI8M42cRSidvJZ6SGZXM3WpV28RPyF7+0Ba 0stEQwBGNF8uxgytY9rOJ7obmIpEZKx1p3W1O1hadOjBo2110jMDirRXtktMDfBDvVKkOZ06 vLu16uZLb0O52euhl2dMcEI3ZoCAFTKtdwMITIDj1TcMBZar6+bcwOicSFFogOLHQLJZRO5q I5szOIYW7+c0yNqPRLT3Sq7HzDyuyTUjmPZSAcqOwzX8GwUFkwARAQABzRtEYWxlIDxyZGFs ZWsxOTY3QGdtYWlsLmNvbT7CwZQEEwEIAD4WIQSUDVlCt0m0Z/PsCaxgB5lCagHqugUCWkS1 1AIbIwUJCWYBgAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRBgB5lCagHqurR7D/45/q20 vXdrJGxrkNphotmlBtTpNbVauu5A2NDv3E1Il6yqRBfh4Xw7xFuwhz9DqThuvByU6566vr0z 7oVCK33dxRm4WA7YaogRQZy4VVIbHdrksnh2f702CFllqtn5Y29M2JtXG5jiiL3aZNEhoyP9 eMtzLPGs56yZ3eMkz4U7DEmWCAUr8bbuXW+eq/A0V5djcFdHfmanuDZSxzg+cZTpVOLolS2b pmNsPTSMkJ2MDY2Kfdg3gPhSaawo2agQfgnf9E7vSm7z/rlk8bBUWcPAP/XTN9ndVwOO3x74 EQv/M4EiCTtNpw3yapVZI2NhA1wqW648D7RxIPD8Y3nkJVDS4x5g54xDe1IUFOtVUSDAh+vY wUJt7vgbCeRjyT8XbuGW9RokIos3ALNaPoq/FDNEqefbmop0CPRih6aLFHWT4YBA6xQjLJuP LSNvalNqE5mef0giCtnLxo/lkjnP9Sv+t/5VSHda5zkVuN0+2w46SbGvXIHRkSoSi6XH3ccq KayJC+oTqo4xf9J30c7CV4rEcYnJcnxMw0vcYmU3DwjGfKxuKcLHgPr9mDNWvhteroA5wNWw NzQ72yAj7rsZVUXCxZgiPldSH3SXZJ/Jo6E9JouzQgRb/I4Vy4jx0Yw8rJLDx/ha82fn+FVe cFbiodVV5UD0inw488IAAtJE+Zi0t87BTQRaRLXUARAA38iHcF7M7GnkS73dazdLBgz2YJsu fpix/N/x4CvoHMqTuwi4ASz1WroYjl3KajeH0DSybyPdEQ7nffxIUt48deT3j/rwsJkPRvCF BpmcwxErd/Mbq0BgikYxXvO68aEAs4jBDR26YtONfjobEfd+Juhxci9UN9vTOCgSPhY+dxHs MZ0gHRzvMnpM3o3+oht/XRZr05RQx83DvTIqWnjDQlCseYYlbFp+rFTZi7ro71ULDThfCE0p +f+IQ3zX0cRKOcJGtNRvyWH6PxmN4td6Q7gPHfAsFPLsCpg7nZwOejtAktPejtSEXlN6QOKv bmRQxNtzgMtjzJNNJW2NtBz0DIW394+9stchQRKLqH8n8GnB6tlkfPg4vgf/kq14QQSZcb0F M36wk+i1Hk+TWYWbOBoUw9+X941Pw1JnglJ3tzpBh+36+pdG02Lbm2v6SaZ69zkDfzJ2Sfhb E+KQLibLkiCOhuSDLDWUgUeb0lJ/0qlo3vcQMTBuG5eiWiwBkp4C+ACb1f1Akq0mFvim+gCJ qJOTu0IDK9DjKLKglA3Z6sbeepnXq8fxB2Mo/SFSYEsGqUu4MLxgwnPg7zi+rKg7MhqdiBBE fqugmNguCEYZjJrGCCzwuqPXAZAcyzEYTGFKwI6NdEZ6v8Xc3om9MJomB3y1uzG6K7T9ue5H aw/2aqEAEQEAAcLBfAQYAQgAJhYhBJQNWUK3SbRn8+wJrGAHmUJqAeq6BQJaRLXUAhsMBQkJ ZgGAAAoJEGAHmUJqAeq6/ykP/ib6xEHednaXvzZvvj854PB5ffBqKkphbf51g6pxPvFBWMwY E7Bu/kq8e3hkp3rzX42BjqiUmfEe2OyfZCabXLybP8i/QRkHTzD5nLoIYLeL+62N/WQFW1NU VhqdfQbMhphNgP1mvG2Ib5R6S+Fb+vkw776oq6jLwUBP/o6PPpp62GyvFvFb9ekxV9+sE4yG V3DTqURBY+aXfc/MTzlCXp4u4QzFW9odfcb/kb9f1m/gZbWGihAqeMd1HViXQoMzTx6IuP13 eQAkKj4FlA2QMzbEOOKO6fliSt1JweJoh0OLCEAM/3q+LaflMvvjhl9ht00IUT/ySj3/dZdf EdTpuUAtnC3A3flwgK/aetkkOhrkx9hx4SKn6UHtAl+eCqP1Mae+nWzkisBL0/hBPEz713md 5I+4Y4QjIokRiz/5l/TFwpGu26zmDfDUkZmxZR/iNCW0VAmZE2YdyRm3PYcFcVXuZ1f/ff0D us9xGsO8V6F5EIwx/9Y6AWQdW7PoKHA21ri93PoRgjv+QoOifXEkhJwTKg5k5b1Tr7h9eRU/ Se2XigPVODjrN9FRfkx/JxlJcCs/igGJS05BmiZNIIRDKBGdXy/Fj5HQB2q5v5DfvrLMNTwK Aa8pn/em1SKC/l9aV9ygpN+cQPKoQjGxPPaId/rwX+GVxKl2vakjHLPLQmm3 Message-ID: Date: Sat, 10 Nov 2018 18:45:38 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.9.1 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Archives-Salt: 632f878b-4a05-43c9-9e41-3c09b6f9e9a2 X-Archives-Hash: 2b028cbb9fa2f1cf7005da6acef3aafe Rich Freeman wrote: > On Thu, Nov 8, 2018 at 8:16 PM Dale wrote: >> I'm trying to come up with a >> plan that allows me to grow easier and without having to worry about >> running out of motherboard based ports. >> > So, this is an issue I've been changing my mind on over the years. > There are a few common approaches: > > * Find ways to cram a lot of drives on one host > * Use a patchwork of NAS devices or improvised hosts sharing over > samba/nfs/etc and end up with a mess of mount points. > * Use a distributed FS > > Right now I'm mainly using the first approach, and I'm trying to move > to the last. The middle option has never appealed to me. And this is what I'm trying to avoid.  Doing one thing, realizing I should have done it different and then having to spend even more money to do it the right way.  I'm trying to get advice on what is the best way forward that I can afford.  Obviously I don't need a setup like facebook, google or something but I don't want to spend a few hundred dollars doing something only to realize, it needs to be sold to the next idiot on Ebay.   ROFL  You're giving me some good options to think on here.  ;-)  > > So, to do more of what you're doing in the most efficient way > possible, I recommend finding used LSI HBA cards. These have mini-SAS > ports on them, and one of these can be attached to a breakout cable > that gets you 4 SATA ports. I just picked up two of these for $20 > each on ebay (used) and they have 4 mini-SAS ports each, which is > capacity for 16 SATA drives per card. Typically these have 4x or > larger PCIe interfaces, so you'll need a large slot, or one with a > cutout. You'd have to do the math but I suspect that if the card+MB > supports PCIe 3.0 you're not losing much if you cram it into a smaller > slot. If most of the drives are idle most of the time then that also > demands less bandwidth. 16 fully busy hard drives obviously can put > out a lot of data if reading sequentially. > > You can of course get more consumer-oriented SATA cards, but you're > lucky to get 2-4 SATA ports on a card that runs you $30. The mini-SAS > HBAs get you a LOT more drives per PCIe slot, and your PCIe slots are > you main limiting factor assuming you have power and case space. > > Oh, and those HBA cards need to be flashed into "IT" mode - they're > often sold this way, but if they support RAID you want to flash the IT > firmware that just makes them into a bunch of standalone SATA slots. > This is usually a PITA that involves DOS or whatever, but I have > noticed some of the software needed in the Gentoo repo. > > If you go that route it is just like having a ton of SATA ports in > your system - they just show up as sda...sdz and so on (no idea where > it goes after that). Software-wise you just keep doing what you're > already doing (though you should be seriously considering > mdadm/zfs/btrfs/whatever at that point). > > That is the more traditional route. > > Now let me talk about distributed filesystems, which is the more > scalable approach. I'm getting tired of being limited by SATA ports, > and cases, and such. I'm also frustrated with some of zfs's > inflexibility around removing drives. These are constraints that make > upgrading painful, and often inefficient. Distributed filesystems > offer a different solution. > > A distributed filesystem spreads its storage across many hosts, with > an arbitrary number of drives per host (more or less). So, you can > add more hosts, add more drives to a host, and so on. That means > you're never forced to try to find a way to cram a few more drives in > one host. The resulting filesystem appears as one gigantic filesystem > (unless you want to split it up), which means no mess of nfs > mountpoints and so on, and all the other headaches of nfs. Just as > with RAID these support redundancy, except now you can lose entire > hosts without issue. With many you can even tell it which > PDU/rack/whatever each host is plugged into, and it will make sure you > can lose all the hosts in one rack. You can also mount the filesystem > on as many hosts as you want at the same time. > > They do tend to be a bit more complex. The big players can scale VERY > large - thousands of drives easily. Everything seems to be moving > towards Ceph/CephFS. If you were hosting a datacenter full of > VMs/containers/etc I'd be telling you to host it on Ceph. However, > for small scale (which you definitely are right now), I'm not thrilled > with it. Due to the way it allocates data (hash-based) anytime > anything changes you end up having to move all the data around in the > cluster, and all the reports I've read suggests it doesn't perform all > that great if you only have a few nodes. Ceph storage nodes are also > RAM-hungry, and I want to run these on ARM to save power, and few ARM > boards have that kind of RAM, and they're very expensive. > > Personally I'm working on deploying a cluster of a few nodes running > LizardFS, which is basically a fork/derivative of MooseFS. While it > won't scale nearly as well, below 100 nodes should be fine, and in > particular it sounds like it works fairly well with only a few nodes. > It has its pros and cons, but for my needs it should be sufficient. > It also isn't RAM-hungry. I'm going to be testing it on some > RockPro64s, with the LSI HBAs. > > I did note that Gentoo lacks a LizardFS client. I suspect I'll be > looking to fix that - I'm sure the moosefs ebuild would be a good > starting point. I'm probably going to be a whimp and run the storage > nodes on Ubuntu or whatever upstream targets - they're basically > appliances as far as I'm concerned. > > So, those are the two routes I'd recommend. Just get yourself an HBA > if you only want a few more drives. If you see your needs expanding > then consider a distributed filesystem. The advantage of the latter > is that you can keep expanding it however you want with additional > drives/nodes/whatever. If you're going over 20 nodes I'd use Ceph for > sure - IMO that seems to be the future of this space. > This is a lot to think on.  Money wise, and maybe even expansion wise, I may go with the PCI SATA cards and add drives inside my case.  I have plenty of power supply since it pulls at most 200 watts and I think my P/S is like 700 or 800 watts.  I can also add a external SATA card or another USB drive to do backups with as well.  At some point tho, I may have to build one of those little tiny systems that is basically nothing but SATA drive controllers and ethernet enabled.  Have that sitting in a closet somewhere running some small OS.  I can always just move the drives from my system to it if needed. One thing is for sure, you gave a lot of info and different ways to think on this.  Thanks to Rich and everyone else to for their thoughts.  It's certainly helped give me ideas I haven't thought of.  Dale :-)  :-) P. S.  For those who may wonder, my Mom is home and doing pretty well.  :-D