From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id E429B158094 for ; Fri, 26 Aug 2022 23:08:08 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 58BD9E08E0; Fri, 26 Aug 2022 23:08:01 +0000 (UTC) Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id A539FE0874 for ; Fri, 26 Aug 2022 23:08:00 +0000 (UTC) Received: by mail-qk1-x729.google.com with SMTP id j6so2268461qkl.10 for ; Fri, 26 Aug 2022 16:08:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:mime-version:user-agent:date :message-id:autocrypt:openpgp:from:references:to:subject:from:to:cc; bh=Kp+4x8Skg9sudXwCQoKkf7GbM5MubyeVWmI6z+0MU2s=; b=Kgs7tB0kZB+1hbFjZDu5XBUDJLP7Xt0Hq0mOWesP+EzEP55usxdbAymNhvrjcZlY2y rnoJPJuS5fwU28EHzUmD7+E+x6gV1Xo+mPPCAk6BqfQiVkz9VbtcB1MPzl0ObUCtH9y5 NPCA+beI9uZKzJtK2splWx56PqIcbWPulFOmKEjVLK6n6WmQjjNVXvw8jDC1fyuSOhjf /2Q4CLMyGnj4VYBZpXz01t85nJkZcZRQ86khjBhOjjOavwiSQrpL/KpToUhwyj6TY6zN 2m5MtrzSOykxQ1o3+PRvBMc9X3zz6LAmddzrpHxsaZCEN03R5oz9TKk6P4C7Y/BQS99u ZhYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:mime-version:user-agent:date :message-id:autocrypt:openpgp:from:references:to:subject :x-gm-message-state:from:to:cc; bh=Kp+4x8Skg9sudXwCQoKkf7GbM5MubyeVWmI6z+0MU2s=; b=SzChJRrlMmlQ/eTSMXh58KEh50kLFvLI/rJ2NxFiYcgFiFCYG23PekPbuVFQoi+7t5 JXamx+M9zwla3KKG74xb3EmTEEOLurJXXtqbClCvoU1WCaVnmarp7/Kt3IPrIvZy4uRE pnaUDVW5NWu+hf6Hkhtaj4Ss4A+2SYFC6TotUjQl625ddox3o1+z8lf47w3A/FfDahQG eNTvMP5+jPvF8JJ6EFSIGOIct/Zf/eDZqsa7VXGjDNlc9hcEV4CHo+Tw+Peg46BnMxMK jN6US1M6tMz2Qmhpw7kW9nO3gvgn+T1aWm0CyBy/YJ+Y9MRF25GZmj5xVK6XsmpC1hXg CtPg== X-Gm-Message-State: ACgBeo1GhOz4ID9ki7Gxxs9VE0qJhqw5OaG2J+bE0GqjXg4SWZuGYFi8 RiRQ9rZNO+Q8kG8E9Y/mIAG1DYreVZk= X-Google-Smtp-Source: AA6agR405K4m+PIMJityWSs3brgIz3josBu53eUQipG7H1xRXVcvVu9PkW6pyTj8nRpXLiRhYWju7Q== X-Received: by 2002:a05:620a:468e:b0:6bb:1974:ef78 with SMTP id bq14-20020a05620a468e00b006bb1974ef78mr1346024qkb.87.1661555279704; Fri, 26 Aug 2022 16:07:59 -0700 (PDT) Received: from [10.0.0.2] ([207.244.201.197]) by smtp.gmail.com with ESMTPSA id ay6-20020a05620a178600b006b59f02224asm638141qkb.60.2022.08.26.16.07.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 26 Aug 2022 16:07:59 -0700 (PDT) Subject: Re: [gentoo-user] Getting maximum space out of a hard drive To: gentoo-user@lists.gentoo.org References: <57a9895b-9357-17f1-8fb5-d0ede952eefc@gmail.com> <20220819042614.bj5crtjkgszbnshh@grusum.dhaller.de> <289fe32e-2815-c361-ea80-73d8df539417@iinet.net.au> <6f3feff2-eac9-f6ec-4a3c-c511cf469603@gmail.com> <6e3ee99d-46da-4cdf-93ed-838591a50f67@users.sourceforge.net> <12905e2c-f3ad-b7b9-78e1-4604e38f8a8e@gmail.com> From: Dale Openpgp: preference=signencrypt Autocrypt: addr=rdalek1967@gmail.com; prefer-encrypt=mutual; keydata= mQINBGFSciYBEADcEGMyJBSuavKO/XKUVvgkxck7Nl8Iuu8N2lcnRji/rSKg5c1Acix1ll9i oW8JBCHwvn0+Xy60BvEsqcup3YSHw5STl/bR1ePEehtnYrg8FdjdS91+B805RfnKMm69rFVI wLSBHQrSG1yxHd8CloWoEdhmVtP24buajbh114bgXd9ahtpZrCVMrWdWYUg2mEXguGV5uNAh Rf8SWxDNc79w24JxsV34a8niMUYMjzWr0rafIbzk732X38vGjVMLo/2mMpkbp9mPp++LHoY+ 0Pet8zxxdXPJSCd475kza1AD+hhSyBZXB9yknYWgyY3cZe1rGmooJSi2KX4QxO7npwLThcO1 be6KKRkd35+Fi/a1BzVOHsZMiK/gcwxEFoMd27gir4ehaeHJfFXl+65w4hj0EsOZSxrJrm2C R50g5By2czSKP1bADEygFNpIJj51AR+wM88NImG2RPtlT2maYBzazvF05g65cdHXGp1C7W5P wwwKU2DgABB2t7N7z5A69LnryBRw4zUYDRRYLTYlBlYgg+xILm2c0OrBdxJgLJa7JE50Eo25 d3PFwt9J0gYvqy6sPFLl9So0sDg9zm0hKQtXOP5kgropUFGrNoJI+mjwF4rYLRBVzZwNAvlO OhEvHubBo3mEllv4x+FeptwXZxlk7gUsdqI8AxnFB8K9wi6FVQARAQABtBtEYWxlIDxyZGFs ZWsxOTY3QGdtYWlsLmNvbT6JAk4EEwEIADgCGyMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AW IQQSG1h01ruv/WNXc3Q3RqOgiQH1GwUCYVJy8gAKCRA3RqOgiQH1G+waEACeTZCt77jnRAmQ AV7otKuZekDWiLi3Eig8tj5ZJiCNSYA/hIxzmexRP0GMqjitcXK1iGwWcvMzzvIq30GAjIfB 4BR38cnXbtBa6fNewiT7QaZe/Hn6yBRldXNQypzbHy+/o27bUEy+oX4rE7etUgEHQAjuw7xz XFWg4tH1/KJvsOVY5upnWc5LdxYhsuQ3dQD4b22GsK0pOBDfb9PiirYM8eGKvrVuq4E/c75z lDDFhINl18lNZ9D0ZFL3IkTjHsAAqFH9uhnnEB8CWdHbBewPEfRaOhBUYWZ3Q8uTkmDgZT8q D9jlvLEdw7Nh2ApdxoepnI/4D+ql2Gr4DtH7SEPydr5gcf1Qr/2bXRb1hAYnIVcbncs/Bm3Z bkRKPVWMfE3Fusa+p5hMzixk0YysMaTHlc7mYRYAEZGnPMXnmcCbetwARU7A0yz1M1kCMOAQ Lsz8KH5kv3cRenMB6SFfjND2JfAK61H5TtnPq3L8noS2ZykRYxq9Nm3X64O1tJojIKBoZFr8 AwYNCvqC6puUyGMuzHPh7jPof8glfrrEKIYUvNPGMDoVX3IGetxh/9l6NcxgFA4JGoR+LS3C zmeNrwlllAe3OEUfKoWVQ+pagpSdM+8hHolaSda4Ys66Z3fCR4ZvcTqfhTAVskpqdXa4isAk 7vTcXu3L499ttywEp7rJTbkCDQRhUnImARAAncUdVhmtRr59zqpTUppKroQYlzR0jv8oa7DG K4gakTAT2N7evnI9wpssmzyVk8VEiLzhnFQ/Ol3FRt6hZCXDJt0clyHOyTfvz/MNFttWuZTc mLpSvmRR6VRjAH+Tz3Eam2xUw3PGuH97BcXQ3NnX3msv1UDxtxxBu6e2YrdeOhrCUSgzokcJ 98ChUNy934cgepPybAI12lSWqVFQ1aG7jExZfiUk+333fPSDbpKoZbTW5YJLXbycmW/C1IWL qYQyNjRWKaGoJtUWFhhmNiOQct7n90aKivNVPavmN+UQ9LlMaINtf9T6XCzLfogCFsulDCDJ 0yNQLDTurHaB4E71xoctgXmLLq9z1RQ0W2XiVAAOZQj6K3+d0AOUjDhCQ2QW8dUSq0ckkZXV DKVJOGS8Nhf2eIWIqRnP3AcUiiaiFGqUaVUmUAZ6h/oJmgghEu/1S+pcuUKU5i69+XCZ3hH2 Jzwzbf7K+FAIkOhCfHncF8i1N1pk00pOVykNnqHTfFo3qFusHt0ZWgXVnnn4pYdXqZNoDhvF BRE5Vm4k/k96Pw8HRx6Os6eFSRrlqGzRgqsu86FekxusXB9UGv4lJhtU/J+8MRWsh22K718s DbQnABicGKFz1qQlWvcf59oTByhLINJCBt1WXl+TzJDXepr3QSkqmK41dO9Hob97C9dMiK8A EQEAAYkCNgQYAQgAIAIbDBYhBBIbWHTWu6/9Y1dzdDdGo6CJAfUbBQJhUnLyAAoJEDdGo6CJ AfUbVHIQAKSWw620vPhR3A/njU2z77F3z/Jk+HTKdE3fIyWSWdkYN7CBFL0NguOMP30WZ+qE sJhZu7T5hf251MwQUUt27xlfnKYOmQs7CqONlXuXlGZI6WufrUjxNcVz+5gJsqvUWuuJWsgg sDmE92IBnfG/f81fPHWQyfr/SF4wYDMyoFp5xCCQpp1zB63iuFvvrhxBkEHzmbRtVDOhl0Xp BVEDR1w3QRACw9QJD/KM05Czv9JNQYlwinWO/OaQ9cMlUpKLgswUPg9IZ5vucxScfuAUA5uC B1jlAQ8ZPlVukBmbEv5RGOv+lpuEbA3YDMVtEeH4YMFbjt/+vH3Cr2vTbp5JlpByLburJEH0 WXZLUawEfUsZvVwpOuJK75vaa2HYXee+Cb3iCIzwfIfctdlqzUcbGRczlRNM59hpvj4z29Gh 3kAxVHItAYq54ikxQ9l4hQ8s9sLYPbX/WtcBxNX8crBSw0FLnmzGleVEtBHyqtt5CLzQNgrj GYWl1vKDUmRPw1CdZ1c+fMN9CY11jOM5B5ZnqZWfDeVYO2iJ5SuvTycChexCb8WYn1bdCBIo bBtga2RBXbVt4Mh9E4owsszefn51MwfjXxB20Fc5k3GU1AVpTCMs3ayYCzo0b2pvEvdjtDcA CYLEFPWgaFX9iQAM/CDfKvTtvgGWpqtCL2raq/mQoJEU Message-ID: <3242fccb-2a4b-f02d-f86e-938610a6e936@gmail.com> Date: Fri, 26 Aug 2022 18:07:58 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0 SeaMonkey/2.53.13 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Archives-Salt: f4b9ac49-4bdb-4727-b1ce-a5fd0c98f966 X-Archives-Hash: 58f9c6e31e1c21a441924ce512211eee Rich Freeman wrote: > On Fri, Aug 26, 2022 at 7:26 AM Dale wrote: >> I looked into the Raspberry and the newest version, about $150 now, doesn't even have SATA ports. > The Pi4 is definitely a step up from the previous versions in terms of > IO, but it is still pretty limited. It has USB3 and gigabit, and they > don't share a USB host or anything like that, so you should get close > to full performance out of both. The CPU is of course pretty limited, > as is RAM. Biggest benefit is the super-low power consumption, and > that is something I take seriously as for a lot of cheap hardware that > runs 24x7 the power cost rapidly exceeds the purchase price. I see > people buying old servers for $100 or whatever and those things will > often go through $100 worth of electricity in a few months. > > How many hard drives are you talking about? There are two general > routes to go for something like this. The simplest and most > traditional way is a NAS box of some kind, with RAID. The issue with > these approaches is that you're limited by the number of hard drives > you can run off of one host, and of course if anything other than a > drive fails you're offline. The other approach is a distributed > filesystem. That ramps up the learning curve quite a bit, but for > something like media where IOPS doesn't matter it eliminates the need > to try to cram a dozen hard drives into one host. Ceph can also do > IOPS but you're talking 10GbE + NVMe and big bucks, and that is how > modern server farms would do it. > > I'll describe the traditional route since I suspect that is where > you're going to end up. If you only had 2-4 drives total you could > probably get away with a Pi4 and USB3 drives, but if you want > encryption or anything CPU-intensive you're probably going to > bottleneck on the CPU. It would be fine if you're more concerned with > capacity than storage. > > For more drives than that, or just to be more robust, then any > standard amd64 build will be fine. Obviously a motherboard with lots > of SATA ports will help here. However, that almost always is a > bottleneck on consumer gear, and the typical solution to that for SATA > is a host bus adapter. They're expensive new, but cheap on ebay (I've > had them fail though, which is probably why companies tend to sell > them while they're still working). They also use a ton of power - > I've measured them using upwards of 60W - they're designed for servers > where nobody seems to care. A typical HBA can provide 8-32 SATA > ports, via mini-SAS breakout cables (one mini-SAS port can provide 4 > SATA ports). HBAs tend to use a lot of PCIe lanes - you don't > necessarily need all of them if you only have a few drives and they're > spinning disks, but it is probably easiest if you get a CPU with > integrated graphics and use the 16x slot for the HBA. That or get a > motherboard with two large slots (they usually aren't 16x, but getting > 4-8x slots on a consumer motherboard isn't super-common). > > For software I'd use mdadm plus LVM. ZFS or btrfs are your other > options, and those can run on bare metal, but btrfs is immature and > ZFS cannot be reshaped the way mdadm can, so there are tradeoffs. If > you want to use your existing drives and don't have a backup to > restore or want to do it live, then the easiest option there is to add > one drive to the system to expand capacity. Put mdadm on that drive > as a degraded raid1 or whatever, then put LVM on top, and migrate data > from an existing disk live over to the new one, freeing up one or more > existing drives. Then put mdadm on those and LVM and migrate more > data onto them, and so on, until everything is running on top of > mdadm. Of course you need to plan how you want the array to look and > have enough drives that you get the desired level of redundancy. You > can start with degraded arrays (which is no worse than what you have > now), then when enough drives are freed up they can be added as pairs > to fill it out. > > If you want to go the distributed storage route then CephFS is the > canonical solution at this point but it is RAM-hungry so it tends to > be expensive. It is also complex, but there are ansible playbooks and > so on to manage that (though playbooks with 100+ plays in them make me > nervous). For something simpler MooseFS or LizardFS are probably > where I'd start. I'm running LizardFS but they've been on the edge of > death for years upstream and MooseFS licensing is apparently better > now, so I'd probably look at that first. I did a talk on lizardfs > recently: https://www.youtube.com/watch?v=dbMRcVrdsQs > This is some good info.  It will likely start off with just a few hard drives but that will grow over time.  I also plan to have a large drive as a spare as well, in case one starts having issues and needs replacing quick.  I'd really like to be using RAID at least the two copies one but may take time, plus I got to learn how to do the thing. ;-) I may use NAS software.  I've read/seen people use that and it is on a USB stick or something.  They say once it is booted up, it doesn't need to access the USB stick much.  I guess it is set to load into memory at boot up, like some rescue media does.  I think that uses ZFS for the file system which is a little like LVM.  The bad thing about using that tho, I may not be able to just move the drives over to the new NAS since it may not accept LVM well.  I don't know much on that yet.  I may end up having to buy drives and just rsync them over.  One good thing, I have a 1GB capable router.  It even has fast wifi if I get that.  Plus, I'll have extra drives depending on how I work this. My first thing, buy a case to organize all this.  I looked at a Fractal Design Node 804 case.  It is a NAS case.  I think it can handle a wide variety of mobos too.  It can also hold a lot of drives as well.  Eight if I recall.  If I put the OS on a USB stick or something, that is a lot of drive spaces.  I may could add a drive cage if needed.  It's a fair sized case and certainly a good start.  It will take me a while to build all this.  Thing is, since I'll use it like a backup tool, I need to be able to put it in a safe place.  I wish I could get a fire safe to stick it into.  The biggest part is 16" I think.  Plenty of cooling as well.  I wish I had another Cooler Master HAF-932 like I have now but that almost certainly won't fit in a fire safe.  Dang thing is huge.  ROFL  Lots to think on.  Dale :-)  :-)