From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 8C8C0158020 for ; Thu, 8 Dec 2022 20:45:10 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 86E62E08AD; Thu, 8 Dec 2022 20:45:05 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id B34A7E082B for ; Thu, 8 Dec 2022 20:45:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1670532297; bh=y7gKOn3K8dNLATqJOjRwbPZIqx4iCw5vgt03VX7iQxY=; h=X-UI-Sender-Class:Date:From:To:Subject:References:In-Reply-To; b=M8+B2lnMRxRw+IAs53kFvMoiFqrBaA16L2jj6eCkh+MdwINdqreHolCd35IF5f06E iwPSm2vQZJ3GLs5F1MvzgqmrMuAtm0OyoDxtO0D1sx0x0p2LTer9q1O3rD+S7fAFjm oVyxwCew6CXGRlqyGba1ynSHZN9kZ93BKt0PL8uTp+umvj/aduH3673Bu6F+WAHCAO 0IqIhJkv0GnKajjlF4ow0ZFCWFJaCy2c0nTYsHAjDNSx8z3I0DGJVCuVQzb1V3f18m 1xndfmlHEdj4ZvCnlaFAbxIiLFmMbHcAcwOg+c5jDHV5sVc2uy1hwgko4T2u/26uKs yfXjpyZBipyPA== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from schatulle ([92.117.59.32]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MEm6F-1p9klG2S3X-00GK8q for ; Thu, 08 Dec 2022 21:44:57 +0100 Date: Thu, 8 Dec 2022 21:44:56 +0100 From: Frank Steinmetzger To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] NAS and replacing with larger drives Message-ID: Mail-Followup-To: gentoo-user@lists.gentoo.org References: <9407e524-2226-6ba9-dd7f-bac635d083e3@gmail.com> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="YMa2wykAS18R+Zzs" Content-Disposition: inline In-Reply-To: User-Agent: Mutt 2.2.9; VIM - Vi IMproved 9.0 X-Provags-ID: V03:K1:5sc1hBHc7VlQKEqGmDRB0cTcFSKQHim5QVBr5t6Xr4M3ORIf3Ha 5d748JjPrJq+Iwuq5Ii1xacJZidgsJCeZwE9JFSX8TxadkmxKv/Abi2yf9EiQkGZKeFw+b3 UfN1S2MtTuu2lHgNre4r2jerBL5PONTLS+ao9YELewMOmcz0KplgdHYnH4FBnT03jdk1BLb XFuXhV7TkZc5UlPGpviWQ== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:YC7AB+0YZ5M=;7KVhexIJZkN14mAZJRgsVyFkZzD Gn+14/zbefdalknVCCC7dJfL3Ni/sXP9KCdQWc+Gw4rwYkgucRT/reJN7mGgux14aXjbz2HTO jqzLpSVmkoDSs9lJa70LFDv89LGHOXs4DY0aS3RfUWYSxz69lZ65RZMjiGbB1tNXj2xjCMehY NYmod1BzCaH5GGRAkrbCnHiLvUwiIM65vC5DXlYuOYavh6fl0eFE0xycheTgq5A5vRX+nmew6 HseISSNDeWVsdqEbVxcdUUvIp5LfnJocCrbsdc6BzRgqRp5QiuNDWrDiyZFn4I5CUuYacEAXf dWNUH4R0/Ppgccvq2Z90StdLcVbpG8fANXNYbIragG0tOas4NOlnk6S18q/22LRN/4dRUlXOt ofyy9T7BUyQ1mVD4lUSU9UsaDIJaoY0IXz6COzadOysiNeJWElP/3KfdLGV74Zbah5jNg2hsw ZNioTvCbT9KmJVXJ8ErKEaOeJJRZT4jXEBerss4TIbE1pFBItgy+R3d6FFQlhn7/mUjJGpmfe bKXf+fo6VOnCxhqcmwBU0J0vojluBA9FGj/uYNVOhOKCOSOLA8vAyV9eOJ40a8zkUDEeoq1YL bW0IgdpXCmHqJ5YKxjZ5V+skYel5YJ4eKqUShZ9RpnuBkucEOhRkrT6O1BRCYxk4g+iFgn2CT m8FHpvYeOutplMdKWHwITbR1912QJV3x8NQH7EPkye74tLI1N61S+uRf6nXPotuv2/fZG/GgI aBo1RC/edi8/3kEGUgASoNFDaD86UojoUXyTiqZBLDBqCKKI4k9NPJEaaLNQ6fojI9XnZ3/9M twOmcLsP+tS2CD9HVkpLkX8Y9NNxiVNtKYITt49MovvE5yPGxkaYLEojx/E742HeiHPQJQl7S FCEv4jqjeBP6ZT/ZhbyxO1edrQ6Z+oNu7g9Dx8Cj+N1bW/la9SARkBJxLO1RtAnlyVVV2eFXQ 8oaq+YsusKq/2KcqhviJa8bZH0c= X-Archives-Salt: daff0691-528c-4f42-94ba-d0baee3e6c8d X-Archives-Hash: 3adfab63a58216772c36bf4398a77f3c --YMa2wykAS18R+Zzs Content-Type: multipart/mixed; boundary="hZ6x1XkILTbodPsl" Content-Disposition: inline --hZ6x1XkILTbodPsl Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Am Thu, Dec 08, 2022 at 06:36:14PM +0000 schrieb Wols Lists: > > > I've pretty much reached a limit on my backups.=C2=A0 I'm up to a 16= TB hard > > > drive for one and even that won't last long.=C2=A0 Larger drives are= much > > > more costly.=C2=A0 A must have NAS is quickly approaching.=C2=A0 I'v= e been > > > searching around and find some things confusing.=C2=A0 I'm hoping so= meone can > > > clear up that confusion.=C2=A0 I'm also debating what path to travel= down. > > > I'd also like to keep costs down as well.=C2=A0 That said, I don't m= ind > > > paying a little more for one that would offer a much better option. > > > > > > Path one, buy a NAS, possibly used, that has no drives.=C2=A0 If pos= sible, I > > > may even replace the OS that comes on it or upgrade if I can.=C2=A0 = I'm not > > > looking for fancy, or even RAID.=C2=A0 Just looking for a two bay NA= S that > > > will work.=C2=A0 First, what is a DAS?=C2=A0 Is that totally differe= nt than a > > > NAS?=C2=A0 From what I've found, a DAS is not what I'm looking for s= ince I > > > want a ethernet connection and the ability to control things over the > > > network.=C2=A0 It seems DAS lacks that feature but not real sure.=C2= =A0 I'm not > > > sure I can upgrade the software/OS on a DAS either. > > > [=E2=80=A6] > > > > DAS is direct-attached-storage. I don't think you want that. > > Depends. If it fits in the safe, and can be connected using one of these > eSATA thingy connectors, it might be a very good choice. > > [=E2=80=A6] > > I get the impression Dale isn't actually PLANNING his disk storage. It's > just a case of "help I'm downloading all this stuff where do I put it!!!" Haha, thanks for the laugh. > Get yourself a basic 4-way DAS/JBOD setup, PLAN where you're putting all > this stuff, and plug in and remove drives as required. You don't need all > these huge drives if you think about what you're going to do with it all. That=E2=80=99s actually a good idea. Either use a hot swap frame for an int= ernal 5=C2=BC=E2=80=B3 PC bay, a desktop dock for bare drives or a multi-bay enclosure. The market is big, you have lots of choices. USB (with or without integrated hub), eSATA, one or two bays, etc: https://skinflint.co.uk/?cat=3Dhddocks Advantages: - no separate system to maintain just for storage: save $$$, time and power - very flexible: no chassis limitation on number of disks - no bulky external enclosures, each using a different power brick and cable - minimum volume to put into a safe (just get or make a bulk storage case) Disadvantages: - not as =E2=80=9Cfancy=E2=80=9D as a NAS - possibly not all disks can be used at the same time - physical handling of naked disks takes more care - LVM is not practical, so use each disk separately - you gotta remember which files are where=C2=B9 - SATA connectors aren=E2=80=99t made for very many insertion cycles (I thi= nk the spec says 50?), which doesn=E2=80=99t mean they endure much more, but sti= ll =E2=80=A6 > (And while it takes time and hammers the system, I regularly record off t= he > TV getting a 2GB .ts file, convert it to mp4 - same resolution - and redu= ce > the size by an order of magnitude - maybe more. Well, ts uses mpeg2 encoding, just like old video DVDs, which is very inefficient when compared with modern h264/h265. Modern digital TV broadcast uses h264 by now. Incidentally, I got myself a new HDD today: an external 2.5=E2=80=B3 WD Pas= sport Ultra 5 TB with USB-C 3.0. Just because I like portable storage and also because I need temporary space if I want to convert my NAS RAID-Z2 to Z1. =C2=B9 I do have several external USB disks, plus the big NAS. All of which= don=E2=80=99t run very often. And I don=E2=80=99t want to turn them on just to look for a= certain file. That=E2=80=99s why I have another little script. ;-) It uses the `tre= e` command to save the complete content listing of a directory into a text file and names the file automatically by the name of the directory it crawls. So if I want to find a file, I just need to grep through my text files. --=20 Gr=C3=BC=C3=9Fe | Greetings | Salut | Qapla=E2=80=99 Please do not share anything from, with or about me on any social network. The whale is characterised by its bulky form factor. --hZ6x1XkILTbodPsl Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=FileTree #!/bin/sh # history # 2018-02-10 initial version # 2020-06-10 added -a option # ????-??-?? gzip result # 2021-03-22 put current date into output filename # 2021-04-06 code refactoring with more functions and input sanitisation # 2021-12-22 prefer zstd over gzip # 2022-01-23 don't create symlink if there is no previous file # 2022-01-30 added xz to compressors # 2022-12-08 some cleanup, added -C and -K options die() { echo "${@}" > /dev/stderr exit 1 } usage() { cat <<-EOF Usage: $(basename "$0") [-o NAME] [DIR] A wrapper to tree, it writes the content of DIR into a text file. The file is named after DIR and the current date, and symlink to the most recent version is set. The file is automatically compressed to zstd, xz or gzip, whichever is available in that order. Options: -a access attributes: owner, group, permissions -C do not compress result file -K do not keep backup of existing file in case of overwriting -o The destination where to write the trees. Default: . If it is a directory: write in there If it is a filename: use that as base. If not, use the name of the directory as name base. EOF } test_writable() { touch "$1" 2>/dev/null || die "Cannot create file in $(dirname "$1")." } # run tree and redirect output to destination file call_to_tree() { WHICH="$1" OUTPATH="$2-$1" shift 2 declare TREE_ARGS TREE_ARGS+=("$@") local DATED_PATH="$OUTPATH-$TODAY" local EXT local PACK local CREATE_SYMLINK=no if [ "$COMPRESS" = "no" ]; then TREE_ARGS+=("-o") TREE_ARGS+=("$DATED_PATH") elif command -v zstd > /dev/null; then PACK="zstd --rm -q -13" EXT=".zst" TREE_ARGS+=("-o") TREE_ARGS+=("$DATED_PATH") elif command -v xz > /dev/null; then PACK="xz" EXT=".xz" TREE_ARGS+=("-o") TREE_ARGS+=("$DATED_PATH") else PACK="gzip" EXT=".gz" fi ls "$OUTPATH-*" > /dev/null 2>&1 && CREATE_SYMLINK=yes # pack yet unpacked file if [ "$COMPRESS" != "no" ]; then [ -f "$DATED_PATH" ] && [ ! -f "$DATED_PATH$EXT" ] && $PACK "$DATED_PATH" fi # move away old file if [ -f "$DATED_PATH$EXT" ]; then [ "$KEEP_OLD" = "no" ] && rm -f "$DATED_PATH$EXT" || mv -f "$DATED_PATH$EXT" "$DATED_PATH.old$EXT" fi test_writable "$DATED_PATH" echo "Writing $WHICH tree to $(realpath "$DATED_PATH")$EXT" BEGIN_TIME=$(date +%s) "${TREE_ARGS[@]}" END_TIME=$(date +%s) echo >> "$DATED_PATH" df -BM . | awk 'NR==2 { print "Size: " $3 " Available: " $4 " Usage: " $5; }' >> "$DATED_PATH" [ "$PACK" ] && $PACK "$DATED_PATH" if [ "$CREATE_SYMLINK" = yes ]; then if [ -e "$OUTPATH" ] && [ ! -L "$OUTPATH" ]; then echo "Cannot set symlink, filename already taken." else echo "Setting symlink $OUTPATH -> $(basename "$DATED_PATH")" ln -sfn "$(basename "$DATED_PATH$EXT")" "$OUTPATH$EXT" fi fi echo "Reading of directory took $((END_TIME - BEGIN_TIME)) seconds." } # parse arguments unset ACCESSRIGHTS unset COMPRESS unset KEEP_OLD while getopts "CKaho:" OPTION do case $OPTION in C) COMPRESS=no ;; K) KEEP_OLD=no ;; a) ACCESSRIGHTS=yes ;; h) usage; exit 0 ;; o) OUTARG="$OPTARG" ;; esac done shift $((OPTIND-1)); OPTIND=1 # input sanitisation and derivation of destination paths DIR="${1:-$(pwd)}" [ -d "$DIR" ] || die "The given directory does not exist." [ "$OUTARG" ] && OUTARG="$(realpath "$OUTARG")" || OUTARG="$(pwd)" if [ -d "$OUTARG" ]; then OUTDIR="$OUTARG" OUTNAME="$(basename "$(realpath "$DIR")")" # realpath to catch `videotree .` [ "$OUTNAME" = "/" ] && OUTNAME=root else OUTDIR="$(dirname "$OUTARG")" OUTNAME="$(basename "$OUTARG")" fi TODAY="$(date +%F)" cd "$DIR" # write a very verbose file with permission and owner information call_to_tree "detailed" "$OUTDIR/$OUTNAME" tree -af -DFins --dirsfirst --du --timefmt "%Y-%m-%d %T" ${ACCESSRIGHTS:+-pug} # write a smaller version with only file size information #call_to_tree "simple" "$OUTDIR/$OUTNAME-tree" tree -ax -n --du -h --dirsfirst echo "Done." --hZ6x1XkILTbodPsl-- --YMa2wykAS18R+Zzs Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEVbE9o2D2lE5fhoVsizG+tUDUMMoFAmOSTMUACgkQizG+tUDU MMoj5A/9ERCIgHeJpINuvEQ9FSCes2JZ84TjnwxdUpNbR0Oiqrsqd62kY3V5zLlZ 8/k/yeWzVssnNXbIru2AnFfHugLTAYpAzXQsWQF4VIHrbHKzk7iYMqXqhb5P/KFD uHy6Qb1tcFT7oCzv6ycRUzHsoMoi9O6cztv1pfMSUo2U6nZC80SnjjE5MWeue6q5 czsoR7qy89hQcHl/xdfgC1zdzpzjVVLGAHQnUjndvowwfBtAI7rV9QlLWMIhUYDJ Z7g0+vZ+VPdmw41+gusye2zYA1lZM58Kri7IIS5vHh7Ju1NAJKaB3WDOE/UiRF9G lc9zJ9YgF4+Q7k5jA85PDPF83er5RCYIIe2+FPJN7NbKt1TAMoz7RHguNpmlr2hu 1GNAF4XHeswdm4ti0VJVfFGfpG7eZTECGpqnBSq8oLab7uxHT+FVRA2EZ6ePZ9vq Z6EeFxR7S03e0p+6vp3yBU1MD8QbbtzELB3933ZVhu3l5L3A8gAWUHkH/skWGa0h qBg6BW2UTqcVshyrtp/zSF1yxFuNSE36lXpfRTXN1rlaLoY7tU5+RyTwdvXspywA N0obUvqaZvt8rqN1wU+8IO5VsRHx5l5dCGvtkWXkP3Sny/PpU747fZEBgeC/yHXa 93b2xGp9FkhGEv4TIsYDCpUNHk7zEf9a03uKdb1LnUOTZAxBIUs= =aCtk -----END PGP SIGNATURE----- --YMa2wykAS18R+Zzs--