From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 9D7A1158020 for ; Fri, 9 Dec 2022 13:38:46 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id DACDAE089A; Fri, 9 Dec 2022 13:38:41 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 40003E0858 for ; Fri, 9 Dec 2022 13:38:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1670593114; bh=70P6ZdupDSST1dD9Pcc7nbplNEtae6v7cTVm5LUrisc=; h=X-UI-Sender-Class:Date:From:To:Subject:References:In-Reply-To; b=W4A7fhq8e1YLsSbrGooRkDShxUx6c1lPPaxJEFiuT7YCoHDdWrZbhfCdWeF8HxPn4 9Js07+5lehY5PcGTbXLeAwbsK0hqFiRm2BTApQS4MS874bVuxdDwnI3MEin4jvDQLV E3/Nvk/nnTddskfcJJnfJjDM/vNLMchGq3m+KuMMAqZbtxdnp4q6dIuP0MXo/yVgu9 7F72ZLeOWvugI9a2rR6RobJAJQM018soQ+OjADrevOTlar6oHc1g1H7T754jPt09FP 1AtKlVk8R2PeepH7TEdiSEda36EDrypwPTz11hv+pFWKk3sbFkjCe55iybMO3n0N3m d0xNyflmkLxkg== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from tp ([92.117.61.148]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1M9FjR-1ozUyr0zjz-006PmJ for ; Fri, 09 Dec 2022 14:38:34 +0100 Date: Fri, 9 Dec 2022 14:38:32 +0100 From: Frank Steinmetzger To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] NAS and replacing with larger drives Message-ID: Mail-Followup-To: gentoo-user@lists.gentoo.org References: <9407e524-2226-6ba9-dd7f-bac635d083e3@gmail.com> <2398014.jE0xQCEvom@lenovo.localdomain> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="0aei8xfzD1So9RAS" Content-Disposition: inline In-Reply-To: <2398014.jE0xQCEvom@lenovo.localdomain> User-Agent: Mutt 2.2.9; VIM - Vi IMproved 9.0 X-Provags-ID: V03:K1:lRfX3ToWVTaWwhdY1BiulA5mOFhuHIpcWOBIHlaYegdF5eJJtLm txisFx/0fUixjK+Ka02bE81Ecskr+j8Q69vqFWjS8gpUfuDxdz2QOojhzoItPVozdDuFv0s xKtMF6TtLoSBlyFcMF7CDR2HzO6iRfjkDvaF1s6nuYwslkFhpfVLy0UV+tp4lmw0q3vcHgr noM+RTHKkeYBwWCQjC6GA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:2MhnMMXyBWo=;/0yNY0xCcaPi4cVdM7kdKCKqpiF /+R1EVFmTbZmk8MR2OBPPHvtzs6oTQ30aB9GqZ144VaRi6q5Pj7AbjTmyPw2fyWD7gdHnV8Ae HZRCkKkuRb/H94Zak4/Q3zknze2gXI37fUVeC4ZO4LLLfYg/9CM0Js39fqvuGPl7Guhod6rYd iSc2WmMZn12Aq2Er1blXUoW2Tw97Zn5FkZWkPM/nemA0mTZaz3a0yfcUiJ3pls00tFmGYrqdE UFMljsz3aYHV4bh15SdnpqKEj5WcWpmacBgPHVdz/uhCIXUu1OoN5KmGPu9wnvp/UAgKt1iw7 Aa4BSy7HT1xrce1aLdJ5eeJNCE09IY/AdqDv5Bf/zOaL5wRVzEtMKADfVXL76E/qBsR/ZXka5 sXWBccCgvFGrSVD324xtshWZ0/CB/AM8oRKacwUl4aMSsjoT777soHEaEDgov7+BrK56ZkZHB bmmjdRAvgWfoq1x29jEpseu6bYTe47UEtZ+K5Blia+2Suga2ud6a8CJKieDxfa7rOCJ2ea7cl CmkLib+zYI3Sluomat8U+SbAMc6LcPYGBijjZcLj2pgIZr/SmJclfQPFpvSIp0/ptcWd6yW/K YJPZV4UjUgEeqfSXnmK19oNYdlkbRcCU/cQBBCrRsUp9CKtotHHE7ZJBpWEAc6mVbvy8clSlA xWKh+4Xokv8p7sSzIzyUDmD1RO38kjKcJ8XwoxFXm4fPEE/1YdQIQ6th2U0BX2bfXA5HP1xfw kMkaVi8nEItXW7AWMcKZQM3YY5++zkQ3oSb55oyxi7JGi5SA0o6ShWpsUemyFInx2cyWF0LrX 9H8Z8jwJglQmP+IczRE9g+ff4te8zg/+L0DdWI/JPebS6ujK9E3POVo7HFASZE0XFMMa+bv8O tu6+ORMW0m8M2cOzZDOVwaBg9g34kPFPxzk5CwZPYDzEQWLDZklLWo2GaemexhToacQlZKGgM V5xhGw== X-Archives-Salt: a89039f7-596d-4156-9d74-de6d435ee0cf X-Archives-Hash: b887ce9e19abf217182819583739c316 --0aei8xfzD1So9RAS Content-Type: multipart/mixed; boundary="kIwv9rVLZxM8L1KQ" Content-Disposition: inline --kIwv9rVLZxM8L1KQ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Am Fri, Dec 09, 2022 at 01:13:50PM +0000 schrieb Michael: > > > I get the impression Dale isn't actually PLANNING his disk storage. I= t's > > > just a case of "help I'm downloading all this stuff where do I put it= !!!" > > > > Haha, thanks for the laugh. > > Actually this had me thinking what is the need to back up the ... Interne= t? > [=E2=80=A6] > > I appreciate some of these video files may be rare finds, or there may be= a > risk some of these may be taken off the interwebs sooner or later. This > should leave a rather small subset of all downloads, which may merit a lo= cal > backup, just in case. I'd thought the availability of higher fiber downl= oad > speeds negates the need for local backups, of readily downloadable media. Good points. I am a big fan of having stuff locally as well, because I don= =E2=80=99t want to be dependent on a company=E2=80=99s servers and a working Internet = connection. But this mostly applies to my mobile device, because I don=E2=80=99t have a= data plan for mobile Internet. > > Well, ts uses mpeg2 encoding, just like old video DVDs, which is very > > inefficient when compared with modern h264/h265. Modern digital TV broa= dcast > > uses h264 by now. > > Depending on the PVR make/model I've seen 1080p resolution recordings with > .m2ts and .ts file extensions, while the codecs inside them are the same. I wasn=E2=80=99t aware that ts could contain h264. But then again=E2=80=94I= never really bothered with live TV recordings in recent years. These days, if I find something interesting, I download the show form the TV channel=E2=80=99s we= bsite (called Mediathek in Germany, a word play on Bibliothek, meaning library). Interestingly though, the picture quality is noticably worse than what I receive via DVB-T. > > =C2=B9 I do have several external USB disks, plus the big NAS. All of w= hich don=E2=80=99t > > run very often. And I don=E2=80=99t want to turn them on just to look f= or a certain > > file. That=E2=80=99s why I have another little script. ;-) It uses the = `tree` > > command to save the complete content listing of a directory into a text > > file and names the file automatically by the name of the directory it > > crawls. So if I want to find a file, I just need to grep through my text > > files. > > Backup scripts utilising rsync, tar, etc. can output a log file which con= tains > (some) details of all the backed up files. Nothing as sophisticated as > Frank's script, but it allows for a quick search against the name of the = file > or directory, before extraction. Naturally, I just discovered two bugs in the script while I was re-reading my mail. One of them broke the creation of the symlink which points to the most recent version of a script output. The other prevented normal operation if only gzip was available amongst the used compressors. --=20 Gr=C3=BC=C3=9Fe | Greetings | Salut | Qapla=E2=80=99 Please do not share anything from, with or about me on any social network. We promise nothing, but that we keep. --kIwv9rVLZxM8L1KQ Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=FileTree #!/usr/bin/env bash # history # 2018-02-10 initial version # 2020-06-10 added -a option # ????-??-?? gzip result # 2021-03-22 put current date into output filename # 2021-04-06 code refactoring with more functions and input sanitisation # 2021-12-22 prefer zstd over gzip # 2022-01-23 don't create symlink if there is no previous file # 2022-01-30 added xz to compressors # 2022-12-08 some cleanup, added -C and -K options # 2022-12-09 bug fixes die() { echo "${@}" > /dev/stderr exit 1 } usage() { cat <<-EOF Usage: $(basename "$0") [-o NAME] [DIR] A wrapper to tree, it writes the content of DIR into a text file. The file is named after DIR and the current date, and symlink to the most recent version is set. The file is automatically compressed to zstd, xz or gzip, whichever is available in that order. Options: -a access attributes: owner, group, permissions -C do not compress result file -K do not keep backup of existing file in case of overwriting -o The destination where to write the trees. Default: . If it is a directory: write in there If it is a filename: use that as base. If not, use the name of the directory as name base. EOF } test_writable() { touch "$1" 2>/dev/null || die "Cannot create file in $(dirname "$1")." } # run tree and redirect output to destination file call_to_tree() { WHICH="$1" OUTPATH="$2-$1" shift 2 declare TREE_ARGS TREE_ARGS+=("$@") TREE_ARGS+=("-o") TREE_ARGS+=("$DATED_PATH") local DATED_PATH="$OUTPATH-$TODAY" local EXT local PACK local CREATE_SYMLINK=no if [ "$COMPRESS" = "no" ]; then 0 elif command -v zstd > /dev/null; then PACK="zstd --rm -q -13" EXT=".zst" elif command -v xz > /dev/null; then PACK="xz" EXT=".xz" else PACK="gzip" EXT=".gz" fi ls "$OUTPATH-"* > /dev/null 2>&1 && CREATE_SYMLINK=yes # pack yet unpacked file if [ "$COMPRESS" != "no" ]; then [ -f "$DATED_PATH" ] && [ ! -f "$DATED_PATH$EXT" ] && $PACK "$DATED_PATH" fi # move away old file if [ -f "$DATED_PATH$EXT" ]; then [ "$KEEP_OLD" = "no" ] && rm -f "$DATED_PATH$EXT" || mv -f "$DATED_PATH$EXT" "$DATED_PATH.old$EXT" fi test_writable "$DATED_PATH" echo "Writing $WHICH tree to $(realpath "$DATED_PATH")$EXT" BEGIN_TIME=$(date +%s) "${TREE_ARGS[@]}" END_TIME=$(date +%s) echo >> "$DATED_PATH" df -BM . | awk 'NR==2 { print "Size: " $3 " Available: " $4 " Usage: " $5; }' >> "$DATED_PATH" [ "$PACK" ] && $PACK "$DATED_PATH" if [ "$CREATE_SYMLINK" = yes ]; then if [ -e "$OUTPATH" ] && [ ! -L "$OUTPATH" ]; then echo "Cannot set symlink, filename already taken." else echo "Setting symlink $OUTPATH -> $(basename "$DATED_PATH")" ln -sfn "$(basename "$DATED_PATH$EXT")" "$OUTPATH$EXT" fi fi echo "Reading of directory took $((END_TIME - BEGIN_TIME)) seconds." } # parse arguments unset ACCESSRIGHTS unset COMPRESS unset KEEP_OLD while getopts "CKaho:" OPTION do case $OPTION in C) COMPRESS=no ;; K) KEEP_OLD=no ;; a) ACCESSRIGHTS=yes ;; h) usage; exit 0 ;; o) OUTARG="$OPTARG" ;; esac done shift $((OPTIND-1)); OPTIND=1 # input sanitisation and derivation of destination paths DIR="${1:-$(pwd)}" [ -d "$DIR" ] || die "The given directory does not exist." [ "$OUTARG" ] && OUTARG="$(realpath "$OUTARG")" || OUTARG="$(pwd)" if [ -d "$OUTARG" ]; then OUTDIR="$OUTARG" OUTNAME="$(basename "$(realpath "$DIR")")" # realpath to catch `videotree .` [ "$OUTNAME" = "/" ] && OUTNAME=root else OUTDIR="$(dirname "$OUTARG")" OUTNAME="$(basename "$OUTARG")" fi TODAY="$(date +%F)" cd "$DIR" # write a very verbose file with permission and owner information call_to_tree "detailed" "$OUTDIR/$OUTNAME" tree -af -DFins --dirsfirst --du --timefmt "%Y-%m-%d %T" ${ACCESSRIGHTS:+-pug} # write a smaller version with only file size information #call_to_tree "simple" "$OUTDIR/$OUTNAME-tree" tree -ax -n --du -h --dirsfirst echo "Done." --kIwv9rVLZxM8L1KQ-- --0aei8xfzD1So9RAS Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEVbE9o2D2lE5fhoVsizG+tUDUMMoFAmOTOlYACgkQizG+tUDU MMpe8BAAqVVJYDeJl0LRhbf2hpdIFXBJP6hMV2niWN4G4oRRArWgMUE5PsMBZyw2 45WskIAafNMgty80sVHtQTsfIY+1B3ySjfY6aajzBImFYmn789SasP1FYZ+N0ttr gbGMfOUpQL5wNrirORyJqmUKZezoXVgZI12fv96NK/QGkhJI9Y9eTl6PvrkuwCVp dPQbm4zanS8hRZonu9vBiygn1071Eqg0RuDl2BQyY8o+nP1hjaQzokkT298WC7Ja sELrhb9f/Oj63pE+cjdI+6E+e1/U6jU2YOsDYTBPOmz/dzUxxi4R6GFf19RPGazG WfA6p1k4f3qM7QU8b6A9jv6OfDs+FCsKEJ4JWm/0skF+kCmg+0AMp+0G9mhfPuEj evDZj8LF9keZBVFZqzxjsy3IdQ9/wyWSVdqDrrH5M/bfaTI2FQiwSfkVAfhtsDoG ZPX8RnDHmczj0jJyQETy/aPVfes92W38S/AjYTdq07ZCJa2nWK9jN1yo8zsPs5Iw OuzyIST7VCsxufk9ke8IizNArnAfHS91sR1u2/y6Xs5V20+4JmJIBVHrYCHvkicZ D4N3WBmEtWqK2M3xNuDLnZ6aGiaEs7YTiIs2/nRYnTC6v0bfSIEr+CloLwutZRyp Tvfc1B6gRei2w7QBOlva0mlefNsdosYsJyTuND+ZeGZ0jDmZQls= =ekeO -----END PGP SIGNATURE----- --0aei8xfzD1So9RAS--