Re: [gentoo-user] NAS and replacing with larger drives

[Frank Steinmetzger <Warp_7@gmx.de>]

[-- Attachment #1.1: Type: text/plain, Size: 2972 bytes --]

Am Fri, Dec 09, 2022 at 01:13:50PM +0000 schrieb Michael:

> > > I get the impression Dale isn't actually PLANNING his disk storage. It's
> > > just a case of "help I'm downloading all this stuff where do I put it!!!"
> >
> > Haha, thanks for the laugh.
>
> Actually this had me thinking what is the need to back up the ... Internet?
> […]
>
> I appreciate some of these video files may be rare finds, or there may be a
> risk some of these may be taken off the interwebs sooner or later.  This
> should leave a rather small subset of all downloads, which may merit a local
> backup, just in case.  I'd thought the availability of higher fiber download
> speeds negates the need for local backups, of readily downloadable media.

Good points. I am a big fan of having stuff locally as well, because I don’t
want to be dependent on a company’s servers and a working Internet connection.
But this mostly applies to my mobile device, because I don’t have a data plan
for mobile Internet.

> > Well, ts uses mpeg2 encoding, just like old video DVDs, which is very
> > inefficient when compared with modern h264/h265. Modern digital TV broadcast
> > uses h264 by now.
>
> Depending on the PVR make/model I've seen 1080p resolution recordings with
> .m2ts and .ts file extensions, while the codecs inside them are the same.

I wasn’t aware that ts could contain h264. But then again—I never really
bothered with live TV recordings in recent years. These days, if I find
something interesting, I download the show form the TV channel’s website
(called Mediathek in Germany, a word play on Bibliothek, meaning library).
Interestingly though, the picture quality is noticably worse than what I
receive via DVB-T.

> > ¹ I do have several external USB disks, plus the big NAS. All of which don’t
> > run very often. And I don’t want to turn them on just to look for a certain
> > file. That’s why I have another little script. ;-) It uses the `tree`
> > command to save the complete content listing of a directory into a text
> > file and names the file automatically by the name of the directory it
> > crawls. So if I want to find a file, I just need to grep through my text
> > files.
>
> Backup scripts utilising rsync, tar, etc. can output a log file which contains
> (some) details of all the backed up files.  Nothing as sophisticated as
> Frank's script, but it allows for a quick search against the name of the file
> or directory, before extraction.

Naturally, I just discovered two bugs in the script while I was re-reading
my mail. One of them broke the creation of the symlink which points to the
most recent version of a script output. The other prevented normal operation
if only gzip was available amongst the used compressors.

-- 
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

We promise nothing, but that we keep.

[-- Attachment #1.2: FileTree --]
[-- Type: text/plain, Size: 4193 bytes --]

#!/usr/bin/env bash

# history
# 2018-02-10 initial version
# 2020-06-10 added -a option
# ????-??-?? gzip result
# 2021-03-22 put current date into output filename
# 2021-04-06 code refactoring with more functions and input sanitisation
# 2021-12-22 prefer zstd over gzip
# 2022-01-23 don't create symlink if there is no previous file
# 2022-01-30 added xz to compressors
# 2022-12-08 some cleanup, added -C and -K options
# 2022-12-09 bug fixes

die() {
    echo "${@}" > /dev/stderr
    exit 1
}

usage() {
    cat <<-EOF
	Usage: $(basename "$0") [-o NAME] [DIR]
	A wrapper to tree, it writes the content of DIR into a text file.
	The file is named after DIR and the current date, and symlink to the
	most recent version is set. The file is automatically compressed to
	zstd, xz or gzip, whichever is available in that order.

	Options:
	  -a    access attributes: owner, group, permissions
	  -C    do not compress result file
	  -K    do not keep backup of existing file in case of overwriting
	  -o    The destination where to write the trees.
	        Default: .
	        If it is a directory: write in there
	        If it is a filename: use that as base. If not,
	        use the name of the directory as name base.
	EOF
}

test_writable() {
    touch "$1" 2>/dev/null || die "Cannot create file in $(dirname "$1")."
}

# run tree and redirect output to destination file
call_to_tree() {
    WHICH="$1"
    OUTPATH="$2-$1"
    shift 2

    declare TREE_ARGS
    TREE_ARGS+=("$@")
    TREE_ARGS+=("-o")
    TREE_ARGS+=("$DATED_PATH")

    local DATED_PATH="$OUTPATH-$TODAY"
    local EXT
    local PACK
    local CREATE_SYMLINK=no

    if [ "$COMPRESS" = "no" ]; then
        0
    elif command -v zstd > /dev/null; then
        PACK="zstd --rm -q -13"
        EXT=".zst"
    elif command -v xz > /dev/null; then
        PACK="xz"
        EXT=".xz"
    else
        PACK="gzip"
        EXT=".gz"
    fi

    ls "$OUTPATH-"* > /dev/null 2>&1 && CREATE_SYMLINK=yes

    # pack yet unpacked file
    if [ "$COMPRESS" != "no" ]; then
        [ -f "$DATED_PATH" ] && [ ! -f "$DATED_PATH$EXT" ] && $PACK "$DATED_PATH"
    fi
    # move away old file
    if [ -f "$DATED_PATH$EXT" ]; then
        [ "$KEEP_OLD" = "no" ] && rm -f "$DATED_PATH$EXT" || mv -f "$DATED_PATH$EXT" "$DATED_PATH.old$EXT"
    fi

    test_writable "$DATED_PATH"
    echo "Writing $WHICH tree to $(realpath "$DATED_PATH")$EXT"

    BEGIN_TIME=$(date +%s)
    "${TREE_ARGS[@]}"
    END_TIME=$(date +%s)

    echo >> "$DATED_PATH"
    df -BM . | awk 'NR==2 { print "Size: " $3 "   Available: " $4 "   Usage: " $5; }' >> "$DATED_PATH"
    [ "$PACK" ] && $PACK "$DATED_PATH"

    if [ "$CREATE_SYMLINK" = yes ]; then
        if [ -e "$OUTPATH" ] && [ ! -L "$OUTPATH" ]; then
            echo "Cannot set symlink, filename already taken."
        else
            echo "Setting symlink $OUTPATH -> $(basename "$DATED_PATH")"
            ln -sfn "$(basename "$DATED_PATH$EXT")" "$OUTPATH$EXT"
        fi
    fi

    echo "Reading of directory took $((END_TIME - BEGIN_TIME)) seconds."
}

# parse arguments
unset ACCESSRIGHTS
unset COMPRESS
unset KEEP_OLD

while getopts "CKaho:" OPTION
do
    case $OPTION in
        C) COMPRESS=no ;;
        K) KEEP_OLD=no ;;
        a) ACCESSRIGHTS=yes ;;
        h) usage; exit 0 ;;
        o) OUTARG="$OPTARG" ;;
    esac
done
shift $((OPTIND-1)); OPTIND=1

# input sanitisation and derivation of destination paths
DIR="${1:-$(pwd)}"
[ -d "$DIR" ] || die "The given directory does not exist."

[ "$OUTARG" ] && OUTARG="$(realpath "$OUTARG")" || OUTARG="$(pwd)"
if [ -d "$OUTARG" ]; then
    OUTDIR="$OUTARG"
    OUTNAME="$(basename "$(realpath "$DIR")")" # realpath to catch `videotree .`
    [ "$OUTNAME" = "/" ] && OUTNAME=root
else
    OUTDIR="$(dirname "$OUTARG")"
    OUTNAME="$(basename "$OUTARG")"
fi
TODAY="$(date +%F)"

cd "$DIR"

# write a very verbose file with permission and owner information
call_to_tree "detailed" "$OUTDIR/$OUTNAME"      tree -af -DFins --dirsfirst --du --timefmt "%Y-%m-%d %T" ${ACCESSRIGHTS:+-pug}

# write a smaller version with only file size information
#call_to_tree "simple"   "$OUTDIR/$OUTNAME-tree" tree -ax -n --du -h --dirsfirst

echo "Done."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]