[gentoo-dev] [PATCH v3 0/2] Add esed.eclass for modifying with checks for no-op

[gentoo-dev] [PATCH v3 0/2] Add esed.eclass for modifying with checks for no-op

[Ionen Wolkens]

Semi-new take on the eclass, also incorporating mgorny's erepl idea
making this more or less a toolbox for replacements with no-op checks
(still named esed.eclass, but could be reconsidered if seems better).

See @EXAMPLE in eclass for quick usage overview, and below for
nvidia-drivers ebuild change example.

Please try it in real ebuilds so can get a better idea of anything
lacking or potentially broken that tests/esed.sh doesn't pickup.

Anything is still up for debate/review, unlikely to be final
(if even added at all).

Updated version also available from github PR[1]

Changelog from v2:
- esed simplified to take only specific arguments
 (parsing can still be a bit jarring, but more linear/deterministic)
- remove esedexps, unneeded given esed now understands -e
- simplified enewsed by using cp and removing concat support
 (with the 3 above, is essentially a rewrite)
- add simple bash-based erepl / erepld / ereplp + enew variants
- convert esedfind to "efind" so it's usable by all functions
 (perhaps feel a bit out of place in this eclass now, albeit similar
  theme given dies if no-ops without files)
- no longer hide null byte warnings and indicate this eclass is
  not for binary files (very uncommon usage either way)
- several new/modified esed.sh tests, also now verifies die
  messages to ensure died in the right place

As an usage example, here's differences for nvidia-drivers ebuild
(these could of course be handled other ways / patched, but between
conditionals, rebasing, and basic things like __USER__, can also
be annoying to):

-sed 's/defined(CONFIG_DRM/defined(CONFIG_DRM_KMS_HELPER/g' \
-	-i kernel{,-module-source/kernel-open}/conftest.sh || die
+erepl 'defined(CONFIG_DRM' 'defined(CONFIG_DRM_KMS_HELPER' \
+	kernel{,-module-source/kernel-open}/conftest.sh

-sed 's/__USER__/nvpd/' \
-	nvidia-persistenced/init/systemd/nvidia-persistenced.service.template \
-	> "${T}"/nvidia-persistenced.service || die
+enewrepl __USER__ nvpd \
+	nvidia-persistenced/init/systemd/nvidia-persistenced.service.template \
+	"${T}"/nvidia-persistenced.service

-use !amd64 || sed -i "s|/usr|${EPREFIX}/opt|" \
-	systemd/system/nvidia-powerd.service || die
+use amd64 && erepl /usr "${EPREFIX}"/opt \
+	systemd/system/nvidia-powerd.service

-use !wayland || sed -i '/^#.*modeset=1$/s/^#//' "${T}"/nvidia.conf || die
+use wayland && ereplp modeset=1 '#' '' "${T}"/nvidia.conf
(almost made this one esed, but ereplp does the uncommenting job too --
 also double negation was to skip if/then given `false && true || die`
 would die, but is now unneeded without || die)

-use wayland || sed -i 's/ WAYLAND_LIB_install$//' \
-	nvidia-settings/src/Makefile || die
+use wayland || ereplp ^install: WAYLAND_LIB_install '' \
+	nvidia-settings/src/Makefile
(not essential but make it safer by checking for ^install: too)

... fairly simple cases so no real need for esed over erepl here.

With ESED_VERBOSE=1 exported, in the build.log there's:

 * ereplp ^install: WAYLAND_LIB_install  nvidia-settings/src/Makefile 
[snip]
-install: NVIDIA_SETTINGS_install NVIDIA_GTKLIB_install WAYLAND_LIB_install
+install: NVIDIA_SETTINGS_install NVIDIA_GTKLIB_install 
[snip]

And if I rewrite it as `ereplp ^instypo: WYALAND_LIB_isntall ''`:

 * ERROR: x11-drivers/nvidia-drivers-515.48.07::gentoo failed (prepare phase):
 *   no-op: ereplp ^instypo: WYALAND_LIB_isntall  nvidia-settings/src/Makefile

[1] https://github.com/gentoo/gentoo/pull/25662

Ionen Wolkens (2):
  esed.eclass: new eclass
  eclass/tests/esed.sh: tests for esed.eclass

 eclass/esed.eclass   | 265 +++++++++++++++++++++++++++++++++++++++++++
 eclass/tests/esed.sh | 263 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 528 insertions(+)
 create mode 100644 eclass/esed.eclass
 create mode 100755 eclass/tests/esed.sh

-- 
2.35.1

[gentoo-dev] [PATCH v3 1/2] esed.eclass: new eclass

[Ionen Wolkens]

Signed-off-by: Ionen Wolkens <ionen@gentoo.org>
---
 eclass/esed.eclass | 265 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 265 insertions(+)
 create mode 100644 eclass/esed.eclass

diff --git a/eclass/esed.eclass b/eclass/esed.eclass
new file mode 100644
index 00000000000..414daceaf8b
--- /dev/null
+++ b/eclass/esed.eclass
@@ -0,0 +1,265 @@
+# Copyright 2022 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: esed.eclass
+# @MAINTAINER:
+# Ionen Wolkens <ionen@gentoo.org>
+# @AUTHOR:
+# Ionen Wolkens <ionen@gentoo.org>
+# @SUPPORTED_EAPIS: 8
+# @BLURB: sed(1) and alike wrappers that die if did not modify any files
+# @EXAMPLE:
+#
+# @CODE
+# # sed(1) wrappers, die if no changes
+# esed s/a/b/ file.c # -i is default
+# enewsed s/a/b/ project.pc.in "${T}"/project.pc
+#
+# # bash-only simple fixed string alternatives, also die if no changes
+# erepl string replace file.c
+# ereplp ^match string replace file.c # like /^match/s:string:replace:g
+# erepld ^match file.c # deletes matching lines, like /^match/d
+# use prefix && enewreplp ^prefix= /usr "${EPREFIX}"/usr pn.pc.in pn.pc
+#
+# # find(1) wrapper that sees shell functions, dies if no files found
+# efind . -name '*.c' -erun esed s/a/b/ # dies if no files changed
+# efind . -name '*.c' -erun sed s/a/b/ # only dies if no files found
+# @CODE
+#
+# Migration notes: be wary of non-deterministic cases involving variables,
+# e.g. s|lib|$(get_libdir)|, s|-O3|${CFLAGS}|, or s|/usr|${EPREFIX}/usr|.
+# erepl/esed() die if these do nothing, like libdir being 'lib' on x86.
+# Either verify, keep sed(1), or ensure a change (extra space, @libdir@).
+#
+# Where possible, it is also good to consider if using patches is more
+# suitable to ensure adequate changes.  These functions are also unsafe
+# for binary files containing null bytes (erepl() will remove them).
+
+case ${EAPI} in
+	8) ;;
+	*) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
+esac
+
+if [[ ! ${_ESED_ECLASS} ]]; then
+_ESED_ECLASS=1
+
+# @ECLASS_VARIABLE: ESED_VERBOSE
+# @DEFAULT_UNSET
+# @USER_VARIABLE
+# @DESCRIPTION:
+# If set to a non-empty value, erepl/esed() and wrappers will use diff(1)
+# to display file differences.  Recommended for maintainers to easily
+# confirm the changes being made.
+
+# @FUNCTION: esed
+# @USAGE: [-E|-r|-n] [-e <expression>]... [--] <file>...
+# @DESCRIPTION:
+# sed(1) wrapper that dies if any of the expressions did not modify any files.
+# sed's -i/--in-place is forced, -e can be omitted if only one expression, and
+# arguments must be passed in the listed order with files last.  Each -e will
+# be a separate sed(1) call to evaluate changes of each.
+esed() {
+	(( ${#} >= 2 )) || die "too few arguments for ${_esed_cmd[0]:-${FUNCNAME[0]}}"
+
+	local endopts=false args=() contents=() exps=() files=()
+	local -i i
+	for ((i=1; i<=${#}; i++)); do
+		if [[ ${!i} =~ ^- ]] && ! ${endopts}; then
+			case ${!i} in
+				--) endopts=true ;;
+				-E|-n|-r) args+=( ${!i} ) ;;
+				-e)
+					i+=1
+					[[ ${!i} ]] || die "missing argument to -e"
+					exps+=( "${!i}" )
+					;;
+				*) die "unrecognized option for ${FUNCNAME[0]}" ;;
+			esac
+		elif (( ! ${#exps[@]} )); then
+			exps+=( "${!i}" ) # like sed, if no -e, first non-option is exp
+		else
+			[[ -f ${!i} ]] || die "not a file: ${!i}"
+			files+=( "${!i}" )
+			contents+=( "$(<"${!i}")" ) || die "failed reading: ${!i}"
+		fi
+	done
+	(( ${#files[@]} )) || die "no files in ${FUNCNAME[0]} arguments"
+
+	if [[ ${_esed_output} ]]; then
+		(( ${#files[@]} == 1 )) || die "${_esed_cmd[0]} needs exactly one input file"
+
+		# swap file for output to simplify sequential sed'ing
+		cp -- "${files[0]}" "${_esed_output}" || die
+		files[0]=${_esed_output}
+	fi
+
+	local changed exp newcontents sed
+	for exp in "${exps[@]}"; do
+		sed=( sed -i "${args[@]}" -e "${exp}" -- "${files[@]}" )
+		[[ ${ESED_VERBOSE} ]] && einfo "${sed[*]}"
+
+		"${sed[@]}" </dev/null || die "failed: ${sed[*]}"
+
+		changed=false
+		for ((i=0; i<${#files[@]}; i++)); do
+			newcontents=$(<"${files[i]}") || die "failed reading: ${files[i]}"
+
+			if [[ ${contents[i]} != "${newcontents}" ]]; then
+				changed=true
+
+				[[ ${ESED_VERBOSE} ]] || break
+
+				diff -u --color --label="${files[i]}"{,} \
+					<(echo "${contents[i]}") <(echo "${newcontents}")
+			fi
+		done
+
+		${changed} \
+			|| die "no-op: ${FUNCNAME[0]} ${*}${_esed_cmd[0]:+ (from: ${_esed_cmd[*]})}"
+	done
+}
+
+# @FUNCTION: enewsed
+# @USAGE: <esed-argument>... <output-file>
+# @DESCRIPTION:
+# esed() wrapper to save the result to <output-file>.  Intended to replace
+# ``sed ... input > output`` given esed() does not support stdin/out.
+enewsed() {
+	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
+	local _esed_output=${*: -1:1}
+	esed "${@:1:${#}-1}"
+}
+
+# @FUNCTION: erepl
+# @USAGE: <string> <replacement> <file>...
+# @DESCRIPTION:
+# Do basic bash-only ``${<file>//"<string>"/<replacement>}`` per-line
+# replacement in files(s).  Dies if no changes were made.  Suggested over
+# sed(1) where possible for simplicity and avoiding issues with delimiters.
+# Warning: erepl-based functions strip null bytes, use for text only.
+erepl() {
+	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
+	ereplp '.*' "${@}"
+}
+
+# @FUNCTION: enewrepl
+# @USAGE: <erepl-argument>... <output-file>
+# @DESCRIPTION:
+# erepl() wrapper to save the result to <output-file>.
+enewrepl() {
+	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
+	local _esed_output=${*: -1:1}
+	ereplp '.*' "${@:1:${#}-1}"
+}
+
+# @FUNCTION: erepld
+# @USAGE: <line-pattern-match> <file>...
+# @DESCRIPTION:
+# Deletes lines in file(s) matching ``[[ ${line} =~ <pattern> ]]``.
+erepld() {
+	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
+	local _esed_repld=1
+	ereplp "${@}"
+}
+
+# @FUNCTION: enewrepld
+# @USAGE: <erepld-argument>... <output-file>
+# @DESCRIPTION:
+# erepl() wrapper to save the result to <output-file>.
+enewrepld() {
+	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
+	local _esed_output=${*: -1:1}
+	erepld "${@:1:${#}-1}"
+}
+
+# @FUNCTION: ereplp
+# @USAGE: <line-match-pattern> <string> <replacement> <file>...
+# @DESCRIPTION:
+# Like erepl() but replaces only on ``[[ ${line} =~ <pattern> ]]``.
+ereplp() {
+	local -i argsmin=$(( ${_esed_repld:-0}==1?2:4 ))
+	(( ${#} >= argsmin )) \
+		|| die "too few arguments for ${_esed_cmd[0]:-${FUNCNAME[0]}}"
+
+	[[ ! ${_esed_output} || ${#} -le ${argsmin} ]] \
+		|| die "${_esed_cmd[0]} needs exactly one input file"
+
+	local contents changed=false file line newcontents
+	for file in "${@:argsmin}"; do
+		mapfile contents < "${file}" || die
+		newcontents=()
+
+		for line in "${contents[@]}"; do
+			if [[ ${line} =~ ${1} ]]; then
+				if [[ ${_esed_repld} == 1 ]]; then
+					changed=true
+				else
+					newcontents+=( "${line//"${2}"/${3}}" )
+					[[ ${line} != "${newcontents[-1]}" ]] && changed=true
+				fi
+			else
+				newcontents+=( "${line}" )
+			fi
+		done
+		printf %s "${newcontents[@]}" > "${_esed_output:-${file}}" || die
+
+		if [[ ${ESED_VERBOSE} ]]; then
+			einfo "${FUNCNAME[0]} ${*:1:argsmin-1} ${file} ${_esed_output:+(to ${_esed_output})}"
+			diff -u --color --label="${file}" --label="${_esed_output:-${file}}" \
+				<(printf %s "${contents[@]}") <(printf %s "${newcontents[@]}")
+		fi
+	done
+
+	${changed} || die "no-op: ${_esed_cmd[*]:-${FUNCNAME[0]} ${*}}"
+}
+
+# @FUNCTION: enewreplp
+# @USAGE: <ereplp-argument>... <output-file>
+# @DESCRIPTION:
+# ereplp() wrapper to save the result to <output-file>.
+enewreplp() {
+	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
+	local _esed_output=${*: -1:1}
+	ereplp "${@:1:${#}-1}"
+}
+
+# @FUNCTION: efind
+# @USAGE: <find-argument>... -erun <command> <argument>...
+# @DESCRIPTION:
+# find(1) wrapper that dies if no files were found.  <command> can be a shell
+# function, e.g. ``efind ... -erun erepl /usr /opt``.  -print0 is added to
+# find arguments, and found files to end of arguments (``{} +`` is unused).
+# Found files must not exceed args limits.  Use is discouraged if files add
+# up to a large total size (50+MB), notably with slower erepl/esed().  Shell
+# functions called this way are expected to ``|| die`` themselves on error.
+efind() {
+	(( ${#} >= 3 )) || die "too few arguments for ${FUNCNAME[0]}"
+
+	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
+
+	local find=( find )
+	while (( ${#} )); do
+		if [[ ${1} =~ -erun ]]; then
+			shift
+			break
+		fi
+		find+=( "${1}" )
+		shift
+	done
+	find+=( -print0 )
+
+	local files
+	mapfile -d '' -t files < <("${find[@]}" || die "failed: ${find[*]}")
+
+	(( ${#files[@]} )) || die "no files from: ${find[*]}"
+	(( ${#} )) || die "missing -erun arguments for ${FUNCNAME[0]}"
+
+	# skip `|| die` for shell functions (should be handled internally)
+	if declare -f "${1}" >/dev/null; then
+		"${@}" "${files[@]}"
+	else
+		"${@}" "${files[@]}" || die "failed: ${*} ${files[*]}"
+	fi
+}
+
+fi
-- 
2.35.1

[gentoo-dev] [PATCH v3 2/2] eclass/tests/esed.sh: tests for esed.eclass

[Ionen Wolkens]

Bit sloppy, but should cover most of it.

Signed-off-by: Ionen Wolkens <ionen@gentoo.org>
---
 eclass/tests/esed.sh | 263 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 263 insertions(+)
 create mode 100755 eclass/tests/esed.sh

diff --git a/eclass/tests/esed.sh b/eclass/tests/esed.sh
new file mode 100755
index 00000000000..d9dfe699e42
--- /dev/null
+++ b/eclass/tests/esed.sh
@@ -0,0 +1,263 @@
+#!/usr/bin/env bash
+# Copyright 2022 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+EAPI=8
+source tests-common.sh || exit
+
+inherit esed
+
+cd "${WORKDIR:-/dev/null}" || exit
+
+tsddied=n
+tsddie() {
+	tsddied=y
+	tsddiemsg=${*}
+	echo "would die: ${tsddiemsg}" >&2
+	# silence some further errors given didn't actually die
+	sed() { :; }
+	die() { :; }
+}
+
+tsdbegin() {
+	tbegin "${1}"
+	tsddied=n
+	unset -f sed
+	die() { tsddie "${@}"; }
+}
+
+tsdend() {
+	if [[ ${1} == fatal* && ${tsddied} == n ]]; then
+		tend 127 "should have died"
+	elif [[ ${1} == fatal:* && ${tsddied} == y && ${tsddiemsg} != *"${1#fatal:}"* ]]; then
+		tend 128 "died as expected but die message does not match '*${1#fatal:}*'"
+	elif [[ ${1} == nonfatal && ${tsddied} == y ]]; then
+		tend 129 "should not have died"
+	else
+		tend ${2:-0} "something went wrong(tm)"
+	fi
+}
+
+tsdfile() {
+	local file
+	for file in "${@}"; do
+		if [[ ${file%:*} ]]; then
+			echo "${file%:*}" > "${file#*:}" || exit
+		elif [[ -e ${file#*:} ]]; then
+			rm -- "${file#*:}" || exit
+		fi
+	done
+}
+
+tsdcmp() {
+	local contents
+	contents=$(<"${1}") || exit
+	if [[ ${contents} != "${2}" ]]; then
+		echo "${FUNCNAME[0]}: '${contents}' != '${2}'"
+		return 1
+	fi
+}
+
+tsdbegin "esed: change on single file"
+tsdfile replace:file
+esed s/replace/new/ file
+tsdcmp file new
+tsdend nonfatal ${?}
+
+tsdbegin "esed: die due to no change on a single file"
+tsdfile keep:file
+esed s/replace/new/ file
+tsdcmp file keep
+tsdend fatal:no-op ${?}
+
+tsdbegin "esed: sequential changes"
+tsdfile replace1:file
+esed -e s/replace1/replace2/ -e s/replace2/new/ file
+tsdcmp file new
+tsdend nonfatal ${?}
+
+tsdbegin "esed: change on at least one of two files with ESED_VERBOSE=1"
+tsdfile keep:file1 replace:file2
+ESED_VERBOSE=1 esed s/replace/new/ file1 file2
+tsdcmp file1 keep &&
+	tsdcmp file2 new
+tsdend nonfatal ${?}
+
+tsdbegin "esed: die due to no change on two files with ESED_VERBOSE=1"
+tsdfile keep:file{1..2}
+ESED_VERBOSE=1 esed s/replace/new/ file1 file2
+tsdcmp file1 keep &&
+	tsdcmp file2 keep
+tsdend fatal:'no-op' ${?}
+
+tsdbegin "esed: -E/-n/-r arguments"
+tsdfile $'hide\nshow\nhide\nthis':file
+esed -E -n -r '/(show|this)/p' file
+tsdcmp file $'show\nthis'
+tsdend nonfatal ${?}
+
+tsdbegin "esed: die due to passing unrecognized -i"
+tsdfile replace:file
+esed -i s/replace/new/ file
+tsdend fatal:'unrecognized option'
+
+tsdbegin "esed: change files with one nicely named '-- -e'"
+tsdfile replace:"-- -e" replace:file
+esed -e s/replace/new/ file -- "-- -e"
+tsdcmp file new &&
+	tsdcmp "-- -e" new
+tsdend nonfatal ${?}
+
+tsdbegin "esed: die due to no files in arguments"
+esed -e s/replace/new/ -e s/replace/new/
+tsdend fatal:'no files in'
+
+tsdbegin "esed: die due to missing file"
+tsdfile :missing
+esed s/replace/new/ missing
+tsdend fatal:'not a file'
+
+tsdbegin "enewsed: change on a new file"
+tsdfile replace:file :newfile
+enewsed s/replace/new/ file newfile
+tsdcmp file replace &&
+	tsdcmp newfile new
+tsdend nonfatal ${?}
+
+tsdbegin "enewsed: die due to too many files"
+tsdfile replace:file1 replace:file2 :newfile
+enewsed s/replace/new/ file1 file2 newfile
+tsdend fatal:'exactly one input'
+
+tsdbegin "enewsed: die due to missing output file"
+tsdfile keep:file
+enewsed -e s/replace/new/ -e s/replace/new/ file
+tsdend fatal:'no files in'
+
+tsdbegin "enewsed: die due too few arguments beside files"
+tsdfile keep:file
+enewsed file newfile
+tsdend fatal:'too few arguments'
+
+tsdbegin "erepl: change on a single file"
+tsdfile 0000:file
+erepl 0 1 file
+tsdcmp file 1111
+tsdend nonfatal ${?}
+
+tsdbegin "erepl: die due to no change on a single file"
+tsdfile keep:file
+erepl missing new file
+tsdcmp file keep
+tsdend fatal:'no-op' ${?}
+
+tsdbegin "erepl: change on at least one of two files with ESED_VERBOSE=1"
+tsdfile keep:file1 replace:file2
+ESED_VERBOSE=1 erepl replace new file1 file2
+tsdcmp file1 keep &&
+	tsdcmp file2 new
+tsdend nonfatal ${?}
+
+tsdbegin "erepl: die due to no change on two files with ESED_VERBOSE=1"
+tsdfile keep:file{1..2}
+ESED_VERBOSE=1 erepl replace new file1 file2
+tsdcmp file1 keep &&
+	tsdcmp file2 keep
+tsdend fatal:'no-op' ${?}
+
+tsdbegin "erepl: change containing globs that should be ignored"
+tsdfile "*[0-9]{1,2}()":file
+erepl "*[0-9]{1,2}()" new file
+tsdcmp file new
+tsdend nonfatal ${?}
+
+tsdbegin "enewrepl: change on a new file"
+tsdfile replace:file :newfile
+enewrepl replace new file newfile
+tsdcmp file replace &&
+	tsdcmp newfile new
+tsdend nonfatal ${?}
+
+tsdbegin "enewrepl: die due to too many files"
+tsdfile replace:file1 replace:file2 :newfile
+enewrepl replace new file1 file2 newfile
+tsdend fatal:'exactly one input'
+
+tsdbegin "enewrepl: die due to missing output file"
+tsdfile keep:file
+enewrepl replace new file
+tsdend fatal:'too few arguments'
+
+tsdbegin "erepld: delete matching lines"
+tsdfile $'match\nkeep\nmatch':file
+erepld ^match file
+tsdcmp file keep
+tsdend nonfatal ${?}
+
+tsdbegin "enewrepld: delete matching lines"
+tsdfile $'match\nkeep\nmatch':file :newfile
+enewrepld ^match file newfile
+tsdcmp file $'match\nkeep\nmatch' &&
+	tsdcmp newfile keep
+tsdend nonfatal ${?}
+
+tsdbegin "ereplp: change matching lines"
+tsdfile $'match=0000\nkeep=0000\nmatch=0000':file
+ereplp ^match 0 1 file
+tsdcmp file $'match=1111\nkeep=0000\nmatch=1111'
+tsdend nonfatal ${?}
+
+tsdbegin "enewreplp: change matching lines"
+tsdfile $'match=0000\nkeep=0000\nmatch=0000':file :newfile
+enewreplp ^match 0 1 file newfile
+tsdcmp file $'match=0000\nkeep=0000\nmatch=0000' &&
+	tsdcmp newfile $'match=1111\nkeep=0000\nmatch=1111'
+tsdend nonfatal ${?}
+
+tsdbegin "efind+esed: change found files"
+tsdfile keep:file1.find1 replace:file2.find1
+efind . -type f -name '*.find1' -erun esed s/replace/new/
+tsdcmp file1.find1 keep &&
+	tsdcmp file2.find1 new
+tsdend nonfatal ${?}
+
+tsdbegin "efind+esed: die due no changes to found files"
+tsdfile keep:file1.find2 keep:file2.find2
+efind . -type f -name '*.find2' -erun esed s/replace/new/
+tsdcmp file1.find2 keep &&
+	tsdcmp file2.find2 keep
+tsdend fatal:'no-op' ${?}
+
+tsdbegin "efind: die due to bad command"
+tsdfile keep:file.find3
+efind . -type f -name '*.find3' -erun ./no-such-file
+tsdend fatal:'failed: ./no-such-file' ${?}
+
+tsdbegin "efind: don't die for shell functions"
+tsdfile keep:*.find4
+tsd-test-func() {
+	# would die in the function if there was an issue
+	echo "${FUNCNAME[*]} running for files: ${*@Q}"
+	local i=1
+	(( --i )) # uh-oh, this leaves "unintended" failure return value
+}
+efind . -type f -name '*.find4' -erun tsd-test-func
+tsdend nonfatal
+
+tsdbegin "efind: die due to missing -erun"
+tsdfile keep:ignore-perm.find5
+efind . -type f -name '*.find5'
+tsdend fatal:'missing -erun'
+
+tsdbegin "efind: die due to no files found"
+efind . -type f -name '*.missing' -erun echo -n
+tsdend fatal:'no files from'
+
+echo
+echo "note: any error messages after 'would die:' can be ignored"
+if [[ ${tret} == 0 ]]; then
+	echo "${0##*/} finished successfully"
+else
+	echo "${0##*/} failed (status: ${tret})"
+fi
+texit
-- 
2.35.1

Re: [gentoo-dev] [PATCH v3 0/2] Add esed.eclass for modifying with checks for no-op

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

On Sat, Jun 04, 2022 at 04:46:32PM -0400, Ionen Wolkens wrote:
> Updated version also available from github PR[1]
[snip]
> [1] https://github.com/gentoo/gentoo/pull/25662

Note received style changes already. Will keep it on GH for a
bit before considering a v4 for hopeful final review.

For simple style/nitpicks please do it on github so it's in sync,
but still interested in general feedback from here.
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH v3 1/2] esed.eclass: new eclass

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 10309 bytes --]

On Sat, Jun 04, 2022 at 16:46:33 -0400, Ionen Wolkens wrote:
> Signed-off-by: Ionen Wolkens <ionen@gentoo.org>
> ---
>  eclass/esed.eclass | 265 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 265 insertions(+)
>  create mode 100644 eclass/esed.eclass
> 
> diff --git a/eclass/esed.eclass b/eclass/esed.eclass
> new file mode 100644
> index 00000000000..414daceaf8b
> --- /dev/null
> +++ b/eclass/esed.eclass
> @@ -0,0 +1,265 @@
> +# Copyright 2022 Gentoo Authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: esed.eclass
> +# @MAINTAINER:
> +# Ionen Wolkens <ionen@gentoo.org>
> +# @AUTHOR:
> +# Ionen Wolkens <ionen@gentoo.org>
> +# @SUPPORTED_EAPIS: 8
> +# @BLURB: sed(1) and alike wrappers that die if did not modify any files
> +# @EXAMPLE:
> +#
> +# @CODE
> +# # sed(1) wrappers, die if no changes
> +# esed s/a/b/ file.c # -i is default
> +# enewsed s/a/b/ project.pc.in "${T}"/project.pc
> +#
> +# # bash-only simple fixed string alternatives, also die if no changes
> +# erepl string replace file.c
> +# ereplp ^match string replace file.c # like /^match/s:string:replace:g
> +# erepld ^match file.c # deletes matching lines, like /^match/d
> +# use prefix && enewreplp ^prefix= /usr "${EPREFIX}"/usr pn.pc.in pn.pc
> +#
> +# # find(1) wrapper that sees shell functions, dies if no files found
> +# efind . -name '*.c' -erun esed s/a/b/ # dies if no files changed
> +# efind . -name '*.c' -erun sed s/a/b/ # only dies if no files found
> +# @CODE
> +#
> +# Migration notes: be wary of non-deterministic cases involving variables,
> +# e.g. s|lib|$(get_libdir)|, s|-O3|${CFLAGS}|, or s|/usr|${EPREFIX}/usr|.
> +# erepl/esed() die if these do nothing, like libdir being 'lib' on x86.
> +# Either verify, keep sed(1), or ensure a change (extra space, @libdir@).
> +#
> +# Where possible, it is also good to consider if using patches is more
> +# suitable to ensure adequate changes.  These functions are also unsafe
> +# for binary files containing null bytes (erepl() will remove them).

Some way to test for NULL in the file before reading might be useful.
Possibly die if found? Although right now I can't think of a super
elegant and/or efficient way to do so without writing a simple external
helper.

> +
> +case ${EAPI} in
> +	8) ;;
> +	*) die "${ECLASS}: EAPI ${EAPI:-0} not supported" ;;
> +esac
> +
> +if [[ ! ${_ESED_ECLASS} ]]; then
> +_ESED_ECLASS=1
> +
> +# @ECLASS_VARIABLE: ESED_VERBOSE
> +# @DEFAULT_UNSET
> +# @USER_VARIABLE
> +# @DESCRIPTION:
> +# If set to a non-empty value, erepl/esed() and wrappers will use diff(1)
> +# to display file differences.  Recommended for maintainers to easily
> +# confirm the changes being made.
> +
> +# @FUNCTION: esed
> +# @USAGE: [-E|-r|-n] [-e <expression>]... [--] <file>...
> +# @DESCRIPTION:
> +# sed(1) wrapper that dies if any of the expressions did not modify any files.
> +# sed's -i/--in-place is forced, -e can be omitted if only one expression, and
> +# arguments must be passed in the listed order with files last.  Each -e will
> +# be a separate sed(1) call to evaluate changes of each.
> +esed() {
> +	(( ${#} >= 2 )) || die "too few arguments for ${_esed_cmd[0]:-${FUNCNAME[0]}}"
> +
> +	local endopts=false args=() contents=() exps=() files=()
> +	local -i i
> +	for ((i=1; i<=${#}; i++)); do
> +		if [[ ${!i} =~ ^- ]] && ! ${endopts}; then
> +			case ${!i} in
> +				--) endopts=true ;;
> +				-E|-n|-r) args+=( ${!i} ) ;;
> +				-e)
> +					i+=1
> +					[[ ${!i} ]] || die "missing argument to -e"
> +					exps+=( "${!i}" )
> +					;;
> +				*) die "unrecognized option for ${FUNCNAME[0]}" ;;
> +			esac
> +		elif (( ! ${#exps[@]} )); then
> +			exps+=( "${!i}" ) # like sed, if no -e, first non-option is exp
> +		else
> +			[[ -f ${!i} ]] || die "not a file: ${!i}"

Somewhere here might be a good place for the hypothetical:

            null_free ${!i} || die "file ${!i} contains NULL bytes"

> +			files+=( "${!i}" )
> +			contents+=( "$(<"${!i}")" ) || die "failed reading: ${!i}"
> +		fi
> +	done
> +	(( ${#files[@]} )) || die "no files in ${FUNCNAME[0]} arguments"
> +
> +	if [[ ${_esed_output} ]]; then
> +		(( ${#files[@]} == 1 )) || die "${_esed_cmd[0]} needs exactly one input file"
> +
> +		# swap file for output to simplify sequential sed'ing
> +		cp -- "${files[0]}" "${_esed_output}" || die
> +		files[0]=${_esed_output}
> +	fi
> +
> +	local changed exp newcontents sed
> +	for exp in "${exps[@]}"; do
> +		sed=( sed -i "${args[@]}" -e "${exp}" -- "${files[@]}" )
> +		[[ ${ESED_VERBOSE} ]] && einfo "${sed[*]}"
> +
> +		"${sed[@]}" </dev/null || die "failed: ${sed[*]}"
> +
> +		changed=false
> +		for ((i=0; i<${#files[@]}; i++)); do
> +			newcontents=$(<"${files[i]}") || die "failed reading: ${files[i]}"
> +
> +			if [[ ${contents[i]} != "${newcontents}" ]]; then
> +				changed=true
> +
> +				[[ ${ESED_VERBOSE} ]] || break
> +
> +				diff -u --color --label="${files[i]}"{,} \
> +					<(echo "${contents[i]}") <(echo "${newcontents}")
> +			fi
> +		done
> +
> +		${changed} \
> +			|| die "no-op: ${FUNCNAME[0]} ${*}${_esed_cmd[0]:+ (from: ${_esed_cmd[*]})}"
> +	done
> +}
> +
> +# @FUNCTION: enewsed
> +# @USAGE: <esed-argument>... <output-file>
> +# @DESCRIPTION:
> +# esed() wrapper to save the result to <output-file>.  Intended to replace
> +# ``sed ... input > output`` given esed() does not support stdin/out.
> +enewsed() {
> +	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
> +	local _esed_output=${*: -1:1}
> +	esed "${@:1:${#}-1}"
> +}
> +
> +# @FUNCTION: erepl
> +# @USAGE: <string> <replacement> <file>...
> +# @DESCRIPTION:
> +# Do basic bash-only ``${<file>//"<string>"/<replacement>}`` per-line
> +# replacement in files(s).  Dies if no changes were made.  Suggested over
> +# sed(1) where possible for simplicity and avoiding issues with delimiters.
> +# Warning: erepl-based functions strip null bytes, use for text only.
> +erepl() {
> +	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
> +	ereplp '.*' "${@}"
> +}
> +
> +# @FUNCTION: enewrepl
> +# @USAGE: <erepl-argument>... <output-file>
> +# @DESCRIPTION:
> +# erepl() wrapper to save the result to <output-file>.
> +enewrepl() {
> +	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
> +	local _esed_output=${*: -1:1}
> +	ereplp '.*' "${@:1:${#}-1}"
> +}
> +
> +# @FUNCTION: erepld
> +# @USAGE: <line-pattern-match> <file>...
> +# @DESCRIPTION:
> +# Deletes lines in file(s) matching ``[[ ${line} =~ <pattern> ]]``.
> +erepld() {
> +	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
> +	local _esed_repld=1
> +	ereplp "${@}"
> +}
> +
> +# @FUNCTION: enewrepld
> +# @USAGE: <erepld-argument>... <output-file>
> +# @DESCRIPTION:
> +# erepl() wrapper to save the result to <output-file>.
> +enewrepld() {
> +	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
> +	local _esed_output=${*: -1:1}
> +	erepld "${@:1:${#}-1}"
> +}
> +
> +# @FUNCTION: ereplp
> +# @USAGE: <line-match-pattern> <string> <replacement> <file>...
> +# @DESCRIPTION:
> +# Like erepl() but replaces only on ``[[ ${line} =~ <pattern> ]]``.
> +ereplp() {
> +	local -i argsmin=$(( ${_esed_repld:-0}==1?2:4 ))
> +	(( ${#} >= argsmin )) \
> +		|| die "too few arguments for ${_esed_cmd[0]:-${FUNCNAME[0]}}"
> +
> +	[[ ! ${_esed_output} || ${#} -le ${argsmin} ]] \
> +		|| die "${_esed_cmd[0]} needs exactly one input file"
> +
> +	local contents changed=false file line newcontents
> +	for file in "${@:argsmin}"; do

A good place to put the test might be here:

        null_free ${file} || die "file ${file} contains NULL bytes"

> +		mapfile contents < "${file}" || die
> +		newcontents=()
> +
> +		for line in "${contents[@]}"; do
> +			if [[ ${line} =~ ${1} ]]; then
> +				if [[ ${_esed_repld} == 1 ]]; then
> +					changed=true
> +				else
> +					newcontents+=( "${line//"${2}"/${3}}" )
> +					[[ ${line} != "${newcontents[-1]}" ]] && changed=true
> +				fi
> +			else
> +				newcontents+=( "${line}" )
> +			fi
> +		done
> +		printf %s "${newcontents[@]}" > "${_esed_output:-${file}}" || die
> +
> +		if [[ ${ESED_VERBOSE} ]]; then
> +			einfo "${FUNCNAME[0]} ${*:1:argsmin-1} ${file} ${_esed_output:+(to ${_esed_output})}"
> +			diff -u --color --label="${file}" --label="${_esed_output:-${file}}" \
> +				<(printf %s "${contents[@]}") <(printf %s "${newcontents[@]}")
> +		fi
> +	done
> +
> +	${changed} || die "no-op: ${_esed_cmd[*]:-${FUNCNAME[0]} ${*}}"
> +}
> +
> +# @FUNCTION: enewreplp
> +# @USAGE: <ereplp-argument>... <output-file>
> +# @DESCRIPTION:
> +# ereplp() wrapper to save the result to <output-file>.
> +enewreplp() {
> +	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
> +	local _esed_output=${*: -1:1}
> +	ereplp "${@:1:${#}-1}"
> +}
> +
> +# @FUNCTION: efind
> +# @USAGE: <find-argument>... -erun <command> <argument>...
> +# @DESCRIPTION:
> +# find(1) wrapper that dies if no files were found.  <command> can be a shell
> +# function, e.g. ``efind ... -erun erepl /usr /opt``.  -print0 is added to
> +# find arguments, and found files to end of arguments (``{} +`` is unused).
> +# Found files must not exceed args limits.  Use is discouraged if files add
> +# up to a large total size (50+MB), notably with slower erepl/esed().  Shell
> +# functions called this way are expected to ``|| die`` themselves on error.
> +efind() {
> +	(( ${#} >= 3 )) || die "too few arguments for ${FUNCNAME[0]}"
> +
> +	local _esed_cmd=( ${FUNCNAME[0]} "${@}" )
> +
> +	local find=( find )
> +	while (( ${#} )); do
> +		if [[ ${1} =~ -erun ]]; then
> +			shift
> +			break
> +		fi
> +		find+=( "${1}" )
> +		shift
> +	done
> +	find+=( -print0 )
> +
> +	local files
> +	mapfile -d '' -t files < <("${find[@]}" || die "failed: ${find[*]}")
> +
> +	(( ${#files[@]} )) || die "no files from: ${find[*]}"
> +	(( ${#} )) || die "missing -erun arguments for ${FUNCNAME[0]}"
> +
> +	# skip `|| die` for shell functions (should be handled internally)
> +	if declare -f "${1}" >/dev/null; then
> +		"${@}" "${files[@]}"
> +	else
> +		"${@}" "${files[@]}" || die "failed: ${*} ${files[*]}"
> +	fi
> +}
> +
> +fi
> -- 
> 2.35.1
> 
> 

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] [PATCH v3 1/2] esed.eclass: new eclass

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 1585 bytes --]

On Sun, Jun 05, 2022 at 14:42:55 -0500, Oskari Pirhonen wrote:
> On Sat, Jun 04, 2022 at 16:46:33 -0400, Ionen Wolkens wrote:
> > +# Where possible, it is also good to consider if using patches is more
> > +# suitable to ensure adequate changes.  These functions are also unsafe
> > +# for binary files containing null bytes (erepl() will remove them).
> 
> Some way to test for NULL in the file before reading might be useful.
> Possibly die if found? Although right now I can't think of a super
> elegant and/or efficient way to do so without writing a simple external
> helper.
> 

I came up with _an_ idea. It feels kinda hacky, goes against my previous
statement wrt running sed on binary files, but it seems to work based on
some quick tests:
    
    [ /tmp ]
    oskari@dj3ntoo λ null_free() {
    > [[ $(sed -ne '/\x00/{p;q}' "$1" | wc -c) -eq 0 ]]
    > }
    [ /tmp ]
    oskari@dj3ntoo λ for f in test*.dat; do
    > echo "${f}:"
    > hd $f
    > null_free $f || echo "has null"
    > done
    test1.dat:
    00000000  61 62 0a 63 64 00 65 66                           |ab.cd.ef|
    00000008
    has null
    test2.dat:
    00000000  67 68 0a 69 6a 0a 6b 6c                           |gh.ij.kl|
    00000008
    test3.dat:
    00000000  00 0a 31 32 0a 33 34                              |..12.34|
    00000007
    has null
    test4.dat:
    00000000  61 61 00 61 61 0a 62 62  00 62 62 0a 63 63 00 63  |aa.aa.bb.bb.cc.c|
    00000010  63                                                |c|
    00000011
    has null

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] [PATCH v3 0/2] Add esed.eclass for modifying with checks for no-op

[Florian Schmaus]

FWIW: In an idealistic, but naive attempt, I've submitted a patch to 
upstream sed that adds an --error-if-unmodifed option [1]. Fellow Gentoo 
developers where quick to point out in #gentoo-dev that this patch 
barley scratches the surface of the sphere where sed could mutate state. 
It appears that --error-if-unmodifed in sed can only be reliable 
implemented if --inplace is used, by basically mimicking what Ionen's 
esed class does/did.

This made me wonder if fixing this in sed is really the ideal solution. 
However, Ionen also closed the PR due to "having seconds thoughts" [2].

I guess this leaves us with qa-sed from app-portage/iwdevtools. Which is 
probably not the worst outcome.

- Flow

1: https://lists.gnu.org/archive/html/sed-devel/2022-07/msg00001.html
2: https://github.com/gentoo/gentoo/pull/25662#issuecomment-1165265901