[gentoo-dev] Add Hooks to Eselect

[gentoo-dev] Add Hooks to Eselect

[Redjard]

From: Redjard <eselect.patches.gentoo@redjard.org>

Add Hooks to Eselect

For example for "eselect kernel list" the script 
/etc/eselect/hooks/kernel/list/pre
is called before the eselect acts, and 
/etc/eselect/hooks/kernel/list/post afterwards.
In the functions you can use and modify $params, the value of which the
do_<action> function of the respective select module is then called with.
   by Redjard 2023-08-18 (eselect.patches.gentoo@redjard.org)

https://discord.com/channels/249111029668249601/1140975737168478279

--- a/libs/core.bash.in
+++ b/libs/core.bash.in
@@ -20,10 +20,22 @@
-check_do() {
+check_do() {
  	local function=$1
-	shift
+	shift; params="$@"
  	if is_function "${function}" ; then
-		${function} "$@"
+		run_hook "${ESELECT_MODULE_NAME}" "${function##do_}" pre
+		${function} "${params}"
+		run_hook "${ESELECT_MODULE_NAME}" "${function##do_}" post
  	else
  		die "No function ${function}"
  	fi
  }

+# Redjard patch: call hooks
+run_hook() {
+	local action=$1
+	local subaction=$2
+	local script=$3
+	if [[ -e "/etc/eselect/hooks/${action}/${subaction}/${script}" ]];then
+		source "/etc/eselect/hooks/${action}/${subaction}/${script}"
+	fi
+}
+

Re: [gentoo-dev] Add Hooks to Eselect

[Michał Górny]

On Fri, 2023-08-18 at 21:38 +0200, Redjard wrote:
> From: Redjard <eselect.patches.gentoo@redjard.org>
> 
> Add Hooks to Eselect
> 
> For example for "eselect kernel list" the script 
> /etc/eselect/hooks/kernel/list/pre
> is called before the eselect acts, and 
> /etc/eselect/hooks/kernel/list/post afterwards.
> In the functions you can use and modify $params, the value of which the
> do_<action> function of the respective select module is then called with.
>    by Redjard 2023-08-18 (eselect.patches.gentoo@redjard.org)
> 
> https://discord.com/channels/249111029668249601/1140975737168478279
> 

You have to provide the rationale here.  You can't expect people to sign
up for some shady third-party service to read it.

-- 
Best regards,
Michał Górny

[gentoo-dev] Re: Add Hooks to Eselect

[Duncan]

Redjard posted on Fri, 18 Aug 2023 21:38:37 +0200 as excerpted:

> From: Redjard <eselect.patches.gentoo@redjard.org>
> 
> Add Hooks to Eselect
> 
> For example for "eselect kernel list" the script
> /etc/eselect/hooks/kernel/list/pre is called before the eselect acts,
> and /etc/eselect/hooks/kernel/list/post afterwards.

Suggestion: A more flexible approach would make pre/post directories, such 
that "any" file therein (exceptions for backups, etc) would be sourced.

In run_hook something like (as written assumes bashisms OK, didn't verify 
if that's valid for eselect or not):

local action=$1
local subaction=$2
local hookscriptdir=$3

for $hookfile in
	/etc/eselect/hooks/${action}/${subaction}/${hookscriptdir}/*
do [[
		# maybe skip either the executable or README test?
		# or perhaps only match *.sh filenames instead
		# to reflect the in-shell sourcing?
		-x $hookfile &&
		$hookfile == ${hookfile#README} &&
		$hookfile == ${hookfile#.} &&
		$hookfile == ${hookfile%.bak} &&
		$hookfile == ${hookfile%\~}
	]] && source "$hookfile"
done


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Add Hooks to Eselect

[Redjard]

On 2023-08-19 06:58:10, Michał Górny wrote:
> You have to provide the rationale here.  You can't expect people to sign
> up for some shady third-party service to read it.
My bad, not much of value is locked in the proprietary service but I 
should have stated as such.
To roughly summarize, I was asking for a method to hook into eselect, to 
modify the behavior of eselect kernel set.
I was pointed in the direction of a user patch by konsolebox, and 
consequently wrote the patch.
I provided an example for using the patched hooks, which I will repeat 
below.


 >>> /etc/eselect/hooks/kernel/set/pre:
if [ "$EUID" -ne 0 ]; then die "run as root please"; fi

local -n target=params

if [ -z "$target" ]; then
     target=`ls -1v /usr/src/ | grep '^linux-.*-gentoo$' | tail -n1`
else
     if [ `echo "$target" | grep -P "^\d+(\.\d+){2}$"` ]; then
         target=linux-"$1"-gentoo
     elif [ `echo "${target%/}" | grep -P 
"^linux-\d+(\.\d+){2}-gentoo$"` ]; then
         target="${target%/}"
     fi
fi

if [ "$target" == `basename "$(readlink /usr/src/linux)"` ]; then
     die "kernel $target already selected"
fi

cp /usr/src/linux/.config /tmp/redjard_linux_kernel_eselect_config

echo "selecting kernel $target"


 >>> /etc/eselect/hooks/kernel/set/post:
# (re)generate /etc/portage/sets/active_kernels which selects the 
active, the installed, and the selected kernel
/opt/startup/lock-active-kernel-versions

# stop if .config is already present,
if [[ -e /usr/src/linux/.config ]];then
         die "Warning: .config already exists. Stopping post hook 
execution, you're on your own"
fi

echo "migrating .config"
mv -n /tmp/redjard_linux_kernel_eselect_config /usr/src/linux/.config

oldfile=`sha512sum /usr/src/linux/.config`

#echo "use  make oldconfig  to migrate the kernel configuration"
echo "running  make oldconfig  to migrate .config"
(cd /usr/src/linux && make oldconfig)

if [[ "$oldfile" != `sha512sum /usr/src/linux/.config` ]]; then
         /opt/kernel/backup_config
fi

echo "use  kernel-configure  to edit the kernel configuration"
echo "use  kernel-make  or  kernel-install  to build and install the 
configured kernel"

Re: [gentoo-dev] Add Hooks to Eselect

[Ulrich Mueller]

>>>>> On Mon, 21 Aug 2023, Redjard  wrote:

> To roughly summarize, I was asking for a method to hook into eselect,
> to modify the behavior of eselect kernel set.
> I was pointed in the direction of a user patch by konsolebox, and
> consequently wrote the patch.

In addition to user patches, you can also put your own modules in the
${HOME}/.eselect/modules/ directory.

For example, you could either copy kernel.eselect to there and modify
it. Or, you could have a mykernel.eselect module, along these lines:

do_set() {
	... # execute "pre" stuff
	do_action kernel set "$@"
	... # execute "post" stuff
}

The unchanged subactions would be trivial functions like this:

do_list() { do_action kernel list "$@"; }

> I provided an example for using the patched hooks, which I will repeat
> below.

Sorry, but I don't see much incentive for adding such a hook mechanism.
If there was, it would have been suggested previously since eselect was
created in 2005. Also, by design, eselect itself doesn't rely on any
configuration in /etc so this would be a somewhat intrusive change.

As a side note, your previously posted patch wouldn't work as-is:

>>> -check_do() {
>>> +check_do() {
>>>  	local function=$1
>>> -	shift
>>> +	shift; params="$@"
>>>  	if is_function "${function}" ; then
>>> -		${function} "$@"
>>> +		run_hook "${ESELECT_MODULE_NAME}" "${function##do_}" pre
>>> +		${function} "${params}"

Using a scalar variable instead of "$@" (which is an array) would break
quite a few modules.

>>> +		run_hook "${ESELECT_MODULE_NAME}" "${function##do_}" post
>>>  	else
>>>  		die "No function ${function}"
>>>  	fi
>>>  }

Ulrich

Re: [gentoo-dev] Add Hooks to Eselect

[Redjard]

On 2023-08-21 12:00:18, Ulrich Mueller wrote:
> In addition to user patches, you can also put your own modules in the
> ${HOME}/.eselect/modules/ directory.
> For example, you could either copy kernel.eselect to there and modify
> it. Or, you could have a mykernel.eselect module, along these lines:

I did consider doing so, however with that approach my effective fork of
the kernel.eselect module would get continually outdated, while a
wrapper module requires a different command and isn't very expressive on
what it is, making future maintenance harder. The wrapper would also be
a lot of boilerplate, having to mention all actions, not just modified
ones. Both solutions didn't feel right to me.
I expect to use and modify eselect more in the future, and want an 
approach that scales well while being low maintenance. A patch is the
correct solution for me, whether it gets upstreamed or not.

> Also, by design, eselect itself doesn't rely on any configuration in
> /etc so this would be a somewhat intrusive change.

 > # l /etc/eselect/
 > total 4.0K
 > drwxr-xr-x 1 root root   48 2023-08-15 19:59:07 .
 > drwxr-xr-x 1 root root 3.0K 2023-08-21 12:07:30 ..
 > drwxrwxr-x 1 root root   12 2023-08-15 19:59:31 hooks
 > -rw-r--r-- 1 root root 1.6K 2023-05-19 16:57:34 repository.conf
 > drwxr-xr-x 1 root root  120 2023-08-10 03:09:50 wine

This is what my /etc/eselect dir looks like right now. I have hardly
used eselect in the past, and my system is fairly recent.
Given the apparent existing practice of putting eselect related elements
into /etc/eselect and that the hooks are entirely optional, I see no
difference to what has already been established.
Without knowledge of the feature, the directory doesn't exist and no
action is ever taken.
When someone hears of and intends to use the hooks, they create the
correct directory structure.


> As a side note, your previously posted patch wouldn't work as-is:
>> -		${function} "$@"
>> +		${function} "${params}"
> Using a scalar variable instead of "$@" (which is an array) would break
> quite a few modules.

Thanks, I did not know that. The updated patch below should handle those 
cases now.


--- a/libs/core.bash.in
+++ b/libs/core.bash.in
@@ -20,10 +20,24 @@
-check_do() {
+check_do() {
  	local function=$1
-	shift
+	shift; params=("${@}")
  	if is_function "${function}" ; then
+		run_hook "${ESELECT_MODULE_NAME}" "${function##do_}" pre
-		${function} "$@"
+		${function} "${params[@]}"
+		run_hook "${ESELECT_MODULE_NAME}" "${function##do_}" post
  	else
  		die "No function ${function}"
  	fi
  }

+# Redjard patch: call hooks
+run_hook() {
+	local action=$1
+	local subaction=$2
+	local hookscriptdir=$3
+	for hookfile in 
"/etc/eselect/hooks/${action}/${subaction}/${hookscriptdir}"/* ; do [[
+		`basename "$hookfile"` =~ ^[a-zA-Z0-9_-][a-zA-Z0-9._-]+$
+	]] &&
+		source "$hookfile"
+	done
+}
+

Re: [gentoo-dev] Add Hooks to Eselect

[Ulrich Mueller]

>>>>> On Mon, 21 Aug 2023, Redjard  wrote:

> I did consider doing so, however with that approach my effective fork
> of the kernel.eselect module would get continually outdated, while a
> wrapper module requires a different command [...]

Not necessarily. It could use the same name as the original module,
after adjusting the modules path:

ESELECT_MODULES_PATH=( "${ESELECT_DEFAULT_MODULES_PATH}" )

Still, I'd not recommend using the same name when its behaviour is
different from the original module.

Re: [gentoo-dev] Re: Add Hooks to Eselect

[konsolebox]

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

On Mon, Aug 21, 2023, 08:15 Duncan, <1i5t5.duncan@cox.net> wrote:

> Suggestion: A more flexible approach would make pre/post directories,
>

Or allow both file and directories.


> do [[
>                 # maybe skip either the executable or README test?
>                 # or perhaps only match *.sh filenames instead
>                 # to reflect the in-shell sourcing?
>                 -x $hookfile &&
>                 $hookfile == ${hookfile#README} &&
>                 $hookfile == ${hookfile#.} &&
>                 $hookfile == ${hookfile%.bak} &&
>                 $hookfile == ${hookfile%\~}
>         ]] && source "$hookfile"
> done
>

Just checking if file has +x is better.  Easier to disable scripts that
way.

[-- Attachment #2: Type: text/html, Size: 1460 bytes --]

Re: [gentoo-dev] Add Hooks to Eselect

[konsolebox]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Mon, Aug 21, 2023, 18:00 Ulrich Mueller, <ulm@gentoo.org> wrote:

> Sorry, but I don't see much incentive for adding such a hook mechanism.
>

This actually allows users to virtually extend an eselect module without
needing to fork it.  The things people can do are endless.  Even now I just
thought about using it to auto update Ruby shebangs in $HOME/bin in a
manner that stubs are consistent with existing user gems for the newly
selected version.  I wouldn't need to wait for Gentoo Ruby to change
anything.

If there was, it would have been suggested previously since eselect was
> created in 2005. Also, by design, eselect itself doesn't rely on any
> configuration in /etc so this would be a somewhat intrusive change.
>

Weak arguments in my opinion.  It promotes unnecessary restriction.

[-- Attachment #2: Type: text/html, Size: 1372 bytes --]

Re: [gentoo-dev] Add Hooks to Eselect

[Ulrich Mueller]

>>>>> On Mon, 21 Aug 2023, konsolebox  wrote:

> This actually allows users to virtually extend an eselect module without
> needing to fork it.  The things people can do are endless.

This isn't what eselect modules were designed for. They are specialised
tools that are supposed to do one thing, and one thing only. It's like
suggesting that a command like cat or mv would need configuration or
some extension mechanism.

Also, given that eselect modules are normally run as root and the damage
that can result from bugs, I'd rather keep things simple, instead of
introducing "endless" possibilities. This approach has worked very well
during the last 18 years.

(If anything, a hook mechanism would look very different from what was
proposed here. First, changing a low-level function like check_do() is
really a no-no, because this function is documented in the eselect
developer guide and third-party modules may rely on it. Second, instead
of executing a separate script for every hook, there would be a file
defining shell functions for all subactions of a given module. But I
haven't thought about any details, because as I said, I don't see much
incentive for such a thing.)

Re: [gentoo-dev] Add Hooks to Eselect

[Redjard]

On 2023-08-21 20:50:39, Ulrich Mueller wrote:

 > First, changing a low-level function like check_do() is really a no-no,
 > because this function is documented in the eselect developer guide and
 > third-party modules may rely on it.

The documentation, as far as I can tell, states only

 > check_do
 >   The check_do utility function checks that the first parameter is a
 >   function, and then calls it with any additional parameters as its
 >   arguments. If the function does not exist, die is called.
 >   Again, this is mostly internal.

The behavior here is not modified by adding the hooks. If anything the
implementation ensures that if one eselect action calls another one, hooks
still apply as they should.
I actually just noticed making $params a local variable works too, so that
further ensures nothing in behavior changes for anything that happens to 
use check_do, even if that is documented as being "mostly internal", and 
not having the "PUBLIC" that for example die() has in its comment.
Calling the function with local variable "${params[@]}" instead of "$@" is
not something that's even possible to detect in bash as far as I know,
neither inside the called function nor to something calling check_do.


 > [...] Given that eselect modules are normally run as root and the damage
 > that can result from bugs, I'd rather keep things simple, instead of
 > introducing "endless" possibilities. This approach has worked very well
 > during the last 18 years.

One of the nice things about hooks is their low profile. If you don't know
about them, don't create their directory, you don't have to care. It is
not just all the possibilities of their use, it is that the price for them
is very low, for those not using them and for the maintainers of eselect.
The actions eselect does normally require root, so naturally the things a
user might want to add would too. If they used a wrapper script instead,
they would most likely run that as root too.


 > Second, instead of executing a separate script for every hook, there
 > [should] be a file defining shell functions for all subactions of a
 > given module. But I haven't thought about any details, because as I
 > said, I don't see much incentive for such a thing.

I don't see any advantage to this. Spreading hooks out into their own
directories is a common implementation, that allows one to have multiple
independent hooks with their own version management or any other number
of differences. I have for example now split my elselect kernel set hook,
so that if I want to disable defaulting to the most recent kernel version
when no version is specified I can chmod -x
kernel/set/pre/defaultVersion while keeping the migration of .config.
 From experience I usually prefer cron.hourly over an entry in cron.d
and prefer that over a line in crontab. I realize me disliking crontab is
in part because I have been burned by heavy handed distros (looking at
you synology DSM) that like to overwrite changes in configuration files,
but even a well behaved distribution like gentoo still recently migrated
ssh_config and sshd_config to ssh_config.d and sshd_config.d.
I am open for arguments to the contrary but I am of the belief that a
directory approach where two independently written hooks can almost
certainly be merged on a file level without conflicts is the right
solution here and for hooks in general.


 > If there was, it would have been suggested previously
 > since eselect was created in 2005.

 > This isn't what eselect modules were designed for. They are specialised
 > tools that are supposed to do one thing, and one thing only. It's like
 > suggesting that a command like cat or mv would need configuration or
 > some extension mechanism.

It seems to me there might have been some drift in what eselect is.
As I have experienced it, eselect is a uniform interface to a variety of
"selection decisions" that otherwise would require learning a lot of
different quirks and implementations. When you call eselect, the action
it does internally might be simple or complex, change over time, or
otherwise be annoying to manually implement yourself. What cat or mv do is
predictable, uniform - eselect is not. If I want to modify what mv does
(outside of its flags) I use a different command, I assemble what I
intend to do from other smaller commands because replacing mv or calling
it from somewhere else is easy.
Eselect has a certain bulk to it, which from your position I would assume
it might not have had initially. Eselect modules have accumulated a lot of
quirks, specific behavior, some are truly complex looking through them now,
to the point where writing my own neither seems like something I want to do,
nor like something I should. Even the eselect kernel set command has an
entire page of sanity checks and niceties.
So we have a uniform interface to a set of actions too complex to want
to do with anything else, to me this feels more like a package manager than
a posix command, eselect to me always had the same "vibe" as emerge, cron,
or init systems.
So like with emerge, if I want to change behavior I don't rewrite emerge, or
what emerge does, I call it from a wrapper script. Then I have this uniform
interface, where if I want some tool I write "emerge -a tool" without even
thinking about it, only to then realize I forgot I should have called
install_tool or emerge_tool or whatever I named the script 4 years ago 
when I
wrote it. So I realize what I should have done long ago - that's right -
fork the entire tool into tool_redjard, to append my script to the install
procedure, then modify emerge to see tool as tool_redjard and install my
modified version. No, of course not, I use the post install hook that emerge
helpfully provides me, to keep the uniform interface of my "install
everything" tool while having my hooks in a format that I can rely on
working, because I am doing it in a way the developers of emerge are aware
of.
I just remembered another reason I used the hooks approach in my case is
because bash_completion still works. Those larger tools often also
accumulate external dependencies from all sides, so hooks called from inside
the tool is generally safer for modifying behavior than doing changes
underneath (modifying or wrapping an eselect module) or above (wrapping
the eselect command).


I'm sorry for this having gotten as long as it has
Redjard



--- a/libs/core.bash.in
+++ b/libs/core.bash.in
@@ -20,10 +20,28 @@
-check_do() {
+check_do() {
-	local function=$1
-	shift
+	local function=$1; shift
+	local params=("${@}")
  	if is_function "${function}" ; then
+		run_hook "${ESELECT_MODULE_NAME}" "${function##do_}" pre
-		${function} "$@"
+		${function} "${params[@]}"
+		run_hook "${ESELECT_MODULE_NAME}" "${function##do_}" post
  	else
  		die "No function ${function}"
  	fi
  }

+# run_hook action subaction type
+# Call optional user-provided bash files that can modify ${params[@]}
+run_hook() {
+	local action=$1
+	local subaction=$2
+	local type=$3
+	for hookfile in "/etc/eselect/hooks/${action}/${subaction}/${type}"/* 
; do [[
+		# run-parts checks for +x flag too, so this seems fair to me
+		-x $hookfile &&
+		# regex taken from gentoo version of run-parts
+		`basename "$hookfile"` =~ ^[a-zA-Z0-9_-][a-zA-Z0-9._-]+$
+	]] &&
+		source "$hookfile"
+	done
+}
+

Re: [gentoo-dev] Add Hooks to Eselect

[konsolebox]

[-- Attachment #1: Type: text/plain, Size: 2381 bytes --]

On Tue, Aug 22, 2023, 02:50 Ulrich Mueller, <ulm@gentoo.org> wrote:

> >>>>> On Mon, 21 Aug 2023, konsolebox  wrote:
>
> > This actually allows users to virtually extend an eselect module without
> > needing to fork it.  The things people can do are endless.
>
> This isn't what eselect modules were designed for. They are specialised
> tools that are supposed to do one thing, and one thing only.


So let's say modules weren't initially designed to be extensive.  What's
stopping those from becoming one now?  Tradition?

It's like
> suggesting that a command like cat or mv would need configuration or
> some extension mechanism.
>

Bad comparison and a stretch.

Also, given that eselect modules are normally run as root and the damage
> that can result from bugs, I'd rather keep things simple, instead of
> introducing "endless" possibilities. This approach has worked very well
> during the last 18 years.
>

And when I said unnecessary restriction this was it.  You're trying to
restrict what users can do in a free system like Gentoo because they might
damage their system which is ridiculous.

If you want to keep things simple for yourself don't touch
/etc/eselect/hooks.

And if you think third party code might abuse it, QA it.  Or maybe just
make eselect read /usr/local/etc/eselect/hooks instead.


> (If anything, a hook mechanism would look very different from what was
> proposed here. First, changing a low-level function like check_do() is
> really a no-no, because this function is documented in the eselect
> developer guide and third-party modules may rely on it. Second, instead
> of executing a separate script for every hook, there would be a file
> defining shell functions for all subactions of a given module. But I
> haven't thought about any details, because as I said, I don't see much
> incentive for such a thing.)
>

I just checked.  Check_do is currently documented to be only used
internally so no modules should use it but if you really think check_do
shouldn't be modified then the answer is simple.  Just make the function
that calls the hooks the wrapper instead.

As for declaring functions to represent as hooks yes I thought about that
but that's a terrible idea.  Not only is it hard to declare declaratively
it also taints eselect's environment.

Come think of it, hooks should be source'd from a subshell.  I.e., "(
source ... )".

>

[-- Attachment #2: Type: text/html, Size: 4026 bytes --]