[gentoo-dev] RFC: Support for sys-kernel/dkms via a dkms.eclass in Gentoo

[gentoo-dev] RFC: Support for sys-kernel/dkms via a dkms.eclass in Gentoo

[Nowa Ammerlaan]

Dear all,

This concept has turned out to be more controversial then I anticipated, 
so I think it warrants broader discussion here on the mailing list. Let 
me first explain some background information so everyone knows what we 
are talking about, and then afterwards concretely discuss my idea for 
implementing it in Gentoo.

The Dynamic Kernel Module System (DKMS, sys-kernel/dkms) is a 
distro-agnostic framework for managing out-of-tree kernel modules. 
You'll find it in most Linux distributions, though users usually do not 
have to interact with it directly. In Gentoo this is packaged as 
sys-kernel/dkms, though unlike most other package managers Portage 
currently has no support for it. As such the usefulness of this package 
is limited to some edge cases.

The operating principle of DKMS is simple. The package manager installs 
the sources for one or more kernel modules to 
/usr/src/$PACKAGE_NAME-$PACKAGE_VERSION/, and installs a configuration 
file (dkms.conf) to the root of this source directory. The configuration 
file describes which kernel modules are contained in this source 
directory, how to build them, how to install them, as well as some metadata.

Then the dkms command line utility can be used to register (dkms add ... 
) these kernel sources in the local dkms tree (/var/lib/dkms). After 
this the contained kernel modules can be built (dkms build ...) and 
installed (dkms install ...). Compared to the alternative approach where 
the package manager installs (pre-)built kernel modules directly to 
/lib/modules/KV_FULL, the DKMS approach comes with several advantages:
- Via init system service, kernel modules can be automatically built and 
installed when the system boots into some new kernel.
- Via kernel-install hook, kernel modules can be automatically built and 
installed for multiple kernels in one go (kernel-install add-all).
- The above makes it possible to propagate updates of a kernel module to 
an arbitrary number of kernels (and accompanying initramfs) without user 
intervention.

End result is a "just works" experience when it comes to out-of-tree 
kernel modules. This is something that in my opinion is currently sorely 
lacking in Gentoo. Instead we currently have to fiddle with symlinks 
(eselect kernel ...) or environment variables (KERNEL_DIR), and then 
iteratively invoke the package manager multiple times. This is slow and 
difficult to automate. And yes we could in theory create some solution 
using multibuild.eclass and a KERNEL_TARGETS (similar to the 
PYTHON_TARGETS). But this I think will be messy since the kernel has 
significantly more slots to target then e.g. python, which means these 
targets would have to be updated on every minor kernel version bump. Not 
to mention that we would also have to support a wide variety of kernel 
(source) packages. And then still we would not have the same level of 
"just works" experience compared to dynamically compiling missing 
modules while the system is booting, this we cannot reasonably achieve 
using the package manager.

For this reason I propose to not invent our own new solution to improve 
the situation, but instead adopt the proven industry standard solution: 
DKMS.

I've opened a Pull Request (https://github.com/gentoo/gentoo/pull/40704) 
  with a working implementation of a dkms.eclass. The idea of the eclass 
is relatively simple. We inherit linux-mod-r1.eclass so we don't have to 
duplicate most of the setup and preparation logic. Then in the compile 
phase we create a DKMS configuration file using the linux-mod-r1 modlist 
and modargs defined by the ebuild, alternatively an existing upstream 
DKMS configuration file may be used. The configuration file will contain 
the users compiler etc. preference, *FLAGS and MAKEOPTS (i.e. 
linux-mod-r1's MODULES_MAKEARGS), this ensures that we do not lose any 
user customization in the process. Then in the install phase we find all 
dkms.conf files and install those and the associated sources to 
/usr/src/. Finally in the pkg_postinst and pkg_prerm phase we call dkms 
to add/build/install/uninstall/unbuild/remove the contained kernel 
modules. For convenience we also add a pkg_config function to call dkms 
again to rebuild and reinstall the kernel modules.

Note that all of this is toggled by a new "dkms" USE flag, if this flag 
is disabled then modules are built and installed as usual via 
linux-mod-r1.eclass. This is intended to add a new option to Gentoo, not 
to replace linux-mod-r1.eclass. A downside of this is that the existing 
USE=modules-compress and USE=modules-sign are ineffective when USE=dkms 
is enabled. Though DKMS does support compressing and signing kernel 
modules, which is configured via /etc/dkms/framework.conf. The default 
is to compress kernel modules when the kernel modules currently in 
/lib/modules/KV_FULL are also compressed, and to sign kernel modules 
whenever it is able to do so (note a fix for DKMS to automatically 
respect our MODULES_SIGN_KEY and MODULES_SIGN_CERT has already been 
merged upstream).

Now I already know Sam and Ionen are not enthusiastic about this idea, 
they are concerned about having to support two paths to achieve similar 
ends and are not (yet) convinced of the added value of the DKMS pathway. 
But I'd love to hear more opinions. If really everyone hates this then 
I'll drop it, but currently I am still convinced this is useful and 
worth the effort. Because DKMS support adds a "just works" level of 
experience that we cannot reasonably achieve with the package manager 
and because it is the industry standard solution and thus is often 
tested and supported by kernel modules upstream.

Please find the proposed eclass, as well as implementation in a bunch of 
ebuilds, in my PR: https://github.com/gentoo/gentoo/pull/40704
I could send it to the mailinglist, but it's 41 commits large and I 
don't want to spam everyone.

Best regards,
Nowa

Re: [gentoo-dev] RFC: Support for sys-kernel/dkms via a dkms.eclass in Gentoo

[alexey+gentoo]

вторник, 4 марта 2025 г., 3:50 ПП, Nowa Ammerlaan <nowa@gentoo.org> написал:

> 
> 
> Dear all,
> 
> This concept has turned out to be more controversial then I anticipated,
> so I think it warrants broader discussion here on the mailing list. Let
> me first explain some background information so everyone knows what we
> are talking about, and then afterwards concretely discuss my idea for
> implementing it in Gentoo.
> 
> The Dynamic Kernel Module System (DKMS, sys-kernel/dkms) is a
> distro-agnostic framework for managing out-of-tree kernel modules.
> You'll find it in most Linux distributions, though users usually do not
> have to interact with it directly. In Gentoo this is packaged as
> sys-kernel/dkms, though unlike most other package managers Portage
> currently has no support for it. As such the usefulness of this package
> is limited to some edge cases.
> 
> The operating principle of DKMS is simple. The package manager installs
> the sources for one or more kernel modules to
> /usr/src/$PACKAGE_NAME-$PACKAGE_VERSION/, and installs a configuration
> file (dkms.conf) to the root of this source directory. The configuration
> file describes which kernel modules are contained in this source
> directory, how to build them, how to install them, as well as some metadata.
> 
> Then the dkms command line utility can be used to register (dkms add ...
> ) these kernel sources in the local dkms tree (/var/lib/dkms). After
> this the contained kernel modules can be built (dkms build ...) and
> installed (dkms install ...). Compared to the alternative approach where
> the package manager installs (pre-)built kernel modules directly to
> /lib/modules/KV_FULL, the DKMS approach comes with several advantages:
> - Via init system service, kernel modules can be automatically built and
> installed when the system boots into some new kernel.
> - Via kernel-install hook, kernel modules can be automatically built and
> installed for multiple kernels in one go (kernel-install add-all).
> - The above makes it possible to propagate updates of a kernel module to
> an arbitrary number of kernels (and accompanying initramfs) without user
> intervention.
> 
> End result is a "just works" experience when it comes to out-of-tree
> kernel modules. This is something that in my opinion is currently sorely
> lacking in Gentoo. Instead we currently have to fiddle with symlinks
> (eselect kernel ...) or environment variables (KERNEL_DIR), and then
> iteratively invoke the package manager multiple times. This is slow and
> difficult to automate. And yes we could in theory create some solution
> using multibuild.eclass and a KERNEL_TARGETS (similar to the
> PYTHON_TARGETS). But this I think will be messy since the kernel has
> significantly more slots to target then e.g. python, which means these
> targets would have to be updated on every minor kernel version bump. Not
> to mention that we would also have to support a wide variety of kernel
> (source) packages. And then still we would not have the same level of
> "just works" experience compared to dynamically compiling missing
> modules while the system is booting, this we cannot reasonably achieve
> using the package manager.
> 
> For this reason I propose to not invent our own new solution to improve
> the situation, but instead adopt the proven industry standard solution:
> DKMS.
> 
> I've opened a Pull Request (https://github.com/gentoo/gentoo/pull/40704)
> with a working implementation of a dkms.eclass. The idea of the eclass
> is relatively simple. We inherit linux-mod-r1.eclass so we don't have to
> duplicate most of the setup and preparation logic. Then in the compile
> phase we create a DKMS configuration file using the linux-mod-r1 modlist
> and modargs defined by the ebuild, alternatively an existing upstream
> DKMS configuration file may be used. The configuration file will contain
> the users compiler etc. preference, *FLAGS and MAKEOPTS (i.e.
> linux-mod-r1's MODULES_MAKEARGS), this ensures that we do not lose any
> user customization in the process. Then in the install phase we find all
> dkms.conf files and install those and the associated sources to
> /usr/src/. Finally in the pkg_postinst and pkg_prerm phase we call dkms
> to add/build/install/uninstall/unbuild/remove the contained kernel
> modules. For convenience we also add a pkg_config function to call dkms
> again to rebuild and reinstall the kernel modules.
> 
> Note that all of this is toggled by a new "dkms" USE flag, if this flag
> is disabled then modules are built and installed as usual via
> linux-mod-r1.eclass. This is intended to add a new option to Gentoo, not
> to replace linux-mod-r1.eclass. A downside of this is that the existing
> USE=modules-compress and USE=modules-sign are ineffective when USE=dkms
> is enabled. Though DKMS does support compressing and signing kernel
> modules, which is configured via /etc/dkms/framework.conf. The default
> is to compress kernel modules when the kernel modules currently in
> /lib/modules/KV_FULL are also compressed, and to sign kernel modules
> whenever it is able to do so (note a fix for DKMS to automatically
> respect our MODULES_SIGN_KEY and MODULES_SIGN_CERT has already been
> merged upstream).
> 
> Now I already know Sam and Ionen are not enthusiastic about this idea,
> they are concerned about having to support two paths to achieve similar
> ends and are not (yet) convinced of the added value of the DKMS pathway.
> But I'd love to hear more opinions. If really everyone hates this then
> I'll drop it, but currently I am still convinced this is useful and
> worth the effort. Because DKMS support adds a "just works" level of
> experience that we cannot reasonably achieve with the package manager
> and because it is the industry standard solution and thus is often
> tested and supported by kernel modules upstream.
> 
> Please find the proposed eclass, as well as implementation in a bunch of
> ebuilds, in my PR: https://github.com/gentoo/gentoo/pull/40704
> I could send it to the mailinglist, but it's 41 commits large and I
> don't want to spam everyone.
> 
> Best regards,
> Nowa
> 

Thanks for this!
DKMS is the only feature which I'm still missing from times when I was using Debian/Ubuntu. The @module-rebuild is not as good.

Re: [gentoo-dev] RFC: Support for sys-kernel/dkms via a dkms.eclass in Gentoo

[Nowa Ammerlaan]

Dear all,

Because of the continued opposition to implementing DKMS support in the 
replies to my patchset. And because I feel the discussion is going 
nowhere. I have decided to (for now) move the dkms.eclass and the 
modified ebuilds to my ::natinst overlay.

This is obviously not what I would have preferred since it is bound to 
fall behind ::gentoo and I'm not going to actively sync packages that I 
do not use myself very frequently (but Pull Requests are of course very 
welcome!).

Nonetheless, folks who would like to give this a try are very welcome to 
"eselect repository enable natinst" and enable the "dkms" USE flag. 
Please report any issues you find to me at 
github.com/Nowa-Ammerlaan/natinst/issues

Some documentation is on the wiki: wiki.gentoo.org/wiki/DKMS

Best regards,
Nowa


On 04/03/2025 15:50, Nowa Ammerlaan wrote:
> Dear all,
> 
> This concept has turned out to be more controversial then I anticipated, 
> so I think it warrants broader discussion here on the mailing list. Let 
> me first explain some background information so everyone knows what we 
> are talking about, and then afterwards concretely discuss my idea for 
> implementing it in Gentoo.
> 
> The Dynamic Kernel Module System (DKMS, sys-kernel/dkms) is a distro- 
> agnostic framework for managing out-of-tree kernel modules. You'll find 
> it in most Linux distributions, though users usually do not have to 
> interact with it directly. In Gentoo this is packaged as sys-kernel/ 
> dkms, though unlike most other package managers Portage currently has no 
> support for it. As such the usefulness of this package is limited to 
> some edge cases.
> 
> The operating principle of DKMS is simple. The package manager installs 
> the sources for one or more kernel modules to /usr/src/$PACKAGE_NAME- 
> $PACKAGE_VERSION/, and installs a configuration file (dkms.conf) to the 
> root of this source directory. The configuration file describes which 
> kernel modules are contained in this source directory, how to build 
> them, how to install them, as well as some metadata.
> 
> Then the dkms command line utility can be used to register (dkms 
> add ... ) these kernel sources in the local dkms tree (/var/lib/dkms). 
> After this the contained kernel modules can be built (dkms build ...) 
> and installed (dkms install ...). Compared to the alternative approach 
> where the package manager installs (pre-)built kernel modules directly 
> to /lib/modules/KV_FULL, the DKMS approach comes with several advantages:
> - Via init system service, kernel modules can be automatically built and 
> installed when the system boots into some new kernel.
> - Via kernel-install hook, kernel modules can be automatically built and 
> installed for multiple kernels in one go (kernel-install add-all).
> - The above makes it possible to propagate updates of a kernel module to 
> an arbitrary number of kernels (and accompanying initramfs) without user 
> intervention.
> 
> End result is a "just works" experience when it comes to out-of-tree 
> kernel modules. This is something that in my opinion is currently sorely 
> lacking in Gentoo. Instead we currently have to fiddle with symlinks 
> (eselect kernel ...) or environment variables (KERNEL_DIR), and then 
> iteratively invoke the package manager multiple times. This is slow and 
> difficult to automate. And yes we could in theory create some solution 
> using multibuild.eclass and a KERNEL_TARGETS (similar to the 
> PYTHON_TARGETS). But this I think will be messy since the kernel has 
> significantly more slots to target then e.g. python, which means these 
> targets would have to be updated on every minor kernel version bump. Not 
> to mention that we would also have to support a wide variety of kernel 
> (source) packages. And then still we would not have the same level of 
> "just works" experience compared to dynamically compiling missing 
> modules while the system is booting, this we cannot reasonably achieve 
> using the package manager.
> 
> For this reason I propose to not invent our own new solution to improve 
> the situation, but instead adopt the proven industry standard solution: 
> DKMS.
> 
> I've opened a Pull Request (https://github.com/gentoo/gentoo/pull/40704) 
>   with a working implementation of a dkms.eclass. The idea of the eclass 
> is relatively simple. We inherit linux-mod-r1.eclass so we don't have to 
> duplicate most of the setup and preparation logic. Then in the compile 
> phase we create a DKMS configuration file using the linux-mod-r1 modlist 
> and modargs defined by the ebuild, alternatively an existing upstream 
> DKMS configuration file may be used. The configuration file will contain 
> the users compiler etc. preference, *FLAGS and MAKEOPTS (i.e. linux-mod- 
> r1's MODULES_MAKEARGS), this ensures that we do not lose any user 
> customization in the process. Then in the install phase we find all 
> dkms.conf files and install those and the associated sources to /usr/ 
> src/. Finally in the pkg_postinst and pkg_prerm phase we call dkms to 
> add/build/install/uninstall/unbuild/remove the contained kernel modules. 
> For convenience we also add a pkg_config function to call dkms again to 
> rebuild and reinstall the kernel modules.
> 
> Note that all of this is toggled by a new "dkms" USE flag, if this flag 
> is disabled then modules are built and installed as usual via linux-mod- 
> r1.eclass. This is intended to add a new option to Gentoo, not to 
> replace linux-mod-r1.eclass. A downside of this is that the existing 
> USE=modules-compress and USE=modules-sign are ineffective when USE=dkms 
> is enabled. Though DKMS does support compressing and signing kernel 
> modules, which is configured via /etc/dkms/framework.conf. The default 
> is to compress kernel modules when the kernel modules currently in /lib/ 
> modules/KV_FULL are also compressed, and to sign kernel modules whenever 
> it is able to do so (note a fix for DKMS to automatically respect our 
> MODULES_SIGN_KEY and MODULES_SIGN_CERT has already been merged upstream).
> 
> Now I already know Sam and Ionen are not enthusiastic about this idea, 
> they are concerned about having to support two paths to achieve similar 
> ends and are not (yet) convinced of the added value of the DKMS pathway. 
> But I'd love to hear more opinions. If really everyone hates this then 
> I'll drop it, but currently I am still convinced this is useful and 
> worth the effort. Because DKMS support adds a "just works" level of 
> experience that we cannot reasonably achieve with the package manager 
> and because it is the industry standard solution and thus is often 
> tested and supported by kernel modules upstream.
> 
> Please find the proposed eclass, as well as implementation in a bunch of 
> ebuilds, in my PR: https://github.com/gentoo/gentoo/pull/40704
> I could send it to the mailinglist, but it's 41 commits large and I 
> don't want to spam everyone.
> 
> Best regards,
> Nowa
> 
> 
>