[gentoo-dev] Guidance on adding kernel config checks to ebuilds

[gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Gilbert]

I'm looking to solicit opinions on when it is appropriate for an
ebuild to check for kernel config options using linux-info.eclass. I
don't think we have any guidelines documented, instead leaving it up
to the "common sense" of package maintainers.

Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
when running emerge, so we should do so only when there is a
compensating benefit. It doesn't make sense to check for kernel
options that are very commonly enabled. But what is "very common"?

An obvious example would be CONFIG_INET, which controls IPv4 support
in the kernel. It does not make sense to check for that in every
package that uses AF_INET sockets.

A less obvious example: a user filed a bug against net-misc/dhcpcd
today asking that we check for CONFIG_PACKET [1]. My first thought was
"why would you ever disable that?". The option description even says
"if unsure, say Y". However, I suppose it is technically possible to
run a Linux system with it disabled.

I think a reasonable rule of thumb would be to assume we can rely on
options that are enabled by "make defconfig". If the user chooses to
disable them, they are responsible for anything that breaks.

Thoughts?

[1] https://bugs.gentoo.org/815064

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Pagano]

On 9/27/21 12:10 PM, Mike Gilbert wrote:
> I'm looking to solicit opinions on when it is appropriate for an
> ebuild to check for kernel config options using linux-info.eclass. I
> don't think we have any guidelines documented, instead leaving it up
> to the "common sense" of package maintainers.
> 
> Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
> when running emerge, so we should do so only when there is a
> compensating benefit. It doesn't make sense to check for kernel
> options that are very commonly enabled. But what is "very common"?
> 
> An obvious example would be CONFIG_INET, which controls IPv4 support
> in the kernel. It does not make sense to check for that in every
> package that uses AF_INET sockets.
> 
> A less obvious example: a user filed a bug against net-misc/dhcpcd
> today asking that we check for CONFIG_PACKET [1]. My first thought was
> "why would you ever disable that?". The option description even says
> "if unsure, say Y". However, I suppose it is technically possible to
> run a Linux system with it disabled.
> 
> I think a reasonable rule of thumb would be to assume we can rely on
> options that are enabled by "make defconfig". If the user chooses to
> disable them, they are responsible for anything that breaks.
> 
> Thoughts?
> 
> [1] https://bugs.gentoo.org/815064
> 

The challenge I see is that these config checks head off bugs and issues without our intervention.

We as kernel maintainers depend on ebuild maintainers to check these things so they don't become "kernel bugs" to figure out.

> Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
> when running emerge, so we should do so only when there is a
> compensating benefit.

Is this a significant slowdown? Do you have any numbers?

[gentoo-dev] Re: Guidance on adding kernel config checks to ebuilds

[Luca Barbato]

On 27/09/21 18:10, Mike Gilbert wrote:
> I'm looking to solicit opinions on when it is appropriate for an
> ebuild to check for kernel config options using linux-info.eclass. I
> don't think we have any guidelines documented, instead leaving it up
> to the "common sense" of package maintainers.
> 
> Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
> when running emerge, so we should do so only when there is a
> compensating benefit. It doesn't make sense to check for kernel
> options that are very commonly enabled. But what is "very common"?
> 
> An obvious example would be CONFIG_INET, which controls IPv4 support
> in the kernel. It does not make sense to check for that in every
> package that uses AF_INET sockets.
> 
> A less obvious example: a user filed a bug against net-misc/dhcpcd
> today asking that we check for CONFIG_PACKET [1]. My first thought was
> "why would you ever disable that?". The option description even says
> "if unsure, say Y". However, I suppose it is technically possible to
> run a Linux system with it disabled.
> 
> I think a reasonable rule of thumb would be to assume we can rely on
> options that are enabled by "make defconfig". If the user chooses to
> disable them, they are responsible for anything that breaks.
> 
> Thoughts?

We can document in the wiki that going against defconfig means you keep
the pieces when something explodes colorfully and/or come up with a even
smaller list of config items expected.


lu

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Gilbert]

On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:
> > Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
> > when running emerge, so we should do so only when there is a
> > compensating benefit.
>
> Is this a significant slowdown? Do you have any numbers?

Adding a check for CONFIG_PACKET to the dhcpcd ebuild adds around 7
seconds of delay time to the pkg_setup and/or pkg_pretend phase on my
system.

That's ok if a small number of packages are doing it, but it would
become quite annoying if a significant number of them get queued up.

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

On Mon, Sep 27, 2021 at 01:15:10PM -0400, Mike Gilbert wrote:
> On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:
> > > Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
> > > when running emerge, so we should do so only when there is a
> > > compensating benefit.
> >
> > Is this a significant slowdown? Do you have any numbers?
> 
> Adding a check for CONFIG_PACKET to the dhcpcd ebuild adds around 7
> seconds of delay time to the pkg_setup and/or pkg_pretend phase on my
> system.
> 
> That's ok if a small number of packages are doing it, but it would
> become quite annoying if a significant number of them get queued up.
7 seconds is ridiculous.

I think we need to strip out a lot of the crap about trying to detect
things in the stuff being built, and reduce the check to the simplest
possible form:
$ time zgrep -w CONFIG_PACKET /proc/config.gz
CONFIG_PACKET=y

real	0m0.032s
user	0m0.021s
sys	0m0.018s

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Rich Freeman]

On Mon, Sep 27, 2021 at 1:44 PM Robin H. Johnson <robbat2@gentoo.org> wrote:
>
> I think we need to strip out a lot of the crap about trying to detect
> things in the stuff being built, and reduce the check to the simplest
> possible form:
> $ time zgrep -w CONFIG_PACKET /proc/config.gz
> CONFIG_PACKET=y
>

Sure, just please don't strip out the part that actually does what you
just did - check the running kernel.  I don't think we should assume
that the user has the configured+prepared sources for a kernel just
lying around all the time.  I don't build in /usr/src (which is
apparently discouraged by upstream anyway), so /proc/config.gz is
really the only reliable indication of how things are setup.  Well,
that or a config file in /boot which I'm sure others don't use (but
make install puts it there).

-- 
Rich

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Gilbert]

On Mon, Sep 27, 2021 at 1:44 PM Robin H. Johnson <robbat2@gentoo.org> wrote:
>
> On Mon, Sep 27, 2021 at 01:15:10PM -0400, Mike Gilbert wrote:
> > On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:
> > > > Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
> > > > when running emerge, so we should do so only when there is a
> > > > compensating benefit.
> > >
> > > Is this a significant slowdown? Do you have any numbers?
> >
> > Adding a check for CONFIG_PACKET to the dhcpcd ebuild adds around 7
> > seconds of delay time to the pkg_setup and/or pkg_pretend phase on my
> > system.
> >
> > That's ok if a small number of packages are doing it, but it would
> > become quite annoying if a significant number of them get queued up.
> 7 seconds is ridiculous.

I think the horribly slow performance is mainly due to this change in
how we detect the kernel version:

https://gitweb.gentoo.org/repo/gentoo.git/commit/eclass/linux-info.eclass?id=d1ea596f034285cd044006b107a60902ea059793

That causes "make" to be called 4 times, and each make invocation adds
a couple of seconds to the get_version call.

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Pagano]

On 9/27/21 2:15 PM, Mike Gilbert wrote:
> On Mon, Sep 27, 2021 at 1:44 PM Robin H. Johnson <robbat2@gentoo.org> wrote:
>>
>> On Mon, Sep 27, 2021 at 01:15:10PM -0400, Mike Gilbert wrote:
>>> On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:
>>>>> Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
>>>>> when running emerge, so we should do so only when there is a
>>>>> compensating benefit.
>>>>
>>>> Is this a significant slowdown? Do you have any numbers?
>>>
>>> Adding a check for CONFIG_PACKET to the dhcpcd ebuild adds around 7
>>> seconds of delay time to the pkg_setup and/or pkg_pretend phase on my
>>> system.
>>>
>>> That's ok if a small number of packages are doing it, but it would
>>> become quite annoying if a significant number of them get queued up.
>> 7 seconds is ridiculous.
> 
> I think the horribly slow performance is mainly due to this change in
> how we detect the kernel version:
> 
> https://gitweb.gentoo.org/repo/gentoo.git/commit/eclass/linux-info.eclass?id=d1ea596f034285cd044006b107a60902ea059793
> 
> That causes "make" to be called 4 times, and each make invocation adds
> a couple of seconds to the get_version call.
> 

Changes to these eclasses are like a prison of circular breakage for which there is no escape.

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Gilbert]

On Mon, Sep 27, 2021 at 12:23 PM Mike Pagano <mpagano@gentoo.org> wrote:
>
> On 9/27/21 12:10 PM, Mike Gilbert wrote:
> > I'm looking to solicit opinions on when it is appropriate for an
> > ebuild to check for kernel config options using linux-info.eclass. I
> > don't think we have any guidelines documented, instead leaving it up
> > to the "common sense" of package maintainers.
> >
> > Adding linux-info calls to pkg_pretend or pkg_setup causes slowdowns
> > when running emerge, so we should do so only when there is a
> > compensating benefit. It doesn't make sense to check for kernel
> > options that are very commonly enabled. But what is "very common"?
> >
> > An obvious example would be CONFIG_INET, which controls IPv4 support
> > in the kernel. It does not make sense to check for that in every
> > package that uses AF_INET sockets.
> >
> > A less obvious example: a user filed a bug against net-misc/dhcpcd
> > today asking that we check for CONFIG_PACKET [1]. My first thought was
> > "why would you ever disable that?". The option description even says
> > "if unsure, say Y". However, I suppose it is technically possible to
> > run a Linux system with it disabled.
> >
> > I think a reasonable rule of thumb would be to assume we can rely on
> > options that are enabled by "make defconfig". If the user chooses to
> > disable them, they are responsible for anything that breaks.
> >
> > Thoughts?
> >
> > [1] https://bugs.gentoo.org/815064
> >
>
> The challenge I see is that these config checks head off bugs and issues without our intervention.
>
> We as kernel maintainers depend on ebuild maintainers to check these things so they don't become "kernel bugs" to figure out.

I guess I'm proposing that we set some minimal standard for kernel
configs. It's a waste of time and effort to pepper random ebuilds with
checks for options that everyone should have enabled in the first
place.

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Gilbert]

On Mon, Sep 27, 2021 at 1:56 PM Rich Freeman <rich0@gentoo.org> wrote:
>
> On Mon, Sep 27, 2021 at 1:44 PM Robin H. Johnson <robbat2@gentoo.org> wrote:
> >
> > I think we need to strip out a lot of the crap about trying to detect
> > things in the stuff being built, and reduce the check to the simplest
> > possible form:
> > $ time zgrep -w CONFIG_PACKET /proc/config.gz
> > CONFIG_PACKET=y
> >
>
> Sure, just please don't strip out the part that actually does what you
> just did - check the running kernel.  I don't think we should assume
> that the user has the configured+prepared sources for a kernel just
> lying around all the time.  I don't build in /usr/src (which is
> apparently discouraged by upstream anyway), so /proc/config.gz is
> really the only reliable indication of how things are setup.  Well,
> that or a config file in /boot which I'm sure others don't use (but
> make install puts it there).

I like the idea of a stripped-down implementation that just checks the
running kernel config when packages get installed. I think that
probably means creating a new eclass though, or moving some of the
current complexity into linux-mod.eclass.

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Peter Stuge]

Mike Gilbert wrote:
> It's a waste of time and effort to pepper random ebuilds with checks
> for options that everyone should have enabled in the first place.

It's not for you to say what everyone should have enabled in their kernel.

There's significant value in ebuilds documenting required kernel options in
a structured manner.

So I welcome simplifications in the checking to achieve millisecond
test times. But the benefit of structured requirements is given even
without any runtime checking at all, as long as required options
continue to be codified /somehow/.


Thank you for your work!

//Peter

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Gilbert]

On Mon, Sep 27, 2021 at 3:11 PM Peter Stuge <peter@stuge.se> wrote:
>
> Mike Gilbert wrote:
> > It's a waste of time and effort to pepper random ebuilds with checks
> > for options that everyone should have enabled in the first place.
>
> It's not for you to say what everyone should have enabled in their kernel.

Do you not agree that there are some options that should always be
enabled, or at least that we can assume are enabled?

To use my earlier example, should every package that uses AF_INET
check for CONFIG_INET in the kernel?

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Jason A. Donenfeld]

On Mon, Sep 27, 2021 at 11:44 AM Robin H. Johnson <robbat2@gentoo.org> wrote:
> I think we need to strip out a lot of the crap about trying to detect
> things in the stuff being built, and reduce the check to the simplest
> possible form:
> $ time zgrep -w CONFIG_PACKET /proc/config.gz
> CONFIG_PACKET=y

The results from our current method could of course be cached, and
even cached between runs, by just relying on `/proc/version` changing
for different kernels. This seems easy enough to do.

Alternatively, sure, we could require that everyone has /proc/config
in their kernel config. Many kernels don't have this (mine doesn't),
but we could add it to the base requirements.

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Mike Gilbert]

On Mon, Sep 27, 2021 at 4:28 PM Jason A. Donenfeld <zx2c4@gentoo.org> wrote:
>
> On Mon, Sep 27, 2021 at 11:44 AM Robin H. Johnson <robbat2@gentoo.org> wrote:
> > I think we need to strip out a lot of the crap about trying to detect
> > things in the stuff being built, and reduce the check to the simplest
> > possible form:
> > $ time zgrep -w CONFIG_PACKET /proc/config.gz
> > CONFIG_PACKET=y
>
> The results from our current method could of course be cached, and
> even cached between runs, by just relying on `/proc/version` changing
> for different kernels. This seems easy enough to do.
>
> Alternatively, sure, we could require that everyone has /proc/config
> in their kernel config. Many kernels don't have this (mine doesn't),
> but we could add it to the base requirements.

I think it would be reasonable to check a couple fallback locations:

/proc/config.gz
/lib/modules/$(uname -r)/build/.config
/boot/config-$(uname-r)

The slowness really comes from guessing the kernel build location and
invoking the kernel build system via make. Avoiding that is a big
improvement.

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Peter Stuge]

Mike Gilbert wrote:
> > > It's a waste of time and effort to pepper random ebuilds with checks
> > > for options that everyone should have enabled in the first place.
> >
> > It's not for you to say what everyone should have enabled in their kernel.
> 
> Do you not agree that there are some options that should always be
> enabled, or at least that we can assume are enabled?

"Should be enabled" no, but I agree with the latter - some assumed set
should be fine. I think it would be good if it's (somehow) documented
though.


> To use my earlier example, should every package that uses AF_INET
> check for CONFIG_INET in the kernel?

CONFIG_INET is a perhaps surprisingly tricky example!

A package could e.g. use getaddrinfo() with no address family hint
but because of USE=-ipv6 exclude all AF_INET6 address results and
so end up using AF_INET based on whether it's available in the
running kernel or even based on third party DNS entries.


I'm not sure about the best approach for very basic options,
CONFIG_NET could be another such candidate.

Thinking towards robbat2's proposal (which I like) it might make sense
to try to map requirements of packages, but there will probably be
cases where it can't really be done successfully.

Ultimately that work should not be the responsibility of distribution
package maintainers but something upstreams deliver, similar to systemd
units, but maybe we'll invent it (if noone else has)..


//Peter

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Michael Orlitzky]

On Mon, 2021-09-27 at 15:44 -0400, Mike Gilbert wrote:
> On Mon, Sep 27, 2021 at 3:11 PM Peter Stuge <peter@stuge.se> wrote:
> > 
> > Mike Gilbert wrote:
> > > It's a waste of time and effort to pepper random ebuilds with checks
> > > for options that everyone should have enabled in the first place.
> > 
> > It's not for you to say what everyone should have enabled in their kernel.
> 
> Do you not agree that there are some options that should always be
> enabled, or at least that we can assume are enabled?
> 

There is a distro/Kconfig in gentoo-sources with,

  config GENTOO_LINUX
	bool "Gentoo Linux support"
        ...

that could be extended to include a set of assumed features; otherwise
a sub-setting (default enabled) could be added.

Re: [gentoo-dev] Guidance on adding kernel config checks to ebuilds

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

Il giorno lun 27 set 2021 alle ore 18:11 Mike Gilbert <floppym@gentoo.org>
ha scritto:

> I'm looking to solicit opinions on when it is appropriate for an
> ebuild to check for kernel config options using linux-info.eclass. I
> don't think we have any guidelines documented, instead leaving it up
> to the "common sense" of package maintainers.
>
> <snip>

After so many tentatives to fix the kernel checks in these years (almost
all of which had drawbacks or missed some extreme corner case) it's
probably better to give to the user instruments to do ihs own checks rather
than trying to be smart.

An example of how this could work follow:
A file (or directory) is created in /etc/ that contains a list of
<ebuild>,<config_option>,<state_required>
Then the user is responsible to check that list against the wanna be
running kernel

This save probably both computational and human time

state_required - should be well thought out since it can be required
present, absent or maybe even suggested

[-- Attachment #2: Type: text/html, Size: 1388 bytes --]