[gentoo-user] How to find out all openrc dependencies?

[gentoo-user] How to find out all openrc dependencies?

[J. Roeleveld]

Hi all,

For a while I've been seeing the following ERROR-messages when booting 1 of my 
systems:

* ERROR: cannot start multipathd as localmount would not start
 * ERROR: cannot start zfs-import as localmount would not start

This isn't a big concern as these services will start correctly later:

INIT: Entering runlevel: 3
 * Starting multipathd ...
 [ ok ]
 * Importing ZFS pool(s)  ...
 [ ok ]

But I am trying to find the cause of these errors as they are preventing 
parallel-start from actually working correctly.

When I check with "rc-depend", I don't see an obious cause:

# /lib/rc/bin/rc-depend multipathd
sysfs devfs udev udev-trigger modules fsck root localmount multipathd

# /lib/rc/bin/rc-depend localmount
sysfs devfs udev udev-trigger modules fsck root localmount

# /lib/rc/bin/rc-depend zfs-import
multipath sysfs devfs udev udev-trigger modules fsck root localmount 
multipathd zfs-import

# /lib/rc/bin/rc-depend multipath
multipath

From how I read these, it should be able to start "localmount" properly before 
even trying to start "multipathd" and "zfs-import"
There is also no technical dependency for "localmount" (the root filesystem is 
not on ZFS on this system)

Any help/suggestions on how to find the cause would be appreciated.

--
Joost

Re: [gentoo-user] How to find out all openrc dependencies?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

On Thursday, 11 April 2024 06:19:57 BST J. Roeleveld wrote:
> Hi all,
> 
> For a while I've been seeing the following ERROR-messages when booting 1 of
> my systems:
> 
> * ERROR: cannot start multipathd as localmount would not start
>  * ERROR: cannot start zfs-import as localmount would not start
> 
> This isn't a big concern as these services will start correctly later:
> 
> INIT: Entering runlevel: 3
>  * Starting multipathd ...
>  [ ok ]
>  * Importing ZFS pool(s)  ...
>  [ ok ]
> 
> But I am trying to find the cause of these errors as they are preventing
> parallel-start from actually working correctly.
> 
> When I check with "rc-depend", I don't see an obious cause:
> 
> # /lib/rc/bin/rc-depend multipathd
> sysfs devfs udev udev-trigger modules fsck root localmount multipathd
> 
> # /lib/rc/bin/rc-depend localmount
> sysfs devfs udev udev-trigger modules fsck root localmount
> 
> # /lib/rc/bin/rc-depend zfs-import
> multipath sysfs devfs udev udev-trigger modules fsck root localmount
> multipathd zfs-import
> 
> # /lib/rc/bin/rc-depend multipath
> multipath
> 
> From how I read these, it should be able to start "localmount" properly
> before even trying to start "multipathd" and "zfs-import"
> There is also no technical dependency for "localmount" (the root filesystem
> is not on ZFS on this system)
> 
> Any help/suggestions on how to find the cause would be appreciated.
> 
> --
> Joost

Check if hwclock is in the boot runlevel:

rc-update -s -v | grep hwclock

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] How to find out all openrc dependencies?

[J. Roeleveld]

On Thursday, 11 April 2024 11:35:10 CEST Michael wrote:
> On Thursday, 11 April 2024 06:19:57 BST J. Roeleveld wrote:
> > Hi all,
> > 
> > For a while I've been seeing the following ERROR-messages when booting 1
> > of
> > my systems:
> > 
> > * ERROR: cannot start multipathd as localmount would not start
> > 
> >  * ERROR: cannot start zfs-import as localmount would not start
> > 
> > This isn't a big concern as these services will start correctly later:
> > 
> > INIT: Entering runlevel: 3
> > 
> >  * Starting multipathd ...
> >  [ ok ]
> >  * Importing ZFS pool(s)  ...
> >  [ ok ]
> > 
> > But I am trying to find the cause of these errors as they are preventing
> > parallel-start from actually working correctly.
> > 
> > When I check with "rc-depend", I don't see an obious cause:
> > 
> > # /lib/rc/bin/rc-depend multipathd
> > sysfs devfs udev udev-trigger modules fsck root localmount multipathd
> > 
> > # /lib/rc/bin/rc-depend localmount
> > sysfs devfs udev udev-trigger modules fsck root localmount
> > 
> > # /lib/rc/bin/rc-depend zfs-import
> > multipath sysfs devfs udev udev-trigger modules fsck root localmount
> > multipathd zfs-import
> > 
> > # /lib/rc/bin/rc-depend multipath
> > multipath
> > 
> > From how I read these, it should be able to start "localmount" properly
> > before even trying to start "multipathd" and "zfs-import"
> > There is also no technical dependency for "localmount" (the root
> > filesystem
> > is not on ZFS on this system)
> > 
> > Any help/suggestions on how to find the cause would be appreciated.
> > 
> > --
> > Joost
> 
> Check if hwclock is in the boot runlevel:
> 
> rc-update -s -v | grep hwclock

What does "hwclock" got to do with this?
It has no dependency with multipathd, zfs-import, localmount or anything else 
that is showing an error.

--
Joost

Re: [gentoo-user] How to find out all openrc dependencies?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

On Thursday, 11 April 2024 10:48:15 BST J. Roeleveld wrote:
> On Thursday, 11 April 2024 11:35:10 CEST Michael wrote:
> > On Thursday, 11 April 2024 06:19:57 BST J. Roeleveld wrote:
> > > Hi all,
> > > 
> > > For a while I've been seeing the following ERROR-messages when booting 1
> > > of
> > > my systems:
> > > 
> > > * ERROR: cannot start multipathd as localmount would not start
> > > 
> > >  * ERROR: cannot start zfs-import as localmount would not start
> > > 
> > > This isn't a big concern as these services will start correctly later:
> > > 
> > > INIT: Entering runlevel: 3
> > > 
> > >  * Starting multipathd ...
> > >  [ ok ]
> > >  * Importing ZFS pool(s)  ...
> > >  [ ok ]
> > > 
> > > But I am trying to find the cause of these errors as they are preventing
> > > parallel-start from actually working correctly.
> > > 
> > > When I check with "rc-depend", I don't see an obious cause:
> > > 
> > > # /lib/rc/bin/rc-depend multipathd
> > > sysfs devfs udev udev-trigger modules fsck root localmount multipathd
> > > 
> > > # /lib/rc/bin/rc-depend localmount
> > > sysfs devfs udev udev-trigger modules fsck root localmount
> > > 
> > > # /lib/rc/bin/rc-depend zfs-import
> > > multipath sysfs devfs udev udev-trigger modules fsck root localmount
> > > multipathd zfs-import
> > > 
> > > # /lib/rc/bin/rc-depend multipath
> > > multipath
> > > 
> > > From how I read these, it should be able to start "localmount" properly
> > > before even trying to start "multipathd" and "zfs-import"
> > > There is also no technical dependency for "localmount" (the root
> > > filesystem
> > > is not on ZFS on this system)
> > > 
> > > Any help/suggestions on how to find the cause would be appreciated.
> > > 
> > > --
> > > Joost
> > 
> > Check if hwclock is in the boot runlevel:
> > 
> > rc-update -s -v | grep hwclock
> 
> What does "hwclock" got to do with this?
> It has no dependency with multipathd, zfs-import, localmount or anything
> else that is showing an error.
> 
> --
> Joost

Our systems are certainly different, but I noticed this dependency on my 
localmount which is missing on yours:

# /lib/rc/bin/rc-depend localmount
sysfs devfs udev udev-trigger hwclock modules fsck root dmcrypt localmount
                              ^^^^^^^
Have you compared your system services which has this problem, with other 
systems of yours which can startup properly?


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] How to find out all openrc dependencies?

[J. Roeleveld]

On Thursday, 11 April 2024 12:10:31 CEST Michael wrote:
> On Thursday, 11 April 2024 10:48:15 BST J. Roeleveld wrote:
> > On Thursday, 11 April 2024 11:35:10 CEST Michael wrote:
> > > On Thursday, 11 April 2024 06:19:57 BST J. Roeleveld wrote:
> > > > Hi all,
> > > > 
> > > > For a while I've been seeing the following ERROR-messages when booting
> > > > 1
> > > > of
> > > > my systems:
> > > > 
> > > > * ERROR: cannot start multipathd as localmount would not start
> > > > 
> > > >  * ERROR: cannot start zfs-import as localmount would not start
> > > > 
> > > > This isn't a big concern as these services will start correctly later:
> > > > 
> > > > INIT: Entering runlevel: 3
> > > > 
> > > >  * Starting multipathd ...
> > > >  [ ok ]
> > > >  * Importing ZFS pool(s)  ...
> > > >  [ ok ]
> > > > 
> > > > But I am trying to find the cause of these errors as they are
> > > > preventing
> > > > parallel-start from actually working correctly.
> > > > 
> > > > When I check with "rc-depend", I don't see an obious cause:
> > > > 
> > > > # /lib/rc/bin/rc-depend multipathd
> > > > sysfs devfs udev udev-trigger modules fsck root localmount multipathd
> > > > 
> > > > # /lib/rc/bin/rc-depend localmount
> > > > sysfs devfs udev udev-trigger modules fsck root localmount
> > > > 
> > > > # /lib/rc/bin/rc-depend zfs-import
> > > > multipath sysfs devfs udev udev-trigger modules fsck root localmount
> > > > multipathd zfs-import
> > > > 
> > > > # /lib/rc/bin/rc-depend multipath
> > > > multipath
> > > > 
> > > > From how I read these, it should be able to start "localmount"
> > > > properly
> > > > before even trying to start "multipathd" and "zfs-import"
> > > > There is also no technical dependency for "localmount" (the root
> > > > filesystem
> > > > is not on ZFS on this system)
> > > > 
> > > > Any help/suggestions on how to find the cause would be appreciated.
> > > > 
> > > > --
> > > > Joost
> > > 
> > > Check if hwclock is in the boot runlevel:
> > > 
> > > rc-update -s -v | grep hwclock
> > 
> > What does "hwclock" got to do with this?
> > It has no dependency with multipathd, zfs-import, localmount or anything
> > else that is showing an error.
> > 
> > --
> > Joost
> 
> Our systems are certainly different, but I noticed this dependency on my
> localmount which is missing on yours:
> 
> # /lib/rc/bin/rc-depend localmount
> sysfs devfs udev udev-trigger hwclock modules fsck root dmcrypt localmount
>                               ^^^^^^^
> Have you compared your system services which has this problem, with other
> systems of yours which can startup properly?

Adding additional dependencies into the tree is more likely to cause further 
issues. I am actually looking for how to quickly find out which dependency is 
causing a circular dependency issue as the first time it thinks it needs to 
start a service it fails. But the 2nd time it starts, it goes correctly.

I removed hwclock from ALL VMs as they don't actually have a hwclock.

I did find out the actual cause of the problem through a lot of trial and 
error, but this is not really useful in actually quickly finding the problem.
Being able to "simulate" the startup sequence for how OpenRC wants to do 
things would have simplified and sped up the entire process.

--
Joost

Re: [gentoo-user] How to find out all openrc dependencies?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 3792 bytes --]

On Sunday, 14 April 2024 08:28:07 BST J. Roeleveld wrote:
> On Thursday, 11 April 2024 12:10:31 CEST Michael wrote:
> > On Thursday, 11 April 2024 10:48:15 BST J. Roeleveld wrote:
> > > On Thursday, 11 April 2024 11:35:10 CEST Michael wrote:
> > > > On Thursday, 11 April 2024 06:19:57 BST J. Roeleveld wrote:
> > > > > Hi all,
> > > > > 
> > > > > For a while I've been seeing the following ERROR-messages when
> > > > > booting
> > > > > 1
> > > > > of
> > > > > my systems:
> > > > > 
> > > > > * ERROR: cannot start multipathd as localmount would not start
> > > > > 
> > > > >  * ERROR: cannot start zfs-import as localmount would not start
> > > > > 
> > > > > This isn't a big concern as these services will start correctly
> > > > > later:
> > > > > 
> > > > > INIT: Entering runlevel: 3
> > > > > 
> > > > >  * Starting multipathd ...
> > > > >  [ ok ]
> > > > >  * Importing ZFS pool(s)  ...
> > > > >  [ ok ]
> > > > > 
> > > > > But I am trying to find the cause of these errors as they are
> > > > > preventing
> > > > > parallel-start from actually working correctly.
> > > > > 
> > > > > When I check with "rc-depend", I don't see an obious cause:
> > > > > 
> > > > > # /lib/rc/bin/rc-depend multipathd
> > > > > sysfs devfs udev udev-trigger modules fsck root localmount
> > > > > multipathd
> > > > > 
> > > > > # /lib/rc/bin/rc-depend localmount
> > > > > sysfs devfs udev udev-trigger modules fsck root localmount
> > > > > 
> > > > > # /lib/rc/bin/rc-depend zfs-import
> > > > > multipath sysfs devfs udev udev-trigger modules fsck root localmount
> > > > > multipathd zfs-import
> > > > > 
> > > > > # /lib/rc/bin/rc-depend multipath
> > > > > multipath
> > > > > 
> > > > > From how I read these, it should be able to start "localmount"
> > > > > properly
> > > > > before even trying to start "multipathd" and "zfs-import"
> > > > > There is also no technical dependency for "localmount" (the root
> > > > > filesystem
> > > > > is not on ZFS on this system)
> > > > > 
> > > > > Any help/suggestions on how to find the cause would be appreciated.
> > > > > 
> > > > > --
> > > > > Joost
> > > > 
> > > > Check if hwclock is in the boot runlevel:
> > > > 
> > > > rc-update -s -v | grep hwclock
> > > 
> > > What does "hwclock" got to do with this?
> > > It has no dependency with multipathd, zfs-import, localmount or anything
> > > else that is showing an error.
> > > 
> > > --
> > > Joost
> > 
> > Our systems are certainly different, but I noticed this dependency on my
> > localmount which is missing on yours:
> > 
> > # /lib/rc/bin/rc-depend localmount
> > sysfs devfs udev udev-trigger hwclock modules fsck root dmcrypt localmount
> > 
> >                               ^^^^^^^
> > 
> > Have you compared your system services which has this problem, with other
> > systems of yours which can startup properly?
> 
> Adding additional dependencies into the tree is more likely to cause further
> issues. I am actually looking for how to quickly find out which dependency
> is causing a circular dependency issue as the first time it thinks it needs
> to start a service it fails. But the 2nd time it starts, it goes correctly.
> 
> I removed hwclock from ALL VMs as they don't actually have a hwclock.
> 
> I did find out the actual cause of the problem through a lot of trial and
> error, 

Out of curiosity - what was the cause of this?  I have only come across 
hwclock on my installations (not VMs).


> but this is not really useful in actually quickly finding the
> problem. Being able to "simulate" the startup sequence for how OpenRC wants
> to do things would have simplified and sped up the entire process.
> 
> --
> Joost

I enable the rc log and check how the various services try to start up, 
however the information provided is not always useful.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] How to find out all openrc dependencies?

[J. Roeleveld]

On Sunday, 14 April 2024 10:24:54 CEST Michael wrote:
> On Sunday, 14 April 2024 08:28:07 BST J. Roeleveld wrote:

<snipped>

> > > 
> > > Our systems are certainly different, but I noticed this dependency on my
> > > localmount which is missing on yours:
> > > 
> > > # /lib/rc/bin/rc-depend localmount
> > > sysfs devfs udev udev-trigger hwclock modules fsck root dmcrypt
> > > localmount
> > > 
> > >                               ^^^^^^^
> > > 
> > > Have you compared your system services which has this problem, with
> > > other
> > > systems of yours which can startup properly?
> > 
> > Adding additional dependencies into the tree is more likely to cause
> > further issues. I am actually looking for how to quickly find out which
> > dependency is causing a circular dependency issue as the first time it
> > thinks it needs to start a service it fails. But the 2nd time it starts,
> > it goes correctly.
> > 
> > I removed hwclock from ALL VMs as they don't actually have a hwclock.
> > 
> > I did find out the actual cause of the problem through a lot of trial and
> > error,
> 
> Out of curiosity - what was the cause of this?  I have only come across
> hwclock on my installations (not VMs).

The issue is related to how openrc handles the boot process. It first starts 
everything in "boot" and then switches to whatever is set as default runlevel.

At some point, Gentoo devs found it useful to add multipath and zfs-import to 
the boot-runlevel. This was not originally the case.
I consider it best that the boot-runlevel is kept as empty as possible and 
have the user/admin add necessary stuff to it when needed.

I removed them from the boot runlevel and the issue has been resolved.

This was the last item on the list preventing correct parallel boot and my 
server now boots in 8 minutes instead of the original 20.
It's still 8 because I need to add some tricks to reduce some of the 
simultaneous stuff which will lead to timeout issues which actually make it 
worse if left to defaults.

> > but this is not really useful in actually quickly finding the
> > problem. Being able to "simulate" the startup sequence for how OpenRC
> > wants
> > to do things would have simplified and sped up the entire process.
> > 
> > --
> > Joost
> 
> I enable the rc log and check how the various services try to start up,
> however the information provided is not always useful.

The rc.log is actually less useful then one would expect. It only logs the 
same output I see on the terminal.
Unless there is an option to add additional information to the logs, like how 
it determines what to start next?

--
Joost