[gentoo-user] Openrc service crash

[gentoo-user] Openrc service crash

[Michael]

I noticed 'www-servers/gatling' now crashes at boot.  When I check with 
netstat there is no listening port for connections.  rc-service reports 
gatling has crashed, there is a PID for it, but it is not listed under ps:

# rc-service gatling status
 * status: crashed
# cat /run/gatling.pid 
4021
# ps axfu | grep gatling
root      4165  0.0  0.0   6572  2224 pts/1    S+   13:13   0:00              
\_ grep --colour=auto gatling


If I try to start it I get a service already started complain.  All I see in 
the logs is:

 *   ip link set dev enp2s0 up
 *   Configuring enp2s0 for MAC address 00:AA:BB:CC:DD:FF ...
 [ ok ]
 *   Starting ifplugd on enp2s0 ...
 *   start-stop-daemon: fopen `/run/ifplugd.enp2s0.pid': No such file or 
directory
 *   Detaching to start `/usr/sbin/ifplugd' ...
 [ ok ]
 *     Backgrounding ...
 * WARNING: net.enp2s0 has started, but is inactive
 * WARNING: gatling will start when net.enp2s0 has started


I think this problem started when I changed this PC's LAN network and it now 
has IPv6 as well as IPv4, but this could be a coincidence.  The init.d script 
contains net as a startup dependency:

depend() {
        need net
}

Could it be negotiating IP addresses now takes too long and gatling fails to 
initialize and crashes for this reason?  It is configured to only listen to an 
IPv4 address, so I assume the addition of IPv6 should not really affect it.

Restarting it manually works and the service stays up, until the next reboot.

Any ideas what might be causing this?

Re: [gentoo-user] Openrc service crash

[Michael]

On Friday, 7 April 2023 13:32:52 BST Michael wrote:
> I noticed 'www-servers/gatling' now crashes at boot.  When I check with
> netstat there is no listening port for connections.  rc-service reports
> gatling has crashed, there is a PID for it, but it is not listed under ps:
> 
> # rc-service gatling status
>  * status: crashed
> # cat /run/gatling.pid
> 4021
> # ps axfu | grep gatling
> root      4165  0.0  0.0   6572  2224 pts/1    S+   13:13   0:00
> \_ grep --colour=auto gatling
> 
> 
> If I try to start it I get a service already started complain.  All I see in
> the logs is:
> 
>  *   ip link set dev enp2s0 up
>  *   Configuring enp2s0 for MAC address 00:AA:BB:CC:DD:FF ...
>  [ ok ]
>  *   Starting ifplugd on enp2s0 ...
>  *   start-stop-daemon: fopen `/run/ifplugd.enp2s0.pid': No such file or
> directory
>  *   Detaching to start `/usr/sbin/ifplugd' ...
>  [ ok ]
>  *     Backgrounding ...
>  * WARNING: net.enp2s0 has started, but is inactive
>  * WARNING: gatling will start when net.enp2s0 has started
> 
> 
> I think this problem started when I changed this PC's LAN network and it now
> has IPv6 as well as IPv4, but this could be a coincidence.  The init.d
> script contains net as a startup dependency:
> 
> depend() {
>         need net
> }
> 
> Could it be negotiating IP addresses now takes too long and gatling fails to
> initialize and crashes for this reason?  It is configured to only listen to
> an IPv4 address, so I assume the addition of IPv6 should not really affect
> it.
> 
> Restarting it manually works and the service stays up, until the next
> reboot.
> 
> Any ideas what might be causing this?

It was probably the addition of IPv6 - as a workaround I added 'sleep 5s' 
before the gatling init.d exec command and it now starts normally.

Re: [gentoo-user] Openrc service crash

[Arve Barsnes]

On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
> It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
> before the gatling init.d exec command and it now starts normally.

If it was indeed the network that was missing, you could try changing
the service from "need" to "after" starting the net service.

Add to /etc/rc.conf:
rc_gatling_after="net"

Regards,
Arve

Re: [gentoo-user] Openrc service crash

[Michael]

On Saturday, 8 April 2023 09:28:16 BST Arve Barsnes wrote:
> On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
> > It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
> > before the gatling init.d exec command and it now starts normally.
> 
> If it was indeed the network that was missing, you could try changing
> the service from "need" to "after" starting the net service.
> 
> Add to /etc/rc.conf:
> rc_gatling_after="net"
> 
> Regards,
> Arve

Thanks Arve, I will try it.  Should this replace the "need net" string here, 
or added somewhere else?

depend() {
        need net
}

Re: [gentoo-user] Openrc service crash

[Michael]

On Saturday, 8 April 2023 09:32:31 BST Michael wrote:
> On Saturday, 8 April 2023 09:28:16 BST Arve Barsnes wrote:
> > On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
> > > It was probably the addition of IPv6 - as a workaround I added 'sleep
> > > 5s'
> > > before the gatling init.d exec command and it now starts normally.
> > 
> > If it was indeed the network that was missing, you could try changing
> > the service from "need" to "after" starting the net service.
> > 
> > Add to /etc/rc.conf:
> > rc_gatling_after="net"
> > 
> > Regards,
> > Arve
> 
> Thanks Arve, I will try it.  Should this replace the "need net" string here,
> or added somewhere else?
> 
> depend() {
>         need net
> }

Oops!  Please ignore my question ... I didn't have enough coffee yet.  :-)

Re: [gentoo-user] Openrc service crash

[Michael]

On Saturday, 8 April 2023 09:33:23 BST Michael wrote:
> On Saturday, 8 April 2023 09:32:31 BST Michael wrote:
> > On Saturday, 8 April 2023 09:28:16 BST Arve Barsnes wrote:
> > > On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
> > > > It was probably the addition of IPv6 - as a workaround I added 'sleep
> > > > 5s'
> > > > before the gatling init.d exec command and it now starts normally.
> > > 
> > > If it was indeed the network that was missing, you could try changing
> > > the service from "need" to "after" starting the net service.
> > > 
> > > Add to /etc/rc.conf:
> > > rc_gatling_after="net"
> > > 
> > > Regards,
> > > Arve
> > 
> > Thanks Arve, I will try it.  Should this replace the "need net" string
> > here, or added somewhere else?
> > 
> > depend() {
> > 
> >         need net
> > 
> > }
> 
> Oops!  Please ignore my question ... I didn't have enough coffee yet.  :-)

OK, I added rc_gatling_after="net" in rc.conf, but it didn't work.  gatling 
crashed.  However, adding 'sleep 5s' in its init.d startup script works.  What 
might be causing this discrepancy?

Re: [gentoo-user] Openrc service crash

[Arve Barsnes]

On Sat, 8 Apr 2023 at 10:41, Michael <confabulate@kintzios.com> wrote:
> OK, I added rc_gatling_after="net" in rc.conf, but it didn't work.  gatling
> crashed.  However, adding 'sleep 5s' in its init.d startup script works.  What
> might be causing this discrepancy?

I see now in your first message that the output says that gatling will
wait and start after net.enp2s0 has started, I guess there must be a
crash while waiting. Is there no logs from the daemon? There might be
a way to add a delay in rc.conf as well, so you don't directly edit
the init script.

Regards,
Arve

Re: [gentoo-user] Openrc service crash

[Michael]

On Saturday, 8 April 2023 10:10:49 BST Arve Barsnes wrote:

> I see now in your first message that the output says that gatling will
> wait and start after net.enp2s0 has started, I guess there must be a
> crash while waiting. Is there no logs from the daemon? There might be
> a way to add a delay in rc.conf as well, so you don't directly edit
> the init script.
> 
> Regards,
> Arve

The gatling webserver is rather simple with no logging or debugging options 
that I have found.  The rc.conf log does not capture gatling's internal code 
crash, so I'm none the wiser.  gatling works nicely once up & running with a 
minimal footprint, which is why I chose it when boa fell off the tree.  I 
suppose commensurate with this is its lack of more sophisticated features, 
like backgrounding at launch without crashing.  ;-p

The *BSD's rc.conf has a netwait option, but I don't think Gentoo's rc uses 
this.  Hacking the init script with a simple 'sleep' addition is not the best 
approach, but I'm happy it works for now.  :-)

[gentoo-user] Re: Openrc service crash

[Holger Hoffstätte]

On 2023-04-08 10:41, Michael wrote:
> On Saturday, 8 April 2023 09:33:23 BST Michael wrote:
>> On Saturday, 8 April 2023 09:32:31 BST Michael wrote:
>>> On Saturday, 8 April 2023 09:28:16 BST Arve Barsnes wrote:
>>>> On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
>>>>> It was probably the addition of IPv6 - as a workaround I added 'sleep
>>>>> 5s'
>>>>> before the gatling init.d exec command and it now starts normally.
>>>>
>>>> If it was indeed the network that was missing, you could try changing
>>>> the service from "need" to "after" starting the net service.
>>>>
>>>> Add to /etc/rc.conf:
>>>> rc_gatling_after="net"
>>>>
>>>> Regards,
>>>> Arve
>>>
>>> Thanks Arve, I will try it.  Should this replace the "need net" string
>>> here, or added somewhere else?
>>>
>>> depend() {
>>>
>>>          need net
>>>
>>> }
>>
>> Oops!  Please ignore my question ... I didn't have enough coffee yet.  :-)
> 
> OK, I added rc_gatling_after="net" in rc.conf, but it didn't work.  gatling
> crashed.  However, adding 'sleep 5s' in its init.d startup script works.  What
> might be causing this discrepancy?

Without having more information, I wouldn't be surprised if it's the notorious
IPv6 DAD delay:

https://bugs.gentoo.org/831743

cheers
Holger

Re: [gentoo-user] Re: Openrc service crash

[Michael]

On Saturday, 8 April 2023 11:08:06 BST Holger Hoffstätte wrote:
> On 2023-04-08 10:41, Michael wrote:

> > OK, I added rc_gatling_after="net" in rc.conf, but it didn't work. 
> > gatling
> > crashed.  However, adding 'sleep 5s' in its init.d startup script works. 
> > What might be causing this discrepancy?
> 
> Without having more information, I wouldn't be surprised if it's the
> notorious IPv6 DAD delay:
> 
> https://bugs.gentoo.org/831743
> 
> cheers
> Holger

Yes, quite likely.  Although this application is configured to listen only on 
IPv4, the problem manifested after I moved this system on a new network port 
which also has IPv6.

I've come across posts complaining about services crashing at startup, but 
causal attribution was somewhat conflated between 1Gb switch gear and IPv6.  
Since this PC was on the same 1Gbps switch before and only the addition of 
IPv6 coincided with this problem, I'd think the bug you mention is a more 
probable cause.