[gentoo-user] MySQL startup problem - interface does not have an address yet.

[gentoo-user] MySQL startup problem - interface does not have an address yet.

[Manuel McLure]

I'm having a problem with booting my Gentoo system due to MySQL
hanging at startup. My system is up-to-date stable (including
udev-197, although I believe the problem started while I was still on
171) except for the kernel, which is at 3.1.10 because I can't seem to
get lirc to work correctly on newer kernels. The problem started after
a large system update just before the udev 171 to 197 migration. Among
other upgrades dhcpcd was upgraded from 5.2.12 to 5.6.4, net-tools was
upgraded from 1.60_p20110409135728 to 1.60_p20120127084908, and openrc
was upgraded from 0.9.8.4 to 0.11.8.

For historical reasons, this system has two ethernet adapters, eth0
and eth1. eth0 is not configured any more (it used to be used for IPTV
connectivity to my cable provider for MythTV before they changes their
system.) eth1 is configured via DHCP to an RFC 1918 10.x.x.x address -
the DHCP server is always configured to provide the same IP address
for this system. net.eth1 is symlinked to /etc/runlevels/default.

The boot process starts, configures eth1, starts LCDd, and then
attempts to start MySQL - and hangs for about 15 minutes before
continuing with the boot.

I added several debugging statements to /etc/init.d/mysql and found
that even though the log shows that eth1 got its IP address, when
MySQL attempts to start the interface does not have an IP address
assigned. It's like the net.eth1 script exits before the interface is
completely up.

After the 15 minute hang, the system boots up as usual, except for
MySQL not running. If I then start mysql manually it comes up
immediately.

Here's the rc.log output:

 * Starting D-BUS system messagebus ...
 [ ok ]
 * Bringing up interface eth1
 *   dhcp ...
 *     Running dhcpcd ...
dhcpcd[8400]: version 5.6.4 starting
dhcpcd[8400]: eth1: waiting for carrier
dhcpcd[8400]: eth1: carrier acquired
dhcpcd[8400]: eth1: sending IPv6 Router Solicitation
dhcpcd[8400]: eth1: sendmsg: Cannot assign requested address
dhcpcd[8400]: eth1: rebinding lease of 10.x.y.14
dhcpcd[8400]: eth1: acknowledged 10.x.y.14 from 10.x.y.1
dhcpcd[8400]: eth1: checking for 10.x.y.14
dhcpcd[8400]: eth1: Router Advertisement from NNNN::NNNN:NNNN:NNNN:NNNN
dhcpcd[8400]: forked to background, child pid 8454
 [ ok ]
 *     received address
 [ ok ]
 *   Adding routes
 *     239.0.0.0/8 ...
 [ ok ]
 * Starting LCDd ...
 [ ok ]
 * Starting mysql ...
 [ !! ]
 * ERROR: mysql failed to start
 * Starting syslog-ng ...
 [ ok ]
 * Mounting network filesystems ...
 [ ok ]

I added the following commands before the start-stop-daemon line in
/etc/init.d/mysql:

        date >/var/log/mysqlstart.log
        netstat -anp | grep 3306 >>/var/log/mysqlstart.log
        ps -efl | grep mysql >>/var/log/mysqlstart.log
        lsof -i @10.x.y.14:3306 >>/var/log/mysqlstart.log
        ifconfig -a >>/var/log/mysqlstart.log

The output of this was:

Wed Jan 23 09:56:03 PST 2013
0 S root      8497  8290  0  80   0 -  4301 poll_s 09:56 ?
00:00:00 /sbin/runscript /etc/init.d/mysql --lockfd 10 start
4 S root      8498  8497  1  80   0 -  3425 wait   09:56 ?
00:00:00 /bin/sh /lib64/rc/sh/runscript.sh /etc/init.d/mysql start
0 S root      8537  8498  0  80   0 -  2142 pipe_w 09:56 ?
00:00:00 grep mysql
eth0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 00:22:15:b7:ff:bc  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 22  base 0xa000

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 NNNN:NNNN:NNNN:NNNN:NNNN:NNNN:NNNN:NNNN  prefixlen 64
scopeid 0x0<global>
        inet6 NNNN::NNNN:NNNN:NNNN:NNNN  prefixlen 64  scopeid 0x20<link>
        ether 00:1b:21:b1:cb:bb  txqueuelen 1000  (Ethernet)
        RX packets 9  bytes 1149 (1.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 936 (936.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18  memory 0xfebe0000-fec00000

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 16436
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

As you can see, eth1 has no IPV4 address assigned. Once the server
boots (after the 15 minute mysql timeout ends) ifconfig shows the
following:


eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.x.y.14  netmask 255.255.255.0  broadcast 10.x.y.255
        inet6 NNNN:NNNN:NNNN:NNNN:NNNN:NNNN:NNNN:NNNN  prefixlen 64
scopeid 0x0<global>
        inet6 NNNN::NNNN:NNNN:NNNN:NNNN  prefixlen 64  scopeid 0x20<link>
        ether 00:1b:21:b1:cb:bb  txqueuelen 1000  (Ethernet)
        RX packets 2732  bytes 496650 (485.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1853  bytes 277789 (271.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 18  memory 0xfebe0000-fec00000

In rc.conf I have rc_depend_strict set to YES. I do *not* have
rc_parallel set. I attempted adding "need net" to the depend() section
of the mysql init script with no effect.

The entries for eth1 in /etc/conf.d/net are:

config_eth1="dhcp"
routes_eth1="239.0.0.0/8"

I also tested adding

dhcpcd_eth1="--waitip"

to /etc/conf.d/net again to no avail.

I could add a delay to the mysql script to ensure startup, but I'd
rather figure out why the IP address is not yet available even though
the net.eth1 script has completed. Does anyone have any hints on what
could be going wrong?

Thanks!
-- 
Manuel A. McLure WW1FA <manuel@mclure.org> <http://www.mclure.org>
...for in Ulthar, according to an ancient and significant law,
no man may kill a cat.                       -- H.P. Lovecraft

Re: [gentoo-user] MySQL startup problem - interface does not have an address yet.

[Stroller]

On 23 January 2013, at 18:58, Manuel McLure wrote:
> … eth0 is not configured any more (it used to be used for IPTV
> connectivity to my cable provider for MythTV before they changes their
> system.) eth1 is configured via DHCP to an RFC 1918 10.x.x.x address -
> the DHCP server is always configured to provide the same IP address
> for this system. net.eth1 is symlinked to /etc/runlevels/default.
> 
> The boot process starts, configures eth1, starts LCDd, and then
> attempts to start MySQL - and hangs for about 15 minutes before
> continuing with the boot.
> 
> I added several debugging statements to /etc/init.d/mysql and found
> that even though the log shows that eth1 got its IP address, when
> MySQL attempts to start the interface does not have an IP address
> assigned. It's like the net.eth1 script exits before the interface is
> completely up.

I think, to be rigorous, I would want to test this system (I'm not saying you should keep it this way) by setting the IP address statically for eth1. 

From what I'm understanding, this doesn't sound like a MySQL problem, but a DHCP problem.

You're checking the IP address with ifconfig - I believe that is deprecated. I guess at some point you might replace that in your `echo > test.log` scripts with the newer tools, just to make sure they say the same thing. Maybe this is paranoia, but you know what they say about that.

http://blog.timheckman.net/2011/12/22/why-you-should-replace-ifconfig/

> ...
> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>        inet6 NNNN:NNNN:NNNN:NNNN:NNNN:NNNN:NNNN:NNNN  prefixlen 64
> scopeid 0x0<global>
>        inet6 NNNN::NNNN:NNNN:NNNN:NNNN  prefixlen 64  scopeid 0x20<link>

I don't think these are valid IPv6 addresses. If you're not using IPv6 then I would recommend removing the IPv6 USE flag globally and remerging everything --newuse. This will ensure nothing is depending upon IPv6 or expecting it or waiting for it.

> The entries for eth1 in /etc/conf.d/net are:
> 
> config_eth1="dhcp"
> routes_eth1="239.0.0.0/8"

Is this right?

> I could add a delay to the mysql script to ensure startup, but I'd
> rather figure out why the IP address is not yet available even though
> the net.eth1 script has completed. Does anyone have any hints on what
> could be going wrong?

There are several DHCP clients available in Portage. 

You need to tell us which one you're using.

It wouldn't do any harm to experiment with one or two others.

I think that a busybox version may be installed on some systems, and that this may interfere or misbehave. I don't know if a bug report has been filed for this.6

Stroller.

Re: [gentoo-user] MySQL startup problem - interface does not have an address yet.

[Paul Hartman]

On Wed, Jan 23, 2013 at 12:58 PM, Manuel McLure <manuel@mclure.org> wrote:
> I'm having a problem with booting my Gentoo system due to MySQL
> hanging at startup. My system is up-to-date stable (including
> udev-197, although I believe the problem started while I was still on
> 171) except for the kernel, which is at 3.1.10 because I can't seem to
> get lirc to work correctly on newer kernels. The problem started after
> a large system update just before the udev 171 to 197 migration. Among
> other upgrades dhcpcd was upgraded from 5.2.12 to 5.6.4, net-tools was
> upgraded from 1.60_p20110409135728 to 1.60_p20120127084908, and openrc
> was upgraded from 0.9.8.4 to 0.11.8.

try adding the line "after net" to the depend section of the init.d
script for mysql to cause it not to start until the network is up

Re: [gentoo-user] MySQL startup problem - interface does not have an address yet.

[Kerin Millar]

On 23/01/2013 18:58, Manuel McLure wrote:
> I'm having a problem with booting my Gentoo system due to MySQL
> hanging at startup. My system is up-to-date stable (including
> udev-197, although I believe the problem started while I was still on
> 171) except for the kernel, which is at 3.1.10 because I can't seem to
> get lirc to work correctly on newer kernels. The problem started after
> a large system update just before the udev 171 to 197 migration. Among
> other upgrades dhcpcd was upgraded from 5.2.12 to 5.6.4, net-tools was
> upgraded from 1.60_p20110409135728 to 1.60_p20120127084908, and openrc
> was upgraded from 0.9.8.4 to 0.11.8.

<snip additional info>

> I could add a delay to the mysql script to ensure startup, but I'd
> rather figure out why the IP address is not yet available even though
> the net.eth1 script has completed. Does anyone have any hints on what
> could be going wrong?

I can't determine what the problem is exactly but I can offer some hints ...

You should detail your my.cnf file. By default, the mysql runscript 
depends only on net.lo. There's nothing wrong with that unless you are 
binding to a specific address. Using "skip-name-resolve" is also 
helpful, provided that you don't require host names to be resolved for 
your grant tables to work correctly.

If the nature of your configuration is such that a specific interface 
must be up then you can insert the following lines in /etc/conf.d/mysql:

rc_use="net.eth1"
rc_after="net.eth1"

NOTE: You need not - and should not - edit the mysql runscript directly 
in order to customize its dependencies.

This particular caveat is described by the comments in that file, along 
with several others. I'd suggest taking a look at these comments if you 
haven't already done so.

Cheers,

--Kerin

Re: [gentoo-user] MySQL startup problem - interface does not have an address yet.

[Paul Hartman]

On Thu, Jan 24, 2013 at 11:50 AM, Kerin Millar <kerframil@fastmail.co.uk> wrote:
> If the nature of your configuration is such that a specific interface must
> be up then you can insert the following lines in /etc/conf.d/mysql:
>
> rc_use="net.eth1"
> rc_after="net.eth1"


This is a better solution than what I suggested. Pay no attention to
my previous message. :)

Re: [gentoo-user] MySQL startup problem - interface does not have an address yet.

[Manuel McLure]

On Thu, Jan 24, 2013 at 7:03 AM, Stroller
<stroller@stellar.eclipse.co.uk> wrote:
> I think, to be rigorous, I would want to test this system (I'm not saying you should keep it this way) by setting the IP address statically for eth1.
>
> From what I'm understanding, this doesn't sound like a MySQL problem, but a DHCP problem.

You're correct. When I set the interface statically, MySQL starts up
like a champ.

> You're checking the IP address with ifconfig - I believe that is deprecated. I guess at some point you might replace that in your `echo > test.log` scripts with the newer tools, just to make sure they say the same thing. Maybe this is paranoia, but you know what they say about that.
>
> http://blog.timheckman.net/2011/12/22/why-you-should-replace-ifconfig/

Thanks for the link. I'll definitely check that out.

>> ...
>> eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>        inet6 NNNN:NNNN:NNNN:NNNN:NNNN:NNNN:NNNN:NNNN  prefixlen 64
>> scopeid 0x0<global>
>>        inet6 NNNN::NNNN:NNNN:NNNN:NNNN  prefixlen 64  scopeid 0x20<link>
>
> I don't think these are valid IPv6 addresses. If you're not using IPv6 then I would recommend removing the IPv6 USE flag globally and remerging everything --newuse. This will ensure nothing is depending upon IPv6 or expecting it or waiting for it.

The real addresses are valid, I just didn't want to put them in a
public mailing list message so I edited them. IPv6 on my network
(through a Hurricane Electric tunnel) is working fine.

>> The entries for eth1 in /etc/conf.d/net are:
>>
>> config_eth1="dhcp"
>> routes_eth1="239.0.0.0/8"
>
> Is this right?

Yes. The extra route is so that DLNA/uPnP works correctly on my network.

>> I could add a delay to the mysql script to ensure startup, but I'd
>> rather figure out why the IP address is not yet available even though
>> the net.eth1 script has completed. Does anyone have any hints on what
>> could be going wrong?
>
> There are several DHCP clients available in Portage.
>
> You need to tell us which one you're using.

I did, although it was sort of hidden in the list of upgraded
packages: "dhcpcd was upgraded from 5.2.12 to 5.6.4".

> It wouldn't do any harm to experiment with one or two others.
>
> I think that a busybox version may be installed on some systems, and that this may interfere or misbehave. I don't know if a bug report has been filed for this.6

In any case, I did some more debugging and found that the problem
seems to be an interaction between dhcpcd and IPv6 configuration.  The
latest version of dhcpcd now attempts by default to take over IPv6
stateless configuration instead of letting the kernel take care of it
as happened before. dhcpcd appears to be getting an IPv6 address set
up before the IPv4 configuration is complete, and considers this "good
enough" to background itself instead of waiting for the IPv4
configuration to be complete. The workaround was to add "noipv6rs"
(Disable solicition of IPv6 Router Advertisements) to the
/etc/dhcpcd.conf file so that dhcpcd ignores IPv6 configuration. The
kernel still configures IPv6 correctly, and dhcpcd now waits for the
IPv4 configuration to be complete before backgrounding itself.

-- 
Manuel A. McLure WW1FA <manuel@mclure.org> <http://www.mclure.org>
...for in Ulthar, according to an ancient and significant law,
no man may kill a cat.                       -- H.P. Lovecraft

Re: [gentoo-user] MySQL startup problem - interface does not have an address yet.

[Stroller]

On 24 January 2013, at 18:35, Manuel McLure wrote:

> In any case, I did some more debugging and found that the problem
> seems to be an interaction between dhcpcd and IPv6 configuration.  The
> latest version of dhcpcd now attempts by default to take over IPv6
> stateless configuration instead of letting the kernel take care of it
> as happened before. dhcpcd appears to be getting an IPv6 address set
> up before the IPv4 configuration is complete, and considers this "good
> enough" to background itself instead of waiting for the IPv4
> configuration to be complete. The workaround was to add "noipv6rs"
> (Disable solicition of IPv6 Router Advertisements) to the
> /etc/dhcpcd.conf file so that dhcpcd ignores IPv6 configuration. The
> kernel still configures IPv6 correctly, and dhcpcd now waits for the
> IPv4 configuration to be complete before backgrounding itself.

That sounds queer. You should file a bug.

Stroller.