mysql server will not start on boot

[nomadicME]

Here is an exerpt from /var/log/mysql/mysqld.err:

120715 14:08:03 [ERROR] Can't start server: Bind on TCP/IP port: Cannot assign requested address 
120715 14:08:03 [ERROR] Do you already have another mysqld server running on port: 3306 ? 
120715 14:08:03 [ERROR] Aborting

Here is an exerpt from /etc/mysql/my.cnf

# The following options will be passed to all MySQL clients 
[client] 
#password	 = your_password 
port	 = 3306 
socket	 = /var/run/mysqld/mysqld.sock 

[mysql] 
character-sets-dir=/usr/share/mysql/charsets 
default-character-set=utf8 

[mysqladmin] 
character-sets-dir=/usr/share/mysql/charsets 
default-character-set=utf8 

[mysqlcheck] 
character-sets-dir=/usr/share/mysql/charsets 
default-character-set=utf8 

[mysqldump] 
character-sets-dir=/usr/share/mysql/charsets 
default-character-set=utf8 

[mysqlimport] 
character-sets-dir=/usr/share/mysql/charsets 
default-character-set=utf8 

[mysqlshow] 
character-sets-dir=/usr/share/mysql/charsets 
default-character-set=utf8 

[myisamchk] 
character-sets-dir=/usr/share/mysql/charsets 

[myisampack] 
character-sets-dir=/usr/share/mysql/charsets 

# use [safe_mysqld] with mysql-3 
[mysqld_safe] 
err-log	 = /var/log/mysql/mysql.err 

# add a section [mysqld-4.1] or [mysqld-5.0] for specific configurations 
[mysqld] 
character-set-server	 = utf8 
user = mysql 
port = 3306 
socket = /var/run/mysqld/mysqld.sock 
pid-file = /var/run/mysqld/mysqld.pid 
log-error = /var/log/mysql/mysqld.err 
basedir = /usr 
datadir = /var/lib/mysql 
skip-external-locking 
key_buffer = 16M 
max_allowed_packet = 1M 
table_open_cache = 64 
sort_buffer_size = 512K 
net_buffer_length = 8K 
read_buffer_size = 256K 
read_rnd_buffer_size = 512K 
myisam_sort_buffer_size = 8M 
language = /usr/share/mysql/english 

# security: 
# using "localhost" in connects uses sockets by default 
# skip-networking 
#bind-address	 = 127.0.0.1 
bind-address = 192.168.2.5

The bind-address is set to be assigned to eth0 by wicd and wicd does start before mysql tries to start. The firewall is shutdown. Here are the permissions on /var/run/mysqld/:

drwxr-xr-x 2 mysql mysql 4096 Jul 15 14:12 . 
drwxr-xr-x 9 root root 4096 Jul 15 14:10 .. 
-rw-rw---- 1 mysql mysql 5 Jul 15 14:12 mysqld.pid 
srwxrwxrwx 1 mysql mysql 0 Jul 15 14:12 mysqld.sock

Here is the output from "rc-status -a":

Runlevel: default 
syslog-ng [ started ] 
vixie-cron [ started ] 
ufw [ started ] 
netmount [ started ] 
ntpd [ started ] 
clamd [ started ] 
mysql [ started ] 
sshd [ started ] 
apache2 [ started ] 
udev-postmount [ started ] 
local [ started ] 
Runlevel: boot 
hwclock [ started ] 
modules [ started ] 
fsck [ started ] 
root [ started ] 
mtab [ started ] 
localmount [ started ] 
sysctl [ started ] 
bootmisc [ started ] 
procfs [ started ] 
net.lo [ started ] 
termencoding [ started ] 
urandom [ started ] 
wicd [ started ] 
swap [ started ] 
keymaps [ started ] 
hostname [ started ] 
Runlevel: shutdown 
savecache [ stopped ] 
killprocs [ stopped ] 
mount-ro [ stopped ] 
Runlevel: sysinit 
dmesg [ started ] 
udev [ started ] 
devfs [ started ] 
Dynamic Runlevel: hotplugged 
Dynamic Runlevel: needed 
udev-mount [ started ] 
dbus [ started ] 
Dynamic Runlevel: manual

My current workaround has been to take this service out of the default run level and then start it manually once I'm logged in, which works fine btw. It just will not connect automatically on boot, instead it hangs on "starting mysql" (if I leave mysql in the default run level).

Any ideas? Thanks.

[atreyu]

that seems strange that Gentoo would create the startup script for MySQL but not have it configured to run after wicd starts networking. anyway, i do not doubt you that it does not start up properly.

i do not have a Gentoo box at my disposal - so can you tell me: how does Gentoo init/startup work? If it is Sys V init-based, then you should have symlinks in the appropriate runlevel dirs; /etc/rc.d/rcX.d/, where X = the runlevel you want a daemon to run in (typically 2 or 3, or 5 if graphical). the symlinks point to the corresponding script in /etc/rc.d/init.d/ (or /etc/init.d/).

for example, on my RHEL 4 box, I have MySQL installed:

# find /etc/rc.d/|grep mysql|sort
/etc/rc.d/init.d/mysqld
/etc/rc.d/rc0.d/K36mysqld
/etc/rc.d/rc1.d/K36mysqld
/etc/rc.d/rc2.d/S64mysqld
/etc/rc.d/rc3.d/S64mysqld
/etc/rc.d/rc4.d/S64mysqld
/etc/rc.d/rc5.d/S64mysqld
/etc/rc.d/rc6.d/K36mysqld

this says that mysql script /etc/rc.d/init.d/mysqld will be called w/the "start" argument in run leves 2,3,4 and 5, and will be called with the "stop" argument in run levels 0, 1 and 6.

the S means "start", the number (S64mysqld) means the order in which the script is processed. So if another script has an earlier number (e.g., S10network) then that daemon would get started first. so maybe you need to make wicd start up sooner than mysql. that is IF your system uses SysV Init, and IF wicd uses SysV Init to start up.

If your system uses something else, or wicd has a special start-up method, let us know - or perhaps a Gentoo pro here will read this and chime in.

[nomadicME]

atreyu,

Thank you for your help. Unfortunately/fortunately gentoo does not use a Sys V init system. They use OpenRC. OpenRC allows you to interactively start processes during boot as a way to get out of a jam. This is exactly how I get around the problem when I have mysql set to start in the default run level. It hangs, then I reboot, push "I" and start all other processes manually, but skip mysql. My point in describing all of this is that you see the start up order. I manually allow the wicd process to start a few steps before it asks me if I want to start mysql. So I'm fairly certain that wicd has started by the time mysql starts (or doesn't start in my case).

Nevertheless, I did a little digging in /etc/runlevels and here is what I found:

$ ls /etc/runlevels/boot/

drwxr-xr-x 2 root root 4096 Mar 23 07:53 .
drwxr-xr-x 6 root root 4096 Nov  8  2011 ..
lrwxrwxrwx 1 root root   20 Nov 12  2011 bootmisc -> /etc/init.d/bootmisc
lrwxrwxrwx 1 root root   16 Nov 12  2011 fsck -> /etc/init.d/fsck
lrwxrwxrwx 1 root root   20 Nov 12  2011 hostname -> /etc/init.d/hostname
lrwxrwxrwx 1 root root   19 Nov 12  2011 hwclock -> /etc/init.d/hwclock
lrwxrwxrwx 1 root root   19 Nov 12  2011 keymaps -> /etc/init.d/keymaps
lrwxrwxrwx 1 root root   22 Nov 12  2011 localmount -> /etc/init.d/localmount
lrwxrwxrwx 1 root root   19 Nov 12  2011 modules -> /etc/init.d/modules
lrwxrwxrwx 1 root root   16 Nov 12  2011 mtab -> /etc/init.d/mtab
lrwxrwxrwx 1 root root   18 Nov 12  2011 net.lo -> /etc/init.d/net.lo
lrwxrwxrwx 1 root root   18 Nov 12  2011 procfs -> /etc/init.d/procfs
lrwxrwxrwx 1 root root   16 Nov 12  2011 root -> /etc/init.d/root
lrwxrwxrwx 1 root root   16 Nov 12  2011 swap -> /etc/init.d/swap
lrwxrwxrwx 1 root root   21 Mar 23 07:53 swapfiles -> /etc/init.d/swapfiles
lrwxrwxrwx 1 root root   18 Nov 12  2011 sysctl -> /etc/init.d/sysctl
lrwxrwxrwx 1 root root   24 Nov 12  2011 termencoding -> /etc/init.d/termencoding
lrwxrwxrwx 1 root root   19 Nov 12  2011 urandom -> /etc/init.d/urandom
lrwxrwxrwx 1 root root   16 Jan 24 15:57 wicd -> /etc/init.d/wicd

$ ls /etc/runlevels/default
total 8

drwxr-xr-x 2 root root 4096 Mar 29 19:18 .
drwxr-xr-x 6 root root 4096 Nov  8  2011 ..
lrwxrwxrwx 1 root root   19 Nov 18  2011 anacron -> /etc/init.d/anacron
lrwxrwxrwx 1 root root   19 Feb  7 23:37 apache2 -> /etc/init.d/apache2
lrwxrwxrwx 1 root root   15 Nov 17  2011 atd -> /etc/init.d/atd
lrwxrwxrwx 1 root root   17 Nov 18  2011 clamd -> /etc/init.d/clamd
lrwxrwxrwx 1 root root   22 Nov 17  2011 consolekit -> /etc/init.d/consolekit
lrwxrwxrwx 1 root root   16 Dec  7  2011 dbus -> /etc/init.d/dbus
lrwxrwxrwx 1 root root   17 Nov 12  2011 local -> /etc/init.d/local
lrwxrwxrwx 1 root root   17 Mar 29 01:46 mysql -> /etc/init.d/mysql
lrwxrwxrwx 1 root root   24 Dec 21  2011 mytv_backend -> /etc/init.d/mytv_backend
lrwxrwxrwx 1 root root   20 Nov 12  2011 netmount -> /etc/init.d/netmount
lrwxrwxrwx 1 root root   16 Nov 18  2011 ntpd -> /etc/init.d/ntpd
lrwxrwxrwx 1 root root   26 Feb  9 22:02 postgresql-9.1 -> /etc/init.d/postgresql-9.1
lrwxrwxrwx 1 root root   16 Nov 12  2011 sshd -> /etc/init.d/sshd
lrwxrwxrwx 1 root root   26 Nov 12  2011 udev-postmount -> /etc/init.d/udev-postmount
lrwxrwxrwx 1 root root   15 Jan 30 13:56 ufw -> /etc/init.d/ufw
lrwxrwxrwx 1 root root   22 Nov 18  2011 vixie-cron -> /etc/init.d/vixie-cron
lrwxrwxrwx 1 root root   27 Dec  4  2011 wakeonlanconfig -> /etc/init.d/wakeonlanconfig
lrwxrwxrwx 1 root root   15 Nov 14  2011 xdm -> /etc/init.d/xdm

Which tells me they are in two different runlevels: boot and default, and the boot run level comes before the default runlevel.

[atreyu]

See the Gentoo documentation here regarding init control. In particular, check out the depend function and controlling the order.

[nomadicME]

Update:

I've narrowed the problem down to using wicd with mysql binding to an externally accessible ip address. If I set the bind-address = 127.0.0.1 in /etc/mysql/my.cnf then everything works as expected, but I cannot reach my mysql server from anywhere else on my network. If anyone has the wicd/mysql bound to external ip configuration working please chime in. This configuration was working fine on this machine until an upgrade around the beginning of 2012.

I've also found another workaround. I really don't need the flexibility of wicd on this machine as it is a server that is rarely rebooted or shutdown and is always plugged into the ethernet cable. So I configured the network manually in /etc/conf.d/net and added /etc/init.d/net.eth0 to the boot runlevel. This configuration works fine so it is definately a problem with wicd.

I wouldn't consider this matter solved, however. If you have any ideas please let us know. Thanks.