Suddenly missing apachectl?

[bradsucks]

After 44 days of uptime, I went to access my web server and it didn't respond. I SSH'd in and found apache wasn't running. I tried to start it and got this error:

Starting web server: Apache2env: /usr/sbin/apache2ctl: No such file or directory

Huh? I went and had a look at the /usr/sbin/apache2ctl and saw this:

lrwxrwxrwx 1 root root 27 2005-11-26 16:52 apache2ctl -> libkommanderplugin.so.0.0.0

I can't run it. The last thing in my apache error log is this:

[Sun Jan 29 04:32:31 2006] [notice] caught SIGTERM, shutting down

I've tried apt-get removing apache2 and reinstalling it, but get the same thing. Anyone know what the heck is going on? I haven't changed anything on the machine and nobody else has access to it.

[jledhead]

is this a publicly accessible site/machine. are there any other services open to the public on this machine?

only reason I ask is I don't know what that file is that apache2ctl is symlinked to.

if this is a publicly accessible machine you might want to check for rootkits and the like. maybe try chkrootkit or something.

sounds like ssh is open on this machine to the public, is it the most recent/updated version. reason I ask is I have been hit several times with rootkits on vulnerable versions of openssh

[bradsucks]

The linux box is behind a linksys router firewall and the only open ports are some bittorrent ones, 80 and 21. So SSH isn't accessible from the outside.

I ran chkroot on the server though and it didn't find anything. Any other ideas? Or maybe suggestions on how I can get it back up and running without having to reinstall linux? Thanks.

[jledhead]

what distro is this?

what about /usr/sbin/apache2

is that available?

and what about your webfiles, are they missing or is it just this strange symlink causing the problem?

also try ps aux and make sure there is no apache pid still running, if it is then kill it and try your reinstall again. report back your findings

[bradsucks]

what distro is this?

Debian 3.1.

what about /usr/sbin/apache2

It was there, I ran it and apache started and everything seems to work all of a sudden. apache2ctl is still unavailable though and the system no longer starts Apache when it reboots.

and what about your webfiles, are they missing or is it just this strange symlink causing the problem?

Everything else seems intact. Samba's working fine, ftp's working fine, web files are all still there.

also try ps aux and make sure there is no apache pid still running, if it is then kill it and try your reinstall again. report back your findings

There's no apache pid. The only weird thing I see in ps aux is this:

root 1232 0.0 0.5 2904 1400 ? Ss 02:57 0:00 /bin/sh /etc/gdm/XKeepsCrashing
root 1249 0.0 0.1 1472 344 ? S 02:57 0:00 /usr/lib/gdm/gdmopen -l /bin/sh -c /etc/gdm/XKeepsCrashing -noopen
root 1250 0.0 0.5 2904 1400 tty7 Ss+ 02:57 0:00 /bin/sh /etc/gdm/XKeepsCrashing -noopen
root 1270 0.0 0.4 2556 1072 tty7 S+ 02:57 0:00 /usr/bin/whiptail --yesno I cannot start the X server (your graphical interface). It is likel


I don't know if this was there before or not, but it looks a bit funny to me.

[jledhead]

it does look funny. just sounds like X is crashing.

Is this a machine that you use X on? or is it just a server and you never actually use the desktop?

if you never use the desktop and manage it thru ssh then why not have gdm not even start, might be a little lighter on your system. you could edit /etc/X11/default-display-manager, and just comment out the one line in there.

also, copy/pasted for your pleasure (also from a debian system, should work)

#!/bin/sh
#
# Copyright 2000-2005 The Apache Software Foundation or its licensors, as
# applicable.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#
# Apache control script designed to allow an easy command line interface
# to controlling Apache. Written by Marc Slemko, 1997/08/23
#
# The exit codes returned are:
# XXX this doc is no longer correct now that the interesting
# XXX functions are handled by httpd
# 0 - operation completed successfully
# 1 -
# 2 - usage error
# 3 - httpd could not be started
# 4 - httpd could not be stopped
# 5 - httpd could not be started during a restart
# 6 - httpd could not be restarted during a restart
# 7 - httpd could not be restarted during a graceful restart
# 8 - configuration syntax error
#
# When multiple arguments are given, only the error from the _last_
# one is reported. Run "apachectl help" for usage info
#
ARGV="$@"
#
# |||||||||||||||||||| START CONFIGURATION SECTION ||||||||||||||||||||
# -------------------- --------------------
#
# the path to your httpd binary, including options if necessary
HTTPD='/usr/sbin/apache2'
#
# pick up any necessary environment variables
if test -f /etc/apache2/envvars; then
. /etc/apache2/envvars
fi
#
# a command that outputs a formatted text version of the HTML at the
# url given on the command line. Designed for lynx, however other
# programs may work.
LYNX="www-browser -dump"
#
# the URL to your server's mod_status status page. If you do not
# have one, then status and fullstatus will not work.
STATUSURL="http://localhost:80/server-status"
#
# Set this variable to a command that increases the maximum
# number of file descriptors allowed per child process. This is
# critical for configurations that use many file descriptors,
# such as mass vhosting, or a multithreaded server.
ULIMIT_MAX_FILES="ulimit -S -n `ulimit -H -n`"
# -------------------- --------------------
# |||||||||||||||||||| END CONFIGURATION SECTION ||||||||||||||||||||

# Set the maximum number of file descriptors allowed per child process.
if [ "x$ULIMIT_MAX_FILES" != "x" ] ; then
$ULIMIT_MAX_FILES
fi

ERROR=0
if [ "x$ARGV" = "x" ] ; then
ARGV="-h"
fi

case $ARGV in
start|stop|restart|graceful)
$HTTPD -k $ARGV
ERROR=$?
;;
startssl|sslstart|start-SSL)
$HTTPD -k start -DSSL
ERROR=$?
;;
configtest)
$HTTPD -t
ERROR=$?
;;
status)
$LYNX $STATUSURL | awk ' /process$/ { print; exit } { print } '
;;
fullstatus)
$LYNX $STATUSURL
;;
*)
$HTTPD $ARGV
ERROR=$?
esac

exit $ERROR

[bradsucks]

Thanks a lot, that apachectl worked perfectly when I put it in. Apache starts now when the box is rebooted.

I don't use the desktop usually, but it's something I've been meaning to play with so I can get a feel for the linux desktop. (Just ordered a KVM for there actually.)

X was working the last time I had a monitor attached, probably a month ago or so. I have no idea what would cause it to start crashing all of a sudden. I checked the syslogs and all it says about X is:

Jan 30 03:41:11 localhost gdm[1221]: gdm_slave_xioerror_handler: Fatal X error - Restarting :0
Jan 30 03:41:17 localhost kernel: keyboard: Timeout - AT keyboard not present?(f3)
Jan 30 03:41:17 localhost kernel: keyboard: Timeout - AT keyboard not present?(f4)
Jan 30 03:41:19 localhost gdm[1232]: gdm_slave_xioerror_handler: Fatal X error - Restarting :0
Jan 30 03:41:24 localhost kernel: keyboard: Timeout - AT keyboard not present?(f3)
Jan 30 03:41:24 localhost kernel: keyboard: Timeout - AT keyboard not present?(f4)
Jan 30 03:41:26 localhost gdm[1235]: gdm_slave_xioerror_handler: Fatal X error - Restarting :0
Jan 30 03:41:26 localhost gdm[1209]: deal_with_x_crashes: Running the XKeepsCrashing script

[TheBigPhish]

those time events coincide 2 seconds after you have a BIOS poll of the keyboard, and your computer faults it...

I've got logs full of those faults on the fileserver that i maintain (headless boxes), but i've never seen one crash X like that. Does X crash if you keep a keyboard plug in?

[bradsucks]

I plugged a keyboard in and it got rid of the keyboard errors, X is still down though:

Jan 30 16:57:04 localhost afpd[1158]: uam: loading (/usr/lib/netatalk/uams_dhx.so)
Jan 30 16:57:04 localhost afpd[1158]: uam: uam not found (status=-1)
Jan 30 16:57:04 localhost afpd[1158]: uam: loading (/usr/lib/netatalk/uams_clrtxt.so)
Jan 30 16:57:04 localhost afpd[1158]: uam: uams_clrtxt.so loaded
Jan 30 16:57:04 localhost afpd[1158]: uam: loading (/usr/lib/netatalk/uams_randnum.so)
Jan 30 16:57:04 localhost afpd[1158]: uam: uam not found (status=-1)
Jan 30 16:57:04 localhost afpd[1158]: uam: "Cleartxt Passwrd" available
Jan 30 16:57:04 localhost kernel: apm: BIOS version 1.2 Flags 0x07 (Driver version 1.16)
Jan 30 16:57:08 localhost gdm[1227]: gdm_slave_xioerror_handler: Fatal X error - Restarting :0
Jan 30 16:57:14 localhost gdm[1236]: gdm_slave_xioerror_handler: Fatal X error - Restarting :0
Jan 30 16:57:20 localhost gdm[1245]: gdm_slave_xioerror_handler: Fatal X error - Restarting :0
Jan 30 16:57:20 localhost gdm[1211]: deal_with_x_crashes: Running the XKeepsCrashing script

[TheBigPhish]

i would look at the graphical setup for this system. Is your display driver up to par? is it configured correctly? Is the resolution set for a card & monitor compatible resolution, as well as the refresh rate? If you stay at the CLI level, do they keep popping up? Did you make any changed to the graphics subsystem for this machine?