Sudden reboot

Printable View

09-16-2010
Freston

Sudden reboot

I was logged in over ssh to my Slackware server (13.0) just editing some PHP files when all of a sudden I lost connection, and I heard the BIOS beep from my server. A few minutes later I could log in again.

As the dog owner told the police officer: "It's never done that before" :rolleyes:

But it's true, I had an uptime measured in months, only took it down one time to add a new disk.

Now I'm going through the logs to see who or what caused it, but neither syslog nor messages gives any indication. As far as I can tell, it was happily running one moment, and booting the next. Not even a trace of 'the system is going down' or such. And judging by the fsck entries, it didn't unmount the filesystems properly.

The server is called 'kannix', and the system came back up at 08:44:27 (local system time)

This are the /var/log/syslog entries around the time of the event. I edited out the iptable messages:

Code:

Sep 14 21:12:10 kannix last message repeated 4 times Sep 14 21:12:12 kannix last message repeated 2 times Sep 14 21:12:15 kannix last message repeated 5 times Sep 16 08:44:27 kannix kernel: KERNEL supported cpus: Sep 16 08:44:27 kannix kernel: Intel GenuineIntel Sep 16 08:44:27 kannix kernel: AMD AuthenticAMD

This are the /var/log/messages entries around the time of the event:

Code:

Sep 16 08:51:48 kannix -- MARK -- Sep 16 08:11:48 kannix -- MARK -- Sep 16 08:31:48 kannix -- MARK -- Sep 16 08:34:37 kannix sshd[11272]: Accepted publickey for freston from 192.168.x.x port 50870 ssh2 Sep 16 08:39:02 kannix sshd[11301]: Accepted publickey for freston from 192.168.x.x port 52616 ssh2 Sep 16 08:44:27 kannix syslogd 1.4.1: restart. Sep 16 08:44:27 kannix kernel: klogd 1.4.1, log source = /proc/kmsg started. Sep 16 08:44:27 kannix kernel: Linux version 2.6.29.6-smp (root@midas) (gcc version 4.3.3 (GCC) ) #2 SMP Mon Aug 17 00:52:54 CDT 2009
09-16-2010
Roxoff

The first thing I'd do is check the physical hardware. This sort of thing happens when your system overheats - so whip the lid off the box and make sure all the fans are turning. If you're shutting it down to do this, it might be a good time to remove any buildup of dust and smeg around them too. A fan doesn't have to fail for it to become inefficient.

You can also look for other physical problems like excessive heat coming from your power supply, correctly seated memory, loose power connections on the mainboard, hard disks, etc. These things could work loose over time.

Also make sure that the kit is running with a surge-protected power supply, as it could just have been a voltage spike that caused the instant power-down/reboot.

If it's none of these, check back over your most recent system config changes. Linux systems don't reboot by themselves with no help (unlike some other OS's...)
09-16-2010
Freston

Quote:

Originally Posted by Roxoff

This sort of thing happens when your system overheats

It happily runs below 60 degrees, and although it's not the cleanest I kinda cleaned it a little some three/four months ago. So it doesn't have too many spiderwebs ;-)

Quote:

Originally Posted by Roxoff

Also make sure that the kit is running with a surge-protected power supply, as it could just have been a voltage spike that caused the instant power-down/reboot.

That doesn't seem to far fetched. It's not protected. They are using powertools next door, and I don't mean 'just' a drill, a contractor is working away what appears to be centuries of deferred maintenance.

I don't know very much about electricity, but it appears they using the mains, could this cause a spike or drop in voltage?

Quote:

Originally Posted by Roxoff

If it's none of these, check back over your most recent system config changes.

Thinking about it, it went like this:
I was typing, and all of a sudden my letters didn't show up on the screen anymore. Within two seconds I heard the beep. That's much to quick for a regular shutdown/reboot imo.

And changes, yes I make changes to system configs, but not without testing and not today.

Quote:

Originally Posted by Roxoff

Linux systems don't reboot by themselves with no help (unlike some other OS's...)

Hahaha, yeah I know.
09-17-2010
Rubberman

It sounds like it may have been a power glitch. I always keep my critical workstations/servers plugged into a UPS. Every once in awhile there is a quick power drop that doesn't even flicker (or barely) the lights or backlit LCD displays, yet the UPS will beep and cycle its fan and my non-powerprotected phone will indicate it lost power as well. That's when I am thankful for a good UPS/surge suppressor... :-)
09-17-2010
Freston

Thanks for your responses, I feel a lot less worried about the machine now. It appears to run very nicely again, not a single issue after this reboot. Temp at +44.5°C and the logs quiet as usual.

It's good to know there is a correlation between server uptime and the availability of electricity... and I know what to ask for christmas ;-)
09-17-2010
MikeTbob

Quote:

Originally Posted by Freston

Thanks for your responses, I feel a lot less worried about the machine now. It appears to run very nicely again, not a single issue after this reboot. Temp at +44.5°C and the logs quiet as usual.

It's good to know there is a correlation between server uptime and the availability of electricity... and I know what to ask for christmas ;-)

Your very own nuclear power plant, right?;)
09-17-2010
Freston

Quote:

Originally Posted by MikeTbob

Your very own nuclear power plant, right? ;)

That, or a hydroelectric dam in the river here. I'd like it to be a surprise :D
09-17-2010
Rubberman

Or a good supply of what one colleague of mine calls "magic flooby dust". It will keep anything running, but without it, nothing works.
09-17-2010
Freston

Quote:

Originally Posted by Rubberman

Or a good supply of what one colleague of mine calls "magic flooby dust". It will keep anything running, but without it, nothing works.

Hahahaha "magic flooby dust"; recommended for ze komputer profezional.
So that's what you guru's use :p

Reminds me, during the interview for my current job, they asked me a particularly difficult question to which I didn't know the answer. I told them <pokerface> that I'd rely on beginners luck, and it would all work out. Now they still say it's 'beginners luck' everytime I do something right :?
The correct answer to the question, I know now, would have been "magic flooby dust" :p
09-17-2010
Rubberman

Yeah! I always keep some around and just sprinkle a few flakes on the system that is having a problem... Ipso, presto - fixed! :lol: