Red Hat Kernel Panic following bad power cycle

[hankyknot]

I have a client who has for the last four years been running a Red Hat 4.1.18 Server with absolutely zero problems.

Yesterday somebody had a problem with one of the attached Windows PC's and some genius decided to reboot the server. By genius I mean idiot and by reboot I mean pull turn off the power to the box.

When they tried to reboot, they get a kernel panic. This is what the screen says;

Uncompressing Linux... Ok, booting the kernel.
audit(1009912323.849:0): initialized
Red Hat nash version 4.1.18 starting
File descriptor 3 left open
Reading all physical volumes. This may take a while...
/dev/hda: open failed: No medium found
No volume groups found
File descriptor 3 left open
/dev/hda: open failed: No medium found
No volume groups found
File descriptor 3 left open
/dev/hda: open failed: No medium found
No volume groups found
mount: error 6 mounting ext3
mount: error 2 mounting none
switchroot: mount failed: 22
umount /initrd/dev failed: 2
Kernel panic - not syncing: Attempted to kill init!

The server runs the database for their entire POS system so they are now dead in the water.

I'm stuck, I've never come across this before. What do I do to recover this?

(and yes the staff member has been educated to the fact that if they do not know what they are doing they should leave well alone

[Roxoff]

Four years, huh?

This could well be a hardware issue. Actually, it's more likely to be a mainboard issue of the battery variety.

If some idiot has removed the power from the server, and the bios backup battery is flat, they could find all their server settings reset, including settings for the hard disks. You may have to go in and manually edit the hard disk configuration, turn on any raid or other controller specials, maybe even provide some info about physical structure that may have been lost in the outage.

Of course, this might not be the issue. The very first thing I'd do is check the hard disk's connections and turn on the computer with my ear next to hda to be certain that it's spinning.

[hankyknot]

Ok so a quick trip into the BIOS did indeed show the date as 01/01/2002.

Set the date to todays date, unplugged one of the two drives and tried to boot but getting the same error. Reconnected that and unplugged the other drive and still no joy.

Checked the BIOS date again and its retaining the current date.

It does appear we are on the right track, what should I try next?

[hankyknot]

OK so switching the drive controller to compatability mode and unplugging one drive changed nothing, switched drives and it booted so it looks like a combination of dead drive and BIOS battery was behind this one.

New drive and battery will be ordered and we should be cooking on gas in any time.

Thanks for your help Roxoff

[greylica]

OK, so we will run a bit different here for a while. I've also run into problems like this, and I took another approach to solve the problem.
First thing, is to disconnect all HDD cables, and use a live CD from any live CD distro, and if everything goes well, assuming your hardware is fully supported by the chosen distro, ( E.G. no Kernel Panics ) , your machine can be considered ''quasi'' fully functional. Check everything, clean memory modules, clean coolers from dirty, etc, etc, etc. ( Do a full hardware maintenance , PLZ , it's still a production machine that deserves care to never stop)

Disconnect HDD cables ( including power cables ) before this procedure, ensuring your data is safe ( no external interference ).

If yor live distro gets up and running, we can assume your system is OK ( your hardware is FF - Fully Functional).


Assuming everything is OK, let's go to the second step. sometimes, even journalized filesystems run into problems, in those rare cases, you will need to do a full file check, then, turn off your system, connect one of the hdds, and turn on your system using your CD-ROM.

If you are an expert, you can avoid more problems if you do a full backup of your hard drive before those procedures, using an image disk converter software (be carefull again, and again, and again...) <- if you are insecure, don't do this, some ''innocent'' tecnicians run into more problems with ''if/of''...

Once you're up and running, if your live CD mounted your filesystems under, for example /media, you will have to ummount all of your SD(x)1, 2, or anything from your hdd ( assuming X = a for sda, b for sdb , etc), and do a full filesystem scan.
fsck all of your hard drive partitions, and do the same for all hard drives, and then, turn off your computer again, and connect cables the same way as it was connected before (same layout).

Then, turn it on again. If everything is OK, then you can check packages for integrity, ensuring your server is OK, if needed.

Good Vibes !

[Roxoff]

OK so switching the drive controller to compatability mode and unplugging one drive changed nothing, switched drives and it booted so it looks like a combination of dead drive and BIOS battery was behind this one.

New drive and battery will be ordered and we should be cooking on gas in any time.

Thanks for your help Roxoff

No problem. Perhaps you should review the machine - as it's in a corporate environment, you may want to use this as an excuse to refresh some of the hardware. If it's off-line anyway, you may not get another chance.

Let us know how this all comes together.

[previso]

When you get this server running, maybe your client won't object to purchase a second machine for backup purposes, on a different physical location. This time it was a bad HDD and battery. Next time it could be fire, electrical, flooding, you name it. If their data is so important, they must preview server failure.