Server hang/crash/panic?

[suiren97]

Install linux in DL380 G2 server for evaluation purposes.
OS version:
Linux 2.6.18-194.el5 #1 SMP i686 i686 i386 GNU/Linux.
RDBMS:
Oralce 11.2.0.1.0

The server need to reboot by power-off when system just get hang, not able to access, no response even press the reset button. Such scenario happened few times after few days. No message show in /var/log/messages..

Not sure the hang is due to hardware or the memory..

This is my first time using linux and HP DL380 server.

Please help. Thank you.

[dlinger]

suiren97:

Can you tell a bit more about your setup? Distro, graphical vs. CLI, KDE or Gnome ...

Can you run a memtest86 on your memory to see if you have a bad DRAM chip somewhere? A lot of times, especially when using stock kernels, kernel panics and bugons happen when you have a bad ram chip out there. For a large amount of ram, this will take a while . . .

If your ram looks okay, you might try downloading and installing a new kernel (kernel.org), but that's a rather non-trivial thing to do and not for the faint of heart. You might also want to do a lsmod to see what modules are in your kernel. There might be modules you don't need that are causing trouble.

You might also want to run some disk checks, but without knowing your storage layout (RAID? SSDs? Straight HDD setup? -- EXT3, EXT4, Reiser, etc.) I could give you some VERY bad advice.

Good luck with your server, and I'll try to keep my eyes open for a post on this thread.

Cheers.

[Rubberman]

From the kernel information you are running some varient of Red Hat Enterprise Linux 5, probably Oracle's OEL, or CentOS? Or did you install RHEL natively? As dlinger suggested, run a memory test on the system. Also, you can download hardware diagnostic tools from HP to make sure the system is viable. Finally, there may be settings in the BIOS that should be changed to run Linux reliably.

[suiren97]

Hi,

The setup is using GNOME if I'm not mistaken.. how to confirm this?

I'd run the memtest as below:

[root@rtest dev]# dd if=/dev/urandom bs=2204496 of=/tmp/memtest count=1050
1050+0 records in
1050+0 records out
2314720800 bytes (2.3 GB) copied, 1221.89 seconds, 1.9 MB/s
[root@rtest dev]# md5sum /tmp/memtest; md5sum /tmp/memtest; md5sum /tmp/memtest
9ddf4364a1951669a8499256ee7e9c7c /tmp/memtest
9ddf4364a1951669a8499256ee7e9c7c /tmp/memtest
9ddf4364a1951669a8499256ee7e9c7c /tmp/memtest

[root@rtest dev]# fdisk -l

Disk /dev/cciss/c0d0: 72.8 GB, 72829501440 bytes
255 heads, 32 sectors/track, 17432 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 1 17432 71122544 83 Linux

Disk /dev/cciss/c0d1: 36.4 GB, 36414750720 bytes
255 heads, 63 sectors/track, 4427 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d1p1 * 1 13 104391 83 Linux
/dev/cciss/c0d1p2 14 4427 35455455 8e Linux LVM

Disk /dev/cciss/c0d2: 36.4 GB, 36414750720 bytes
255 heads, 32 sectors/track, 8716 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d2p1 1 8716 35561264 83 Linux

Disk /dev/cciss/c0d3: 36.4 GB, 36414750720 bytes
255 heads, 32 sectors/track, 8716 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d3p1 1 8716 35561264 83 Linux



[root@rtest bin]# cat /etc/fstab
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
/dev/cciss/c0d3p1 /u00 ext3 defaults 1 2
/dev/cciss/c0d2p1 /u01 ext3 defaults 1 2
/dev/cciss/c0d0p1 /u02 ext3 defaults 1 2

Currently the system is running; I'm afraid it will happen again as this scenario has been happened 3 times since setup.

As for now, we had enable Magis SysRq Key and trying to configure kdump.

[Rubberman]

That's not how you run memtest. That is a tool that runs separately from the OS that will run various tests on the system RAM. When you boot up and get the screen that says "press <something> to get menu in N seconds" or such. Press the key (usually Enter key) and when the menu comes up, there should be an option for memtest (or memtest86, or something like that). Select that, and the system will boot into the memory test utility.

Anyway, all you did was dump random memory into a file and then run the md5sum program against that. Not what we were getting at...