Unexpected restart - RHEL 5.3 ?

[uday123]

Hi,

One of my Linux server restarted unexpectedly and below are log messages from the /var/log/messages. My Server is running on RHEL 5.3 and it is a oracle application server. Restart happened at 13:34.Please review the below logs:

Sep 17 11:59:50 TESTSERVER1 kernel: frmweb[30676]: segfault at 000000006974655e rip 00000000f7cec2c1 rsp 00000000ffd0a9c8 error 4

Sep 17 12:00:49 TESTSERVER1 kernel: frmweb[30759]: segfault at 000000006974655e rip 00000000f7d352c1 rsp 00000000ffd53a18 error 4


Sep 17 12:33:08 TESTSERVER1 : error getting update info: Cannot retrieve repository metadata (repomd.xml) for repository: citrix. Please verify its path and try again


Sep 17 13:33:08 TESTSERVER1 : error getting update info: Cannot retrieve repository metadata (repomd.xml) for repository: citrix. Please verify its path and try again

Sep 17 13:34:58 TESTSERVER1 syslogd 1.4.1: restart.

Sep 17 13:34:58 TESTSERVER1 kernel: klogd 1.4.1, log source = /proc/kmsg started.

Sep 17 13:34:58 TESTSERVER1 kernel: Linux version 2.6.18-128.el5 (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Dec 17 11:41:38 EST 2008

Sep 17 13:34:58 TESTSERVER1 kernel: Command line: ro root=LABEL=/ rhgb quiet crashkernel=128M at 16M

Sep 17 13:34:58 TESTSERVER1 kernel: BIOS-provided physical RAM map:
Sep 17 13:34:58 TESTSERVER1 kernel: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)

Sep 17 13:34:58 TESTSERVER1 kernel: BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)

Sep 17 13:34:58 TESTSERVER1 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)

Sep 17 13:34:58 TESTSERVER1 kernel: BIOS-e820: 0000000000100000 - 00000000ef3f6800 (usable)

Sep 17 13:34:58 TESTSERVER1 kernel: BIOS-e820: 00000000ef3f6800 - 00000000efffb000 (reserved)

Sep 17 13:34:58 TESTSERVER1 kernel: BIOS-e820: 00000000efffc000 - 00000000f0000000 (reserved)

Sep 17 13:34:58 TESTSERVER1 kernel: BIOS-e820: 0000000100000000 - 000000018e200000 (usable)

Sep 17 13:34:58 TESTSERVER1 kernel: DMI 2.4 present.


What might be the reason for unexpected restart? Any clues?

Any help will be highly appreciated.

[waterhead]

Just a wild guess here, but it may be a memory error. When you boot, along with the kernel choices, is there the option for memtest? Give that a try. If not, boot from an installation or LiveCD and use the memtest option on that.

[uday123]

Thanks Paul.

Actually my Linux m/n is a VM (virtual m/n) running on XEN (similar to VMWare). we have many servers running like this. But unexpected restart happened on one m/n only. Any other clues? please let me know if u need any other log files for analysis...

Thanks in advance.

[waterhead]

Nope, sorry. I can't offer anything.

[ursusca]

Hello,

Is it 64-bit Architecture of 32-bit?
Try to add the kernel line settings in your grub.conf file:
for 64-bit RHEL 5
divider=10 notsc iommu=soft elevator=noop

and for a 32-bit RHEL 5:
divider=10 clocksource=acpi_pm iommu=soft elevator=noop

But I think you have problems with Memory Optimization. Please post content of /etc/sysctl.conf and /etc/security/limits.conf files and result of sar command:
sar -r -n DEV -f /var/log/sa/sa<day when it happened> In your case as I see from your log file it's 17

[uday123]

Thanks ursusca.

Mine is 64bit but I dont have those entries in /etc/grub.conf file. The entries are:
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.18-128.1.10.el5.xs5.5.0.51xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.1.10.el5.xs5.5.0.51
module /vmlinuz-2.6.18-128.1.10.el5.xs5.5.0.51xen ro root=LABEL=/ rhgb quiet crashkernel=128M@16M
module /initrd-2.6.18-128.1.10.el5.xs5.5.0.51xen.img
title Red Hat Enterprise Linux Server (2.6.18-128.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-128.el5 ro root=LABEL=/ rhgb quiet crashkernel=128M AT 16M (FYI: here AT is a symbol at)
initrd /initrd-2.6.18-128.el5.img

And /etc/security/limits.conf-- all the entries are commented.
sar command output:
kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
01:00:01 PM 309840 5633904 94.79 963176 2079556 12289716 0 0.00 0
01:10:01 PM 259392 5684352 95.64 963184 2079712 12289716 0 0.00 0
01:20:01 PM 231992 5711752 96.10 963188 2080272 12289716 0 0.00 0
01:30:01 PM 102572 5841172 98.27 963192 2080480 12289716 0 0.00 0

Restart happened at 13:34 (i.e. 01:34PM). If we look at above sar output %memused value is 98.27% at 01:30:01PM. If I am not wrong, may be by 01:34PM all the memory must have used and there is no memory (SWAP/RAM)...hence kernal crashed for frmweb service.

Please analyze from ur end and let me know ur valuable inputs.

once again thanks a lot for guidence.

-Uday.

[ursusca]

Hello again,

You forgot about /etc/sysctl.conf file. And I forgot about free command. please post result of "free -tom" command.

You can delete "crashkernel=128M@16M" kernel line settings from grub.conf file and add "divider=10 notsc iommu=soft elevator=noop". I usually use "elevetor=deadline" kernel line setting on real hots and "divider=10 notsc iommu=soft elevator=noop" on virtual hosts.

[jkpolum]

I have the exact same problem on my servers running RHEL 5.3. Has anyone confirmed a solution based on the information above?

[oz]

Hello and welcome!

This thread is over 2 years old so I'm going to lock it down, but please feel free to start a new thread of your own if you are having any problems with Linux.

Thanks.