kernel panic help

**Blue|Fusion** · 05-21-2011

Hi everybody!

I have been having issues with a Red Hat Enterprise 5 server for a while now. I wanted to squeeze a little more performance out of it using the latest and greatest kernel with a custom configuration.

I used the settings that I generally use on my personal computers, except change a few settings (i.e. preemption and clock rate) to a server-oriented configuration. I configured most if not all of the drivers needed for this server to be built-in to the kernel image.

The kernel I tried using is 2.6.39 right from kernel.org, and like I said, this is on a RHEL5 box that is completely up to date.

Attached are a screenshot of the kernel panic I am getting when connected to a KVM console and the gzipped custom .config.

Screenshot also available in full size here

At first I thought it may be an issue with USB support as the USB driver was initiated just before this kernel panic, but after disabling USB support, this kernel panic still remained. Without an actual error of why the kernel is panicking, I am at a loss.

Any ideas or suggestions greatly appreciated.

**Rubberman** · 05-22-2011

There were significant internal changes made to the kernel after 2.6.3something (32 I think, but don't remember for sure). This means that ALL system libraries have to be replaced, and probably all your applications will have to be rebuilt as well. Just replacing the kernel and kernel modules is not sufficient to get this to work. If you want a newer kernel for RHEL 5.x, then you might try a 2.6.32 kernel, which should be much more compatible.

**Blue|Fusion** · 05-22-2011

Well, this is something I also considered. But to do some trial and error during the day and not on a live server, I installed CentOS5 on VMWare, updated it to the same as the actual server, and compiled the kernel. I got one error telling me it was unable to boot and gave me a similar, although still different kernel panic with useful information. A quick google search showed that I had to enable CONFIG_SYSFS_DEPRECATED_V2. I did that and 2.6.39 worked without issue inside the VM. I have yet to reboot the server with that new option enabled in the kernel.

I copied the config I was using on the live server. All I had to change was the ethernet driver from e1000e to e1000 on the VM. The rest was the same.

Also, I should provide some additional information about this server hardware.

Processor is a 4 core Xeon E5430. No HyperThreading, but it is Core2 based with 64-bit @ 2.66GHz. 8GB of EEC RAM

Code:

[root@quasar ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
stepping        : 6
cpu MHz         : 2666.817
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 lahf_lm
bogomips        : 5333.63
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

Code:

[root@quasar ~]# uname -a
Linux quasar.(redacted).com 2.6.18-238.9.1.el5 #1 SMP Fri Mar 18 12:42:39 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Code:

[root@quasar ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.6 (Tikanga)

Code:

[root@quasar ~]# lspci -v
00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller Hub (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, fast devsel, latency 0
        Capabilities: [50] Power Management version 2
        Capabilities: [58] MSI: Enable- Count=1/2 Maskable- 64bit-
        Capabilities: [6c] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting

00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 2-3 (rev b1) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=05, sec-latency=0
        I/O behind bridge: 00002000-00002fff
        Memory behind bridge: d8000000-d81fffff
        Prefetchable memory behind bridge: 00000000d8300000-00000000d83fffff
        Capabilities: [50] Power Management version 2
        Capabilities: [58] MSI: Enable+ Count=1/2 Maskable- 64bit-
        Capabilities: [6c] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp

00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 4-5 (rev b1) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
        Capabilities: [50] Power Management version 2
        Capabilities: [58] MSI: Enable+ Count=1/2 Maskable- 64bit-
        Capabilities: [6c] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp

00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 6-7 (rev b1) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=07, subordinate=07, sec-latency=0
        Capabilities: [50] Power Management version 2
        Capabilities: [58] MSI: Enable+ Count=1/2 Maskable- 64bit-
        Capabilities: [6c] Express Root Port (Slot+), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp

00:08.0 System peripheral: Intel Corporation 5000 Series Chipset DMA Engine (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, fast devsel, latency 0, IRQ 255
        Memory at d8500000 (64-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [6c] Express Root Complex Integrated Endpoint, MSI 00

00:10.0 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: fast devsel
        Kernel driver in use: i5000_edac
        Kernel modules: i5000_edac

00:10.1 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: fast devsel
        Kernel modules: i5000_edac

00:10.2 Host bridge: Intel Corporation 5000 Series Chipset FSB Registers (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: fast devsel
        Kernel modules: i5000_edac

00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: fast devsel

00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: fast devsel

00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: fast devsel

00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev b1)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: fast devsel

00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09) (prog-if 00 [UHCI])
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, medium devsel, latency 0, IRQ 233
        I/O ports at 1800 [size=32]
        Kernel driver in use: uhci_hcd
        Kernel modules: uhci-hcd

00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09) (prog-if 00 [UHCI])
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, medium devsel, latency 0, IRQ 50
        I/O ports at 1820 [size=32]
        Kernel driver in use: uhci_hcd
        Kernel modules: uhci-hcd

00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09) (prog-if 00 [UHCI])
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, medium devsel, latency 0, IRQ 177
        I/O ports at 1840 [size=32]
        Kernel driver in use: uhci_hcd
        Kernel modules: uhci-hcd

00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09) (prog-if 20 [EHCI])
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, medium devsel, latency 0, IRQ 233
        Memory at d8500400 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Kernel driver in use: ehci_hcd
        Kernel modules: ehci-hcd

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9) (prog-if 01 [Subtractive decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=08, subordinate=08, sec-latency=32
        I/O behind bridge: 00003000-00003fff
        Memory behind bridge: d8200000-d82fffff
        Prefetchable memory behind bridge: 00000000d0000000-00000000d7ffffff
        Capabilities: [50] Subsystem: Super Micro Computer Inc Device 9780

00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, medium devsel, latency 0
        Kernel modules: intel-rng, i8xx_tco

00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller (rev 09) (prog-if 8a [Master SecP PriP])
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, medium devsel, latency 0, IRQ 177
        I/O ports at <unassigned>
        I/O ports at <unassigned>
        I/O ports at <unassigned>
        I/O ports at <unassigned>
        I/O ports at 1860 [size=16]
        Kernel driver in use: PIIX_IDE
        Kernel modules: ata_piix

00:1f.2 IDE interface: Intel Corporation 631xESB/632xESB/3100 Chipset SATA IDE Controller (rev 09) (prog-if 8f [Master SecP SecO PriP PriO])
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 50
        I/O ports at 1890 [size=8]
        I/O ports at 1884 [size=4]
        I/O ports at 1888 [size=8]
        I/O ports at 1880 [size=4]
        I/O ports at 1870 [size=16]
        Memory at d8500800 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [70] Power Management version 2
        Kernel driver in use: ata_piix
        Kernel modules: ata_piix

00:1f.3 SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus Controller (rev 09)
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: medium devsel, IRQ 50
        I/O ports at 1100 [size=32]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c-i801

01:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=01, secondary=02, subordinate=04, sec-latency=0
        I/O behind bridge: 00002000-00002fff
        Memory behind bridge: d8000000-d80fffff
        Prefetchable memory behind bridge: 00000000d8300000-00000000d83fffff
        Capabilities: [44] Express Upstream Port, MSI 00
        Capabilities: [70] Power Management version 2
        Capabilities: [80] Subsystem: Super Micro Computer Inc Device 9780
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp

01:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=01, secondary=05, subordinate=05, sec-latency=64
        Capabilities: [44] Express PCI/PCI-X Bridge, MSI 00
        Capabilities: [6c] Power Management version 2
        Capabilities: [80] Subsystem: Super Micro Computer Inc Device 9780
        Capabilities: [d8] PCI-X bridge device
        Capabilities: [100] Advanced Error Reporting
        Kernel modules: shpchp

02:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
        Capabilities: [44] Express Downstream Port (Slot-), MSI 00
        Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Power Management version 2
        Capabilities: [80] Subsystem: Super Micro Computer Inc Device 9780
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp

02:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E3 (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
        I/O behind bridge: 00002000-00002fff
        Memory behind bridge: d8000000-d80fffff
        Prefetchable memory behind bridge: 00000000d8300000-00000000d83fffff
        Capabilities: [44] Express Downstream Port (Slot-), MSI 00
        Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Power Management version 2
        Capabilities: [80] Subsystem: Super Micro Computer Inc Device 9780
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp

04:00.0 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
        Subsystem: Super Micro Computer Inc Device 1096
        Flags: bus master, fast devsel, latency 0, IRQ 82
        Memory at d8020000 (32-bit, non-prefetchable) [size=128K]
        Memory at d8000000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 2000 [size=32]
        [virtual] Expansion ROM at d8300000 [disabled] [size=64K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-30-48-ff-ff-c4-9f-a2
        Kernel driver in use: e1000e
        Kernel modules: e1000e

04:00.1 Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
        Subsystem: Super Micro Computer Inc Device 1096
        Flags: bus master, fast devsel, latency 0, IRQ 98
        Memory at d8060000 (32-bit, non-prefetchable) [size=128K]
        Memory at d8040000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 2020 [size=32]
        [virtual] Expansion ROM at d8310000 [disabled] [size=64K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-30-48-ff-ff-c4-9f-a2
        Kernel driver in use: e1000e
        Kernel modules: e1000e

08:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) (prog-if 00 [VGA controller])
        Subsystem: Super Micro Computer Inc Device 9780
        Flags: bus master, stepping, fast Back2Back, medium devsel, latency 66, IRQ 11
        Memory at d0000000 (32-bit, prefetchable) [size=128M]
        I/O ports at 3000 [size=256]
        Memory at d8200000 (32-bit, non-prefetchable) [size=64K]
        [virtual] Expansion ROM at d8220000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 2

Code:

[root@quasar ~]# cat /proc/meminfo 
MemTotal:      8177460 kB
MemFree:        895732 kB
Buffers:        374660 kB
Cached:        4886712 kB
SwapCached:        112 kB
Active:        4626412 kB
Inactive:      2023732 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8177460 kB
LowFree:        895732 kB
SwapTotal:     2096472 kB
SwapFree:      2096300 kB
Dirty:            3920 kB
Writeback:           0 kB
AnonPages:     1388616 kB
Mapped:          31516 kB
Slab:           583464 kB
PageTables:      23188 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   6185200 kB
Committed_AS:  2954448 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    264032 kB
VmallocChunk: 34359474039 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

**Rubberman** · 05-23-2011

This is good to know. I haven't tried building a current-issue kernel for an older system, partially because I had read pretty much what I said in my previous post in some other kernel forums and articles. Goes to show - don't assume anything in this business until you have tried it yourself!

Anyway, thanks for the info.

**Blue|Fusion** · 05-23-2011

It worked! I got the go ahead to reboot it a little bit ago and no problems so far.

Code:

[root@quasar ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
[root@quasar ~]# uname -a
Linux quasar.(redacted).com 2.6.39 #6 SMP Sat May 21 12:03:44 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

It hurts your pride so much when it's just one little tiny overlooked option. But oh well, that's why we [Linux users] will inherit the world!

Thread Tools

Display

kernel panic help

Posting Permissions