my oracle app team complaints blade server hangs always :SOS

[chipsettsr]

see the details

mysystem:~ # uname -a
Linux mysystem 2.6.27.19-5-default #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 GNU/Linux
==============================================
mysystem:~ # vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
108 0 0 58156268 663180 4006820 0 0 1 1 3 1 0 1 99 0 0
==============================================
mysystem:~ # mpstat
Linux 2.6.27.19-5-default (mysystem) 09/18/2011 _x86_64_

10:44:41 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
10:44:41 PM all 0.22 0.00 0.85 0.03 0.00 0.00 0.00 0.00 98.90
==============================================
mysystem:~ # top
top - 22:44:45 up 3 days, 21:27, 12 users, load average: 360.00, 359.64, 357.16
Tasks: 460 total, 49 running, 409 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.2%us, 0.9%sy, 0.0%ni, 98.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65834100k total, 7678580k used, 58155520k free, 663188k buffers
Swap: 67103496k total, 0k used, 67103496k free, 4006824k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3188 pipadm 20 0 4488m 2.2g 25m D 1097 3.4 1518:21 jstart
2757 pipadm 20 0 0 0 0 Z 199 0.0 218:18.01 icman <defunct>
12896 root 20 0 17200 1480 860 R 4 0.0 0:00.04 top
1 root 20 0 1064 412 348 S 0 0.0 0:05.56 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.06 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.34 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.30 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/1
6 root 15 -5 0 0 0 S 0 0.0 0:00.10 ksoftirqd/1
7 root RT -5 0 0 0 S 0 0.0 0:00.26 migration/2
8 root 15 -5 0 0 0 S 0 0.0 0:00.10 ksoftirqd/2
9 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/3
10 root 15 -5 0 0 0 S 0 0.0 0:00.06 ksoftirqd/3
11 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/4
12 root 15 -5 0 0 0 S 0 0.0 0:00.06 ksoftirqd/4
13 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/5
14 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/5
15 root RT -5 0 0 0 S 0 0.0 0:00.24 migration/6
16 root 15 -5 0 0 0 S 0 0.0 0:00.02 ksoftirqd/6
17 root RT -5 0 0 0 S 0 0.0 0:00.42 migration/7
18 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/7
19 root RT -5 0 0 0 S 0 0.0 0:00.30 migration/8
20 root 15 -5 0 0 0 S 0 0.0 0:00.32 ksoftirqd/8
21 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/9
22 root 15 -5 0 0 0 S 0 0.0 0:00.12 ksoftirqd/9
23 root RT -5 0 0 0 S 0 0.0 0:00.28 migration/10
24 root 15 -5 0 0 0 S 0 0.0 0:00.06 ksoftirqd/10
25 root RT -5 0 0 0 R 0 0.0 0:00.28 migration/11
26 root 15 -5 0 0 0 S 0 0.0 0:00.04 ksoftirqd/11
27 root RT -5 0 0 0 R 0 0.0 0:00.28 migration/12
28 root 15 -5 0 0 0 S 0 0.0 0:00.02 ksoftirqd/12
mysystem:~ # cat /proc/loadavg
360.00 359.67 357.22 109/841 12920
mysystem:~ #

[chipsettsr]

its a ibm blade HX5 any further info please let know , advice, comments please

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3188 pipadm 20 0 4488m 2.2g 25m D 1097 3.4 1518:21 jstart

see some of process cpu utilization is very high, sometimes 1100 like


so what I have to do to avoid this , any oracle db tips ?
or required any OS tuning ?

[tdsan]

its a ibm blade HX5 any further info please let know , advice, comments please

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3188 pipadm 20 0 4488m 2.2g 25m D 1097 3.4 1518:21 jstart

see some of process cpu utilization is very high, sometimes 1100 like


so what I have to do to avoid this , any oracle db tips ?
or required any OS tuning ?

====================================

Since the data is so jumbled, can you send a file that contains information relating to the type of machine, firmware, memory, type of oracle software you are running and a drawing, this will help to layout the environment, there could be something external we are not aware of (patch levels of the oracle database)

That way, we can sort out the details.

[tdsan]

I think I see what's happening here.

It looks like your backups maybe interfering with your system - IBM US Announcement Letter

I see a process running called icman (this is a backup process)
2757 pipadm 20 0 0 0 0 Z 199 0.0 218:18.01 icman <defunct>

One thing you might want to do is stop the process from running. Do you know at what time the oracle database hangs, that will help determine when the process kicks off.

I think there is some contention there.

I am not sure of your environment but from the memory usage and cpu usage, you are not using that much, exception in 1 instance.

Cpu(s): 0.2%us, 0.9%sy, 0.0%ni, 98.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st - This gives a glimpse of your overall system and you are running at 98.9% idle (CPU) and memory Mem: 65834100k total, 7678580k used, 58155520k free, 663188k buffers, you are not using that much memory.

But be sure to ask them what time the oracle database hangs.

Todd

[chipsettsr]

hi todd prior thanks to interfere my prob

this is IBM HS5 blade and SAN boot Environment. my storage is usp-vm

and I have to reveal that if the jstart process consumes most of my %cpu, that time in "top " utility can view process id and details
but process tree cannot touch further details of that specefic thread.

more over than that process cannot end even with kill -9 <pid>

such a condition i can only thing can do is a manual restart the OS


I am not a DB guy but a system admin , if u need further details please let know that or any command output details

I am expecting your / forum peoples co - operation . please help me

[chipsettsr]

any how the backup process not disturbing the whole db process.
after jstart consume my whole resources it is like that next priority PID will try to consume rest of resources.

[tdsan]

any how the backup process not disturbing the whole db process.
after jstart consume my whole resources it is like that next priority PID will try to consume rest of resources.

====================================

Ok,

One thing we will need, we will need information about your environment (details), storage, network, type of server it is running on, and version of oracle, patches, etc.

If you have a diagram, that would help us as well (I think I mentioned this in the email before) but I will reiterate the point.

Thanks