I have voice IVR on RHEL 4.5 in one of the remote production environment. The system is 32 bit and caters normal load i.e 30% of the total capacity.

The system runs fine for 3 days. On the 3rd day when the load is at its peak it gives the above error in the var log:

Jul 26 20:01:02 localhost kernel: mercd_write: unable to allocate memory 16128
Jul 26 20:01:02 localhost kernel: mercd_write: Unmatching Message Class 16128 and 52 35
Jul 26 20:01:02 localhost kernel: mercd_write: Current Message Class 0xfc0 Id 0x1
Jul 26 20:01:02 localhost kernel: mercd_write: Unmatching Message Class 16128 and 52 35

mercd_write is the telephony card driver.

I took all kinds of logs at this particular time:-

Code:
top - 20:01:18 up 5 days, 10:10, 9 users, load average: 0.13, 0.03, 0.01
Tasks: 128 total, 1 running, 127 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1% us, 0.2% sy, 0.0% ni, 99.1% id, 0.6% wa, 0.0% hi, 0.0% si
Mem: 4151264k total, 4130204k used, 21060k free, 138644k buffers
Swap: 6144820k total, 4k used, 6144816k free, 3756412k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
25537 root 16 0 197m 27m 4548 S 1 0.7 10:58.00 SdpMedia 
25413 root 16 0 239m 46m 6200 S 0 1.2 3:45.55 MGCv1.1 
25773 root 15 0 4536 1728 1140 S 0 0.0 1:44.56 SDP_IVR 
23695 root 16 0 16048 14m 14m S 0 0.4 0:08.20 rsi_lnk 
3633 root 15 0 0 0 0 S 0 0.0 0:28.33 smbiod 
25762 root 16 0 4416 1524 1272 S 0 0.0 0:30.97 DB_MR 
1 root 16 0 2880 544 468 S 0 0.0 0:02.33 init 


Date&Time: 2010-07-26-20:01
total used free shared buffers cached
Mem: 4053 4023 30 0 132 3653
-/+ buffers/cache: 237 3816
Swap: 6000 0 6000

uname -a:
Linux localhost.localdomain 2.6.9-55.ELsmp #1 SMP Fri Apr 20 17:03:35 EDT 2007 i686 i686 i386 GNU/Linux.

It's a 32 bit system. 
vmstat:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 208 15956 20116 3886644 0 0 4 28 31 7 1 2 97 1

ls -lh /proc/kcore

-r-------- 1 root root 897M Jul 31 23:06 /proc/kcore
My appliaction are SdpMedia, MGCv1.1, SDP_IVR and DB_MR. All the stats seems to be ok, but the card fails to allocate memory for playing wav files. I suspect kernel memory leak at device driver or O.S issue. Please help.


Raghuvendra Kumar