Swap usage issue MYSQL

[kingraun]

---

Hi Friends,
I am unable to find where my swap is going. A couple of days back i increased my RAM from 32G to 48G as my ndbmtd process has become RAM hungry.
I have 41G swap but still after upgrading it seems it needs more mem.So i was trying to track where my mem is being used.i found that ndbmtd is taking 40G of my RAM but 33G of my swap is also is being used which i am unable to figure out why?
following are some of the outputs to some standard commands.

free -g
total used free shared buffers cached
Mem: 47 47 0 0 0 6
-/+ buffers/cache: 39 7
Swap: 41 33 8

# cat /proc/meminfo
MemTotal: 49434896 kB
MemFree: 135508 kB
Buffers: 67532 kB
Cached: 7341860 kB
SwapCached: 35522972 kB
Active: 26245272 kB
Inactive: 22549692 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 49434896 kB
LowFree: 135508 kB
SwapTotal: 44040176 kB
SwapFree: 8487964 kB
Dirty: 5228 kB
Writeback: 256 kB
AnonPages: 41386040 kB
Mapped: 13160 kB
Slab: 356420 kB
PageTables: 86432 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 68757624 kB
Committed_AS: 41552796 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 285004 kB
VmallocChunk: 34359453043 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

top - 03:54:09 up 10 days, 13:05, 3 users, load average: 0.18, 0.23, 0.3
Tasks: 263 total, 1 running, 262 sleeping, 0 stopped, 0 zombie
top - 03:54:27 up 10 days, 13:06, 3 users, load average: 0.28, 0.25, 0.3
Tasks: 263 total, 1 running, 262 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.5%us, 0.8%sy, 0.0%ni, 97.2%id, 0.0%wa, 0.1%hi, 0.4%si, 0.
Mem: 49434896k total, 49297420k used, 137476k free, 60780k buffers
Swap: 44040176k total, 35552196k used, 8487980k free, 7344928k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1388 root 15 0 39.5g 39g 6816 S 36.2 83.7 3751:50 ndbmtd
22828 root 15 0 12872 1224 808 R 0.3 0.0 0:00.13 top
1 root 15 0 10348 576 548 S 0.0 0.0 0:04.30 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.64 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.09 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:22.54 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:00.08 migration/2

the output of following command
for i in /proc/*/smaps ; do awk '/Swap/ {sum += $2}; END{if (sum>0) print FILENAME " Total: " sum}' $i; done

/proc/1387/smaps Total: 388
/proc/14893/smaps Total: 560
/proc/14977/smaps Total: 476
/proc/1516/smaps Total: 824
/proc/1/smaps Total: 76
/proc/28009/smaps Total: 712
/proc/28011/smaps Total: 424
/proc/29562/smaps Total: 636
/proc/4098/smaps Total: 220
/proc/4100/smaps Total: 200
/proc/4122/smaps Total: 72
/proc/4125/smaps Total: 84
/proc/4261/smaps Total: 56
/proc/4288/smaps Total: 116
/proc/4365/smaps Total: 140
/proc/4389/smaps Total: 472
/proc/4413/smaps Total: 952
/proc/4425/smaps Total: 312
/proc/4440/smaps Total: 84
/proc/4446/smaps Total: 96
/proc/4549/smaps Total: 708
/proc/4559/smaps Total: 96
/proc/4568/smaps Total: 2268
/proc/4569/smaps Total: 180
/proc/4578/smaps Total: 108
/proc/4624/smaps Total: 104
/proc/4659/smaps Total: 260
/proc/4721/smaps Total: 452
/proc/4735/smaps Total: 124
/proc/5190/smaps Total: 1544
/proc/5195/smaps Total: 1764
/proc/5366/smaps Total: 64
/proc/5375/smaps Total: 508
/proc/5451/smaps Total: 944
/proc/5468/smaps Total: 104
/proc/5625/smaps Total: 68
/proc/5627/smaps Total: 68
/proc/5628/smaps Total: 68
/proc/5629/smaps Total: 64
/proc/5631/smaps Total: 68
/proc/5632/smaps Total: 64
/proc/5633/smaps Total: 616
/proc/5731/smaps Total: 712
/proc/5733/smaps Total: 656
/proc/5734/smaps Total: 1760
/proc/5760/smaps Total: 9216
/proc/8293/smaps Total: 192
/proc/8486/smaps Total: 140

this is in KB.

I am kind of stuck.
Please help people.Thank you in advance.

[Rubberman]

You can use pmap to see what processes are consuming the memory. As to why, that is another issue, and it may be that you will have to visit the source code to determine where memory is leaking. IBM has a great leak testing tool called Purify. It is pricey, but they have a free trial period which may help you figure out where the memory is going. It used object-code insertion to instrument the code - you only need to relink the executables in question. It will also instrument system shared libraries (it doesn't modify the original libraries, but creates instrumented copies that are linked to the executable), which can sometimes be "interesting"...