"Unknown error 513 in pthread_create after several hours of running a Gstrea

[pjain01]

I am using Gstreamer for creating one of my video streaming application. My application creates multimedia session and then stream around 200 video sessions (a very processor intensive job).In doing so, the no. of threads created is around 1500 and I have given around 1536 KB as my stack size for each of these threads. I have one Quadcore(intel Xeon) system of around 8 GB of RAM(64 bit) with RHEL 5.5(kernel - 2.6.18-92.el5).
After few hours of testing I see that my overall memory(RAM) is eating up. I did checked the memory and I saw that my cache size is growing. So far so good, as I can understand that system does cache instead of leaving some RAM free.
My problem comes, when I see that after almost 20-25 hours of testing, RAM is continuously being eaten up and maximum part of that is going into cache. If I leave the system for few more hours I see whole of my memory is consumed up and now my swap started getting used. If I check the cache now, I clearly see most of RAM is now in form of cache. I still didn’t stop my application and allowed it run for few more hours and after sometime I see that process give segmentation fault while creating the pthread (create_pthread::unknown error 513).I also tried to observe the “core” file of application created in this way , but it doesn’t’ show anything.
One thing I must mention that, in “top” I see almost constant values of resident memory, virtual memory and shared memory. There is no increase of memory usage here. So here I can assume that my application is not leaking the memory. I also observed that if I kill my application any time during this, the memory doesn’t go into free pool.

Anyone, here can give any pointer for this kind of behavior. Is it can be because of some kernel BUG which is not freeing the memory to application usage.
No need to mentioned that any help or clue will be highly appreciated and regarded

[Rubberman]

I would look at the application and plugins first, though there have been a number of kernel updates since yours. The current latest is 2.6.18-194.26.1, so you might want to first update the kernel.