Cent os hangs after many hours.

[ankitasingh]

Hi,
I am running my server in centos.Threads are created when a client connects.one thread per client.The problem is after sometime centos hangs,and clients are not able to connect to it.It wont respond to anything for sometime may be hours also.after that it will be fine.if i try to login using putty then i wont get login prompt also it will stay blank with the log ---connecting...by using vnc it will show screen with some memory info and something is going on in there.Why this problem is occuring?how to fix this?

[Irithori]

It sounds like the machine runs out of memory and is swapping.
Do you have graphing and monitoring for that box? If yes: look at the memory consumption graphs. If not: Install monitoring and graphing.
If memory is so low, that the kernel needs to kill processes, then there will be messages in dmesg.
With vmstat and/or iostat you can see, if there is currently swapping going on.

[ankitasingh]

Thanks u so much for helping me in issues.That is true.my process gets killed also.i looked at the memory utilization using "top" command.and it was taking much memory as the number of thread increases.i need to connect 3000-6000 clients.Currently i am using 1 G.B. memory.So how much memory should i use according to u?

[Irithori]

That is a question for you, not for me.

Because you know or have access to information about:
Which technology you use (java, php, etc), which daemons (tomcat, apache), which databases are involved and how they are configured, how many concurrent connections there are, how long one connection lasts, what the average memory consumption per connection is, if and how these threads use shared memory, what your application does, etc,etc

In short: You need to analyse the whole deployment and/or read in the respective documentation to have some basis for a memory calculation.

[voidpointer69]

ankitasingh


Echo irithori on that one. 1GB is clearly not enough for what you are trying to do. Bang some more in there with a matching amount of swap.

You say you are using the thread-per-client model. Have you tried using a thread pool. At least that way, you wont be forever creating and destroying new processes.

Cheers - VP

[abarclay]

Can you post the line which you are using to create the thread?

I have successfully created thousands of threads in a 1 GB machine, but the trick is to create them with a much smaller stack size.

From the pthread_create man page:

On Linux/x86-32, the default stack size for a new thread is 2
megabytes. Under the NPTL threading implementation, if the
RLIMIT_STACK soft resource limit at the time the program started has
any value other than "unlimited", then it determines the default stack
size of new threads. Using pthread_attr_setstacksize(3), the stack
size attribute can be explicitly set in the attr argument used to cre-
ate a thread, in order to obtain a stack size other than the default.

Try setting your stack size to 10,000 bytes and see if you have better luck.

Andy