Apache crashes time-by-time and the system is not reachable by SSH

[dr_faust]

Hello,

I running an apache web server on Fedora 15. Basically it runs fine but all couple of days my server is down and even not reachable any more by SSH. I get something like

ssh: connect to host 192.168.1.13 port 22: No route to host

In the error logs of the httpd I sometimes find something like:

[Fri Mar 02 07:38:09 2012] [error] (12)Cannot allocate memory: fork: Unable to fork new process

as last line, but not always. Anybody can explain that behavior, especially why the server is no longer reachable by SSH?

Thanks a lot!

[Irithori]

"cannot allocate memory" sounds like the machine runs out of memory.
The kernel then begins to kill processes to free some resources.

Do you have monitoring/graphing in place to verify this theory?

[dr_faust]

Thanks for the answer. I do not use any monitoring/graphing for the moment. Is there easy way to set set up?

[Irithori]

Imho, the easiest options are probably sar followed by munin.
But there is a wide variety of tools of any size and featureset. xymon, nagios, ganglia, etc

[dr_faust]

Will try that out. I observed that there was little free disk space too while I observed the problem. Could it be due to a lack of SWAP memory too or is that not relevant?

[Irithori]

The original error doesnt indicate a disk full problem.
But that doesnt mean, that this isnt an addtional issue.

If there is not enough ram, the kernel will use swap space.
Swap is on disk, and therefore *very slow* compared to ram.

The daemons/processes need to be scaled/limited to the machines resources.

In order to do that, you need data.
How many and how big processes are running when?
How much percentage of memory is used when?

Hence: Monitoring

Naturally, I have no idea about your machine.
Just two (wild guess) examples, what might happen on this box:
- the apache childs may be quite heavy and there are simply to many concurrent request
- or the apache itself is fine, but a cronjob takes too much ram?

[dr_faust]

You are quite close with your guesses
Apache is starting child processes to convert images between different formats.

Maybe these processes consume to much RAM. I was wondering why the machine was no longer available via SSH but it sounds reasonable that the demon is shut down. Anyway I will start monitoring these days to find out the exact cause of this issue...

[dr_faust]

One more thing: Could the behaviour also be due to DOS attacks or is that highly unlikely?