Running out of Memory

[BENCHMARKMAN]

Let me first start out by giving you my server specs. I am running a server with 8gigs of ram, NO hard drive, and no swap space. The servers periodically run out of memory, OOM panic. Top shows < 20000k free memory but there is plenty of buffer and cache memory, >800mb. It was my understanding that the system would not run out of memory if there was cache and buffer space? Can somebody explain this to me? Thanks

[HROAdmin26]

Individual threads/processes do not have unlimited memory. The limits are also different for 32 vs 64 bit OS'es.

Also, what do you have running? The most common cause of OOM errors is a memory leak in an application.

[BENCHMARKMAN]

but I'm only getting OOM panics when system memory goes below 20mb. Isn't the kernel supposed to free up the cached and buffer memory so that it can bused for apps?

BTW its a 64 bit OS.

[HROAdmin26]

Individual threads/processes do not have unlimited memory.

No. If the thread has its max memory, no more can be addressed/allocated.

[BENCHMARKMAN]

Its not any specific thread but random threads that get killed. It only happnes when top shows less than 20mbs of free system memory.

[HROAdmin26]

You have memory problems. When memory gets low, Linux will "pick" processes to kill based on a "badness" score.

OOM Killer

The processes getting killed are not the problem, they are a symptom of the problem.

The functions, code excerpts and comments discussed below here are from mm/oom_kill.c unless otherwise noted.

It is the job of the linux 'oom killer' to sacrifice one or more processes in order to free up memory for the system when all else fails. It will also kill any process sharing the same mm_struct as the selected process, for obvious reasons. Any particular process leader may be immunized against the oom killer if the value of it's /proc/<pid>/oomadj is set to the constant OOM_DISABLE (currently defined as -17).

The function which does the actual scoring of a process in the effort to find the best candidate for elimination is called badness(), which results from the following call chain:

** Edit: Process getting killed is *not necessarily* the problem process.

[BENCHMARKMAN]

I guess my point is, why is linux not freeing the buffer and cache memory?

[BENCHMARKMAN]

No ideas or explenations on why linux is not freeing up the buffer and cache memory when system memory goes below 20mb? If I decrease the swappiness value, even though I am not using swap space, will this reduce the amount of memory uses for cache?

[HROAdmin26]

Based on not having any idea of what is causing the low memory condition (the actual problem), I don't see any reason to guess that "buffer/cache is not being released correctly."

[BENCHMARKMAN]

Then why would top show 2 gigs of cache and buffer memory combined when it crashes?

What about the swappiness setting I asked about in the previous post?