Prevent my RHEL 5.5 from going out-of-memory

[Mauwges]

Hi Community,

I'm searching for a solution for that issue quite a time and didn't find any fulfilling one.

My problem:
I've a couple of workstations running REHL 5.5 where several users work remotely using VNC or NX with local sessions.
Sometime some tools get crazy and start to consume all memory available resulting in the machine being not responsive or reachable any more.
Once the machine starts swapping - its over.

All solutions I've found so far limit the mem-usage per user.
And the oom killer seems not to be the right solution, because it kills random processes when running oom.

I'm searching for a solution which monitors the global consumption of physical memory and kills the process which is about to consume the last - lets say - 2 %.
Or something which preserves memory for the kernel so the machine stays reachable. Maybe its enough to prevent swapping?

Would be nice if I get some helpful advices.

Thanks
Markus

[msohail]

Hi Community,

I'm searching for a solution for that issue quite a time and didn't find any fulfilling one.

My problem:
I've a couple of workstations running REHL 5.5 where several users work remotely using VNC or NX with local sessions.
Sometime some tools get crazy and start to consume all memory available resulting in the machine being not responsive or reachable any more.
Once the machine starts swapping - its over.

All solutions I've found so far limit the mem-usage per user.
And the oom killer seems not to be the right solution, because it kills random processes when running oom.

I'm searching for a solution which monitors the global consumption of physical memory and kills the process which is about to consume the last - lets say - 2 %.
Or something which preserves memory for the kernel so the machine stays reachable. Maybe its enough to prevent swapping?

Would be nice if I get some helpful advices.

Thanks
Markus

Create a cron job to flush out garbage memory every 5 minutes (or minutes that u see fit)

*/5 * * * * /bin/echo 3 > /proc/sys/vm/drop_caches

Hopefully this should resolve your memory issue.

Jazak Allah,
Sohail

[Mauwges]

Create a cron job to flush out garbage memory every 5 minutes (or minutes that u see fit)

*/5 * * * * /bin/echo 3 > /proc/sys/vm/drop_caches

Hopefully this should resolve your memory issue.

Jazak Allah,
Sohail

I thought about such a solution - but having a cron job running every minute. Isn't that waste of computation time?

But my thought was - there must be a more sophisticated way to preserve memory for the Kernel. The cron-job-solution seems to my like a dirty workaround.

[msohail]

I thought about such a solution - but having a cron job running every minute. Isn't that waste of computation time?

But my thought was - there must be a more sophisticated way to preserve memory for the Kernel. The cron-job-solution seems to my like a dirty workaround.

You can always go for recompiling the kernel and divide the physical memory into user-based memory and kernel-based memory, part of kernel programming.

Jazak Allah,
Sohail

[Mauwges]

You can always go for recompiling the kernel and divide the physical memory into user-based memory and kernel-based memory, part of kernel programming.

Jazak Allah,
Sohail

Hehe - nice one. Unfortunately I cannot recompile the Kernel on that workstations - security policies...
I'm wondering why this is not done by default.

Wouldn't it be sufficient to forbid swapping? For my impression it's the swapping that makes the machine unresponsive. What would happen if I disable swapping and a process eats up all RAM?

best regards
Markus

[msohail]

Hehe - nice one. Unfortunately I cannot recompile the Kernel on that workstations - security policies...
I'm wondering why this is not done by default.

Wouldn't it be sufficient to forbid swapping? For my impression it's the swapping that makes the machine unresponsive. What would happen if I disable swapping and a process eats up all RAM?

best regards
Markus

Your server will become slow unless you free up some memory.

[msohail]

I thought about such a solution - but having a cron job running every minute. Isn't that waste of computation time?

But my thought was - there must be a more sophisticated way to preserve memory for the Kernel. The cron-job-solution seems to my like a dirty workaround.

You need not run the cron job every minute, you can set it for once in every 1 hour coz garbage memory will not pile up that quickly.

Jazak Allah'
Sohail

[Mauwges]

You need not run the cron job every minute, you can set it for once in every 1 hour coz garbage memory will not pile up that quickly.

Jazak Allah'
Sohail

Unfortunately I need an interval of one minute.
My oom-problem is not caused by garbage memory summing up but by tools which might run crazy and eating up all memory. I shot my machine a couple of times this way in less than 10 min.

This is really an issue because often a restart is required which affect separate users.

Your server will become slow unless you free up some memory.

Why would the machines becomes slow when it doesn't swap - so all running and KERNEL relevant processes reside in memory?

[msohail]

Unfortunately I need an interval of one minute.
My oom-problem is not caused by garbage memory summing up but by tools which might run crazy and eating up all memory. I shot my machine a couple of times this way in less than 10 min.

This is really an issue because often a restart is required which affect separate users.



Why would the machines becomes slow when it doesn't swap - so all running and KERNEL relevant processes reside in memory?

Did you try increasing the swappiness kernel parameter to 90 or more?

Jazak Allah
Sohail

[Mauwges]

Did you try increasing the swappiness kernel parameter to 90 or more?

Jazak Allah
Sohail

But this would increase swapping. From my experience - when the machine once starts swapping it's not reachable anymore - because its full-loaded with swapping.
Am I wrong?