Find the answer to your Linux question:
Results 1 to 5 of 5
Hi, I'm not sure if this is the right place to post this, but since the oom-killer is part of the linux kernel and thus distro-independent I figured it would ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Apr 2007
    Posts
    6

    OOM-killer behaviour


    Hi,

    I'm not sure if this is the right place to post this, but since the oom-killer is part of the linux kernel and thus distro-independent I figured it would go here.
    If not, feel free to move the topic elsewhere.


    We are facing some problems with the OOM-killer on some of our servers.
    There are some applications (in most cases, but not limited to, the apache webserver) which tend to cause an out of memory.
    Most of the time the servers run perfectly, but every once in a while an oom condition pops up and the oom-killer starts killing all sorts of processes.
    I'm saying all sorts of processes, since from my point of view it is using a rather random approach. When an oom condition was caused by the apache web server, the oom-killer started killing the ldap, ssh, ... service instead of the httpd processes causing the problem.
    So even if the server survives the oom condition by finally killing httpd, it's best to just reboot the system since most of the services have been killed.

    Is there a way to change this behaviour?
    I'd rather have the oom-killer kill of the process with the most memory allocated, or the process trying to use the extra memory (and thus placing the server in this situation).

    I've read about completely disabling the oom-killer, but this would result in a kernel panic, which I would like to prevent.


    Kind regards,

    Bsquirle

  2. #2
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,392
    The thing is:
    If you are already in a situation, where ram is extremely low,
    then you cannot implement fancy logic or extended configurability.

    Afaik, the oom killer looks for big processes, that have been idle for a while.
    The activity also has a higher prio than the size, thatīs why even a ssh daemon
    can be killed.

    From outside, the behaviour is pretty much random, as you experience it.

    So it is better to avoid oom.
    Can you move services, split data, add more ram, etc..?
    You must always face the curtain with a bow.

  3. #3
    Just Joined!
    Join Date
    Apr 2007
    Posts
    6
    So if I understand this correctly.
    Big processes that are always active will never get killed.

    I know it is better to avoid oom, but currently I don't have that option right now.
    On top of that, I am not in control of what people run on the servers (two months ago it was running fine, then somebody somewhere changed something in an application on one of them and suddenly it exploded).
    I'm just looking for a solution that will work for all server I have to manage, no matter what people throw at them.

    I was hoping there would be a possibility to change the actions taken by oom-killer, or perhaps select another logic used to kill processes. Guess this was a bit wishful thinking from my side.

    If no solution comes to the surface I might look into adding memory (which is on the roadmap anyway), split services and look after load balancing/high availability for some of them.

  4. #4
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,392
    You cannot assure "never get killed" in that context of your apps overusing memory.
    That would assume infinite memory, and then you dont need oom in the first place.

    Small, always active processes have the least chance to get killed. Not big, always active ones.

    Hopefully, you assigned different users to these apps and people.
    Then you could just use ulimit to set memory limits.
    Also communicate that to the people involved.
    That will most propably cause a lot of "replies", asking/demanding more memory for *their* app, because it is the most important

    Utilize that as a leverage to demand more servers/more ram/better apps (that use less memory).
    Last edited by Irithori; 08-19-2010 at 10:46 AM.
    You must always face the curtain with a bow.

  5. #5
    Just Joined!
    Join Date
    Apr 2007
    Posts
    6
    Quote Originally Posted by Irithori View Post
    Also communicate that to the people involved.
    That will most propably cause a lot of "replies", asking/demanding more memory for *their* app, because it is the most important
    Haha, true, we've already had similar discussion in the past


    Thanks for your help, it gave me some more insight in the oom handling of the kernel. I think I'll be able to take on the problems from here until I can upgrade the servers or install new ones.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •