Find the answer to your Linux question:
Results 1 to 7 of 7
Hello, We have a NFS with a huge amount of file and noticed that the servers who access many files quickly have almost no free (including buffered and cached) memory ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jun 2012
    Location
    Belgium
    Posts
    1

    NFS inode cache eats up memory, design flaw?


    Hello,

    We have a NFS with a huge amount of file and noticed that the servers who access many files quickly have almost no free (including buffered and cached) memory any more. We soon found out that it was mostly taken by the nfs_inode_cache as slabs.

    Cleaning the cache /proc/sys/vm/drop_caches frees up the memory, but setting the swappiness to 1 doesn't help: the cache still grew to the same size and applications where swapped to free up memory. For us swapping is bad, we have strict SLA's for the response times of our applications and very lax SLA's on the NFS.

    We can easily reproduce the issue by doing a full scan of the NFS drive, the nfs_inode_cache is soon much more than 1GB but with a cache and buffer smaller then 100MB (free command).
    The issue is so bad that the find we execute even grinds to a halt. We aren't sure it is due to the cache, but we are investigating it.

    So far the stuff I'm sure about, I investigated and came to the following conclusion (correct me if I'm wrong).

    I found the following image to describe the file system design of linux: tldp.org/LDP/tlk/fs/vfs.gif (sorry, I can't post urls yet). The nfs_inode_cache (and ext4_inode_cache) aren't indicated, which seems to be exactly the problem the kernel has.

    The kernel seems to be only aware of the vfs inodes, it does not (for obvious reasons) know about the nfs or ext4 inode cache. As far as I could understand the kernel code, the nfs inodes are put/removed from cache based on calls that come from the vfs. Because the vfs only sees it own cache, it might think it uses only x MB while in effect, behind it there is another x GB that it can't see. This could explain why the kernel would not clean any more cache due to memory pressure, but that when you clear it manually it frees up the nfs inode cache. This can be explained because the nfs cache is linked to vfs cache.

    It seems to me that the problem is there for all file systems, but is especially visible for NFS. Since NFS is network based, it might require quite a lot more information in the cache per inode (this is pure guessing). This would case a much bigger "hidden" cache.

    So my questions are:
    * Is my conclusion (somewhat) correct?
    * What can I do about it?

    Kind regards,
    Bryan Brouckaert.

  2. #2
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,746
    Have you posted your question to the linux kernel group (www.kernel.org)?
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  3. #3
    Just Joined!
    Join Date
    Jun 2012
    Posts
    9
    The original poster is a colleague of mine so I can answer for him. No we haven’t reported this issue to the kernel group but we did report it to RedHat(which is quite logic for a RedHat server with support) and the answered that this is a known bug ant that they will release a patch for it in the last week of June or the first week of July.

  4. $spacer_open
    $spacer_close
  5. #4
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,746
    That was probably a good approach, especially since you had RH support. After all, that's what you are paying for! Anyway, glad to know that it is a known problem with a fix around the corner. It's not a "fix" or even a decent workaround, but you COULD restart your NFS services during quiet times (3am or something) to release those caches, at least until the fix is in place and verified.
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  6. #5
    Just Joined!
    Join Date
    Jun 2012
    Posts
    9
    The temporary workaround ware are using is to 'sync' and then 'echo 2 > /proc/sys/vm/drop_caches', which cleans all unneeded caches. By doing this every day, we seem to be able to keep everything up and running until we can install the patch.
    And thanks for providing a possible workaround.

  7. #6
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    11,746
    Quote Originally Posted by Nightshadow View Post
    The temporary workaround ware are using is to 'sync' and then 'echo 2 > /proc/sys/vm/drop_caches', which cleans all unneeded caches. By doing this every day, we seem to be able to keep everything up and running until we can install the patch.
    And thanks for providing a possible workaround.
    I think your solution is more "elegant" - didn't think of it myself. It may be of use in our domain, so thanks for the idea!
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

  8. #7
    Just Joined!
    Join Date
    Jun 2012
    Posts
    9
    I'm glad I can return the favor of helping

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •