Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 12
I work on highly threaded processing software. Test machine is a dual xeon ivy bridge (total 20 cores/40 threads) with 128GB ram, Running a task where cpu threads are loading ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jan 2014
    Posts
    5

    clear_page_c_e eating lots of cpu


    I work on highly threaded processing software. Test machine is a dual xeon ivy bridge (total 20 cores/40 threads) with 128GB ram, Running a task where cpu threads are loading images, generating image patches and running statistics on them.

    Running perf top it seems that clear_page_c_e is eating up to 25% total of all the cpus. The only call taking up more total cpu is the function which pulls image pixel windows out of the imagery. The process uses ~10GB of ram when running full bore with a test dataset.

    I was wondering what this clear_page_c_e is doing and what to look for that might be causing this to kick off.

  2. #2
    Linux Engineer
    Join Date
    Dec 2013
    Posts
    1,182
    clear_page_c_e is a kernel function. the "_c_e" refers to the fact that it's a more efficient version of it used where certain assembler commands are available. When a memory page is reserved clear_page() is called (or in your case clear_page_c_e) and it zeros the memory.

    My guess, and it is just a guess, is that your program is allocating a lot of memory for the images. One strategy for reducing memory allocation is to pre-allocate pools of memory that the application itself then manages and reuses.

  3. #3
    Linux Engineer
    Join Date
    Dec 2013
    Posts
    1,182
    Another thought - if the program is creating a lot of threads or processes it might explain it.

  4. #4
    Linux Newbie
    Join Date
    Nov 2009
    Posts
    222
    How does your program allocate heap memory?

    If it uses "calloc", try changing it to "malloc". I doubt it will make a vast difference but at least you will avoid having the memory cleared to 0X00.

    Have you looked at your program with a view to introduce parallel processing?

  5. #5
    Just Joined!
    Join Date
    Jan 2014
    Posts
    5
    Quote Originally Posted by voidpointer69 View Post
    How does your program allocate heap memory?

    If it uses "calloc", try changing it to "malloc". I doubt it will make a vast difference but at least you will avoid having the memory cleared to 0X00.

    Have you looked at your program with a view to introduce parallel processing?
    The code is using c++. Unfortunately dcraw is being used to read in this imagery, I suspect as always the 3rd party stuff might be the problem (an answer is to preprocess all this imagery to png or tiff, run again and see if this clear_page_c_e is causing trouble still).

    Parallel processing is most effective sitting on top of highly multi threaded. We haven't made that move yet because our customers aren't that sophisticated and I absolutely have had poor experiences with customers deploying clusters. Additionally a sufficient network fabric for parallel processing costs many times more than the hardware. I built those dev xeon systems for under 5000usd each and I consider those horribly expensive (our previous dual nehalem xeons were less than half that proce).

    Also...be scared...the software is running under windows at customer sites as well. Of course they have to reboot the windows boxes after every run otherwise the systems go unstable for the next run, but what do you expect with that OS?

  6. #6
    Linux Engineer
    Join Date
    Dec 2013
    Posts
    1,182
    Clear_page isn't invoked directly by the C++ (or C) memory manager - it is invoked every time a new memory page is required so a request for heap space from C++ won't necessarily require it. If a process is being forked to invoke dcraw to process each image there will be a lot of memory being copied by the kernel (the process is copied each time it's forked) and that could explain it.

  7. #7
    Just Joined!
    Join Date
    Jan 2014
    Posts
    5
    Quote Originally Posted by gregm View Post
    Clear_page isn't invoked directly by the C++ (or C) memory manager - it is invoked every time a new memory page is required so a request for heap space from C++ won't necessarily require it. If a process is being forked to invoke dcraw to process each image there will be a lot of memory being copied by the kernel (the process is copied each time it's forked) and that could explain it.
    my apologies, call directly into libraw (not dcraw). I notice it does use calloc for its page allocations. If I can find space (the NEF files are 53GB) I'll convert to a format the code reads directly and see what happens there.

  8. #8
    Linux Engineer
    Join Date
    Dec 2013
    Posts
    1,182
    Quote Originally Posted by bnolsen View Post
    my apologies, call directly into libraw (not dcraw). I notice it does use calloc for its page allocations. If I can find space (the NEF files are 53GB) I'll convert to a format the code reads directly and see what happens there.
    Calloc doesn't do page allocations; it does heap allocation at a user level. Page allocation is a kernel operation. If, when a request is made for more memory in user space, there is none on the free list then a page allocation will occur. Whether you use calloc or malloc won't affect this. Pages are always cleared as a security precaution before being made available to user space.

    So, if page allocation is causing problems then you have to look at your usage patterns and see where the problem occurs and if you can change the pattern.

  9. #9
    Just Joined!
    Join Date
    Jan 2014
    Posts
    5
    Quote Originally Posted by gregm View Post
    Calloc doesn't do page allocations; it does heap allocation at a user level. Page allocation is a kernel operation. If, when a request is made for more memory in user space, there is none on the free list then a page allocation will occur. Whether you use calloc or malloc won't affect this. Pages are always cleared as a security precaution before being made available to user space.

    So, if page allocation is causing problems then you have to look at your usage patterns and see where the problem occurs and if you can change the pattern.
    how about shutting off the page clearing? any option for this? a customer running this software is going to have it in a server room hooked up via 10G to a dedicated storage. Essentially the computers running this will be appliances, not youtube machines. My partner who is working on real time video tracking is also running into clear_page_c_e taking up noticeable cycles as well. Seems windows doesn't do any of this atall.

  10. #10
    Linux Engineer
    Join Date
    Dec 2013
    Posts
    1,182
    I don't know for sure but I believe that it's a driver that is requesting zero'd memory pages. It could be kernel optimizations could fix the problem. Without sleuth work I'm just making guesses.

    This article is about optimizing for heavy memory usage applications - the kernel optimizations might help:
    Optimizing your Linux server for memory-based NoSQL databases (Part I) - BugSense Blog

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •