[SOLVED] Where does memory allocated with malloc actually come from?

[hazel]

I've spent the last few days sorting out a memory corruption problem in a program and it made me wonder how malloc actually works. What kind of memory does malloc allocate? Is it physical or virtual memory?

If it's virtual, it ought to disappear when your program exits, whether you've explicitly freed it or not, but from what I've read, that isn't the case. On the other hand, if it's physical, where is it? Is it a reserved area of core memory? Wikipedia says it comes from something called the "heap", which is a kind of common resource, but it doesn't say where this heap actually is.

It's funny how Linux gets you thinking about things you've previously taken for granted.

[Rubberman]

There is physical memory (RAM) and virtual memory. In Linux/Unix, all application memory is virtual, and unless it is swapped out to disc it points to a location in physical memory. Each page can be separately addressed, so while your application heap (memory allocated with malloc()) appears to your application as contiguous, it may actually be spread out all over your physical RAM. When you call malloc(), it first looks to its internal list of free memory and allocates your request from that. If it doesn't have enough available, then it will use a system call sbrk() to ask to operating system to allocate more virtual memory to it and malloc() will then add that space to its free list to allocate your requested chunk of memory. Each call to sbrk() will return memory to your application in some multiple of page size granules. FYI, a page is generally about 4KB by default.

Once you start to write to the memory you got with malloc(), the system will then either associate the virtual memory with available pages in RAM, or if they aren't available then the OS will swap enough physical pages to disc and give them back to your application. In fact, if you have already written to that memory, and go back to read/write it again, the OS will determine if the requested pages are in RAM and swap them back from disc if necessary, to possibly a different physical RAM location than they were previously. Clear as mud now, right?

[gerard4143]

If your having problems understanding virtual memory you should check out the Intel or AMD manuals. They have a very good explanation of what's going on at that level...

[hazel]

When you call malloc(), it first looks to its internal list of free memory and allocates your request from that. If it doesn't have enough available, then it will use a system call sbrk() to ask to operating system to allocate more virtual memory to it and malloc() will then add that space to its free list to allocate your requested chunk of memory. Each call to sbrk() will return memory to your application in some multiple of page size granules.

So I gather from that that it's virtual memory and that each process has its own heap, which can be extended if necessary by further calls to sbrk(). But then it should all vanish when the process exits so how come there are "memory leaks" when memory is allocated but not freed? According to the valgrind manual, exit() frees all allocated memory that it has pointers to but can't free memory whose pointer has disappeared off the stack. So what happens to this non-freed virtual memory?

[Rubberman]

All memory allocated for a process will be returned to the OS when the process terminates, though some parts may not be if the process is a zombie in that its parent hasn't handled the SIGCHLD signal its demise generated. However, the system can cache that memory for awhile, specifically the code (text) pages that were allocated for the executable and any shared libraries it loaded. However, stack, heap, and global memory should be released.

So, why do you think processes are "leaking" memory after they terminate?

[hazel]

All memory allocated for a process will be returned to the OS when the process terminates. So, why do you think processes are "leaking" memory after they terminate?

Well, I've been using valgrind and it reports loads of memory leaks. For example:

==30041== LEAK SUMMARY:
==30041== definitely lost: 1,150 bytes in 22 blocks
==30041== indirectly lost: 0 bytes in 0 blocks
==30041== possibly lost: 90,296 bytes in 747 blocks
==30041== still reachable: 155,898 bytes in 1,767 blocks

What you say is more or less what I thought before, after reading the kernel manual (which is very good on explaining what virtual memory actually is and how it's organised). But now I'm a wee bit confused.

[Rubberman]

Well, I've been using valgrind and it reports loads of memory leaks. For example:

==30041== LEAK SUMMARY:
==30041== definitely lost: 1,150 bytes in 22 blocks
==30041== indirectly lost: 0 bytes in 0 blocks
==30041== possibly lost: 90,296 bytes in 747 blocks
==30041== still reachable: 155,898 bytes in 1,767 blocks

What you say is more or less what I thought before, after reading the kernel manual (which is very good on explaining what virtual memory actually is and how it's organised). But now I'm a wee bit confused.

Ah! These are reports of the process when it terminates. It has no bearing on the system after it has been finalized by the OS. So, if this were a long running process then you might have a problem. A small memory leak in a process that basically has a single path of execution generally is not much of problem. Memory leaks in a long running process, such as an application server however, is another situation entirely.

[hazel]

Thanks, Rubberman. That makes it crystal clear.