Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 19
During runtime when a program asks for a memory block (say, by malloc()), its provided with a block of virtual addresses, say 0x10000000 to 0x1000A00 (for 32 bit). During the ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    May 2010
    Posts
    12

    [SOLVED] When page's retrieved from swap, how does its application know new virtual a


    During runtime when a program asks for a memory block (say, by malloc()), its provided with a block of virtual addresses, say 0x10000000 to 0x1000A00 (for 32 bit). During the execution, other processes are incorporated into the main memory and pages 0x10000000-0x100000A0 are paged out. After some time, the program requests the page 0x10000000. Its incorporated back into the main memory (at the cost of some other page) but its assigned a new virtual address, but the program is oblivious of the new virtual address assigned to it and references the it using 0x10000000. How is this problem resolved ? Does it proceed in any other way ?

  2. #2
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,346
    To my understanding, the virtual addresses stay the same for a given process.

    The kernel will map them to real ram or swap.
    These real addresses can change and the kernel keeps track, where a give virtual address for each process is at any given time.
    You must always face the curtain with a bow.

  3. #3
    Just Joined!
    Join Date
    May 2010
    Posts
    12
    But that would imply that if all of 2^32 addresses in the virtual memory page table fill up, and none of the process has ended, no more processes can be accomodated in the RAM (as there are no more virtual addresses to assign the new process and each page, whether its in the memory or in the swap space, is still assigned a unique virtual address)

    Also, if a process allocated 20 pages (say 0-19) finishes, and a new process comes requiring 70 pages. The virtual page table has to assign these 20 pages (0-19) to it and then leave a lot of pages in between and then assign pages (say, 2000-2049) to it. Thus, the virtual addresses allocated to the process will not remain consecutive. This will be a problem, I think. Does it happen so that if the kernel can't find all 70 pages contiguously, it denies allocating the process any pages and hence a place in the main memory.

  4. #4
    Linux Guru Cabhan's Avatar
    Join Date
    Jan 2005
    Location
    Seattle, WA, USA
    Posts
    3,252
    Let's imagine a process. It has some set of memory divided into pages. Let's look at page 0. This page holds some amount of memory, we'll imagine that it contains the first 256 bytes, so it holds memory addresses 0x0 - 0xFF.

    However, this is a lie! The process thinks that it is accessing address page 0, but it is not: it might be asking for physical page 8. The MMU, when asked for a particular byte, uses a set of data called page tables to map a "virtual" byte into the actual physical address, which could be anything at all (we'll pretend it's on page 8).

    Now, eventually, this page might get swapped out. At this point, the mapping is erased from the page table. The next time that the process requests its page 0, the MMU will see that it has no mapping, so it will page fault and the CPU will get the data from swap. However, it might write this data to ANY page. Earlier, it was on page 8, but the CPU decides to put it on page 3 this time (because page 3 is available). So in the page table, a new mapping is created: the process's page 0 now maps to physical page 3.

    Although virtual pages appear to be contiguous, the physical pages do not need to be so. So virtual pages 0 and 1 are accessed as though they are next to each other, but the physical pages that they are mapped to might be 13 and 2, or anything. No guarantees can be made. In fact, it is almost guaranteed that your virtual memory will be split up among separate physical pages.

    Also recall that every process has its own set of page tables. So every process has its own set of 2^32 addresses that have independent mappings. When the CPU switches from one process to another, the page tables are wiped, and all pages need to be re-mapped, which makes changing from one process to another very expensive.

  5. #5
    Just Joined!
    Join Date
    May 2010
    Posts
    12
    Thanks a ton! I have one doubt though.

    Suppose the contents of page 0 are swapped out and are replaced by some other data, when the process references page 0 by the virtual address 0x00000000, it'll not be knowing that the original contents of page 0 have been swapped out. Since it uses virtual addresses (since virtual addresses don't disappear, how can the process know that its a page fault), how can it know that the value at that virtual address is the original one or has been replaced.

  6. #6
    Linux Newbie unlimitedscolobb's Avatar
    Join Date
    Jan 2008
    Posts
    120
    Quote Originally Posted by daudiam View Post
    Suppose the contents of page 0 are swapped out and are replaced by some other data, when the process references page 0 by the virtual address 0x00000000, it'll not be knowing that the original contents of page 0 have been swapped out.
    This is okay. Processes don't even know that their memory is actually virtual, because it's the kernel who manages the memory.

    Quote Originally Posted by daudiam View Post
    Since it uses virtual addresses (since virtual addresses don't disappear, how can the process know that its a page fault),
    As I said in the previous paragraph, processes don't realize they are actually running in virtual memory. This means, of course, that a process cannot know whether its attempt to access a certain address generates a page fault.

    Quote Originally Posted by daudiam View Post
    how can it know that the value at that virtual address is the original one or has been replaced.
    It's not the process's business. The idea is that, like Cabhan says, each process has its own set of memory pages and thus page tables. When a process tries to access a certain (virtual) address, an attempt to translate this virtual address to a physical one is made. If the corresponding page is found to be paged out, the kernel brings it back. As far as I remember, on i686 (maybe i386 and amd64, but I can't tell for sure off the top of my head) address mapping is incorporated into the hardware (Memory management unit - Wikipedia, the free encyclopedia), this means that the translation happens pretty transparently for the software. When a page is found to be swapped out, the hardware notifies the kernel, which then brings the page in.

    Please note that, in the ideal situation, no one can replace the value at a virtual address belonging to the memory space of a certain process, because only the kernel and the process itself have access to this (virtual) memory space.

  7. #7
    Just Joined!
    Join Date
    May 2010
    Posts
    12

    Isn't virtual memory infinite (all processes can be assigned 2GiB)

    Thanks. All this time, by page I was understanding the pages in the virtual memory table, though the pages in the memory were meant.

    So, I get these things :

    1. Any process larger than 2GiB can't be accommodated since each process is given only 2GiB virtual addresses.

    2. If the swap space as well as the main memory gets filled up, the kernel gives an error saying "Virtual memory is low" (Actually, shouldn't it say that "swap is too small" as virtual memory is infinite)

    3. The reason swap space is maintained is that if a page fault occurs, the page is first located in the swap space as compared to searching the entire hard disk. If its not found there, then the remaining hard disk is checked. This checking, though occurs just for the first time when the process is being loaded, because otherwise, the only reason for the page to be not present in either the swap or the main memory is that it has ended or the process has been killed.

    4. Even when a process swaps out of the memory, the page table for that process does keep some info about where that page is in the swap

    5. I take it that "increasing virtual memory" is actually a misnomer because virtual memory is infinite. What is meant is increasing the swap space.

    Am I right?

  8. #8
    Linux Newbie unlimitedscolobb's Avatar
    Join Date
    Jan 2008
    Posts
    120
    Quote Originally Posted by daudiam View Post
    Thanks. All this time, by page I was understanding the pages in the virtual memory table, though the pages in the memory were meant.
    Speaking for myself, I did call both virtual and physical pages just "pages", which might have been confusing, sorry.

    Quote Originally Posted by daudiam View Post
    1. Any process larger than 2GiB can't be accommodated since each process is given only 2GiB virtual addresses.
    Right.

    Quote Originally Posted by daudiam View Post
    2. If the swap space as well as the main memory gets filled up, the kernel gives an error saying "Virtual memory is low" (Actually, shouldn't it say that "swap is too small" as virtual memory is infinite)
    The circumstances for the error seem right (though I've never researched this matter on Linux specifically). Perhaps making the error talk about swap space rather than virtual memory may have sense. I'd like to note though, that by "virtual memory" people usually understand "virtual memory of a process", so it probably is better to explicitly say "the total amount of virtual memory". And, by the way, that amount is limited by the number of processes the system can actually handle (finite value), but that's a different question

    Quote Originally Posted by daudiam View Post
    3. The reason swap space is maintained is that if a page fault occurs, the page is first located in the swap space as compared to searching the entire hard disk. If its not found there, then the remaining hard disk is checked. This checking, though occurs just for the first time when the process is being loaded, because otherwise, the only reason for the page to be not present in either the swap or the main memory is that it has ended or the process has been killed.
    When you say "entire disk", it makes me unsure whether we are talking about the same thing. Again, I'm not sure this 100% holds for Linux, but an OS may not (pre-)load the whole executable from the very beginning. Instead, it might fetch the necessary segments when the process first needs them. This is not searching the entire disk.

    As for the OS not finding a page in the swap, I cannot imagine a situation when this may happen, except for attempts to read or write using virtual addresses in ranges not specifically allocated before. In this case, you get something like a segfault and the OS does not "search the entire disk".

    When a process exits or is killed, the OS deallocates all pages that were allocated for it.

    Thus the swap space is a cheap but slow extension of the main memory. The principal purpose of the swap is to (temporarily) store memory pages which don't fit within the main memory.

    Quote Originally Posted by daudiam View Post
    4. Even when a process swaps out of the memory, the page table for that process does keep some info about where that page is in the swap
    As far as I remember, parts of the page tables belonging to the process (I think, the secondary indirect tables) are written into the swap space, too. When the process is brought back into main memory to run, the page tables are brought in, too, and, as usual, they have the information as to where the physical pages are.

    Not sure whether this makes you feel your statement is confirmed.

    Quote Originally Posted by daudiam View Post
    5. I take it that "increasing virtual memory" is actually a misnomer because virtual memory is infinite. What is meant is increasing the swap space.
    I think yes, "increasing virtual memory" is actually increasing the swap space. Nevertheless, virtual memory is not infinite, see above

  9. #9
    Just Joined!
    Join Date
    Feb 2005
    Posts
    2

    How it really works

    This is really pretty simple. When a process is running, each fetch from virtual memory goes through a translation table loaded into the physical CPU hardware registers. This is called a Translation table (TLB for Intel and AMD hw). This table is part of the processes' context and is stored in a data structure within the kernel. So when User A gets to run it's translation table is either loaded into the Translation table HW in the CPU fully or dynamically. So when an address is translated and there is NO valid entry in the Translation Table the OS takes a fault. If the entry is in a data structure in the kernel then it's loaded into the Translation Table, if not it becomes a page fault. Then the Page Fault goes out to disk to fetch the page into memory. When that happens the users (Process) translation table in Kernel is updated. When that user gets swapped back in then the translation happens and the process continues.


    You are also confusing Virtual Memory with Swap space. Virtual memory is a scheme created by the HW manufacturer. OS's conform to that structure. Usually Swap space contains data only. Because the executables are non writable and don't need to be saved there is no need to fetch them from Swap.

  10. #10
    Linux Newbie unlimitedscolobb's Avatar
    Join Date
    Jan 2008
    Posts
    120
    Quote Originally Posted by FastFreddie View Post
    Because the executables are non writable and don't need to be saved there is no need to fetch them from Swap.
    How about the data segments normally incorporated into executables? Often these segments are writable, as far as I know. So, probably, you are specifically referring to text segment of an executable.

    I hope I understand you correctly.

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •