Is there a way I can get information on the virtual memory my application is using?
As far as i'm aware there are several segments of memory that a program uses, such as code segment, data segments and the heap, stack etc.
Is there a way I can get information on the addresses where these segments start and end?
To give some context i'm trying to draw an image which contains a visual representation of the virtual memory of my program (a kind of graph). So i need the start and end addresses of the different segments, and if possible a way to determine if an address is memory is currently being used.
i'm using c++ on windows btw.
You can query memory layout of a process by calling VirtualQueryEx.
It returns the memory map layout you wanted.
The returned MEMORY_BASIC_INFORMATION table list of sections of memory with their protection attributes.
However, the machine works on protection attributes not by its nature(code, data, rodata, bss, heap, etc), you can only guess on them:
PAGE_EXECUTE: code
PAGE_READONLY: rodata
PAGE_READWRITE: data, bss, heap, stack
PAGE_WRITECOPY: data, bss, heap, stack
However, the above guess would not be reliable if the process did it's own customized VirtualAllocate or file mapping, to increase the accurancy you may also query the executable name with GetModuleFileName and parse the file, then compare with the table (note the address space randomization)
Related
I know this function create a "buffer." But what exactly is a buffer? Is it a COM object in memory? If it is, then in my understanding, this function takes in a descriptor and some initial data to create this COM object in memory, and then set the ID3D11Buffer pointer pointed by the input ID3D11Buffer** to the interface in the newly created COM object. Once the COM object is created, the initializing data is not needed any more and we can delete them. And once we call ID3DBuffer::Release(), the underline COM object will be destroyed. Is my understanding correct?
CreateBuffer returns a COM interface object ID3D11Buffer*. As long as it has a non-zero reference count (it starts at 1; each call to AddRef adds 1, each call to Release subtracts 1) then whatever resources it controls are active.
As to where exactly the resources are allocated, it really depends. You may find this article interesting as it covers different ways Direct3D allocates resources.
UPDATE: You should also read this Microsoft Docs introduction to the subset of COM used by DirectX.
In the general case, a buffer is a continuous, managed, area of memory.
Memory is a large set of addresses of read/writable elements (one element per address, of course), say 230 addresses of elements of 8-bit makes a 1GiB memory.
If there is only a single program and it uses these addresses statically (e.g. addresses from 0x1000 to 0x2000 are used to store the list of items) then memory doesn't need to be managed and in this context a buffer is just a continuous range of addresses.
However, if there are multiple programs or a program memory usage is dynamic (e.g. it depends on how many items it's been asked to read from input) then memory must be managed.
You must keep track of which ranges are already in use and which are not. So a buffer becomes a continuous range of addresses with their attributes (e.g. if it's in use or not).
The attributes of a buffer can vary a lot between the different memory allocators, in general, we say that a buffer is managed because we let the memory allocator handle it: find a suitable free range, mark it used, tell it if it can move the buffer aftward, mark it free when where are finished.
This is true for every memory that is shared, so it is certainly true for the main memory (RAM) and the graphic memory.
This is the memory inside the graphic card, that is accessed just like the main memory (from the CPU point of view).
What CreateBuffer return is a COM object in the main memory that contains the metadata necessary to handle the buffer just allocated.
It doesn't contain the buffer itself because this COM object is always in memory while the buffer usually is not (it is in the graphic memory).
CreateBuffer asks the graphic driver to find a suitable range of free addresses, in the memory asked, and fill in some metadata.
Before the CPU can access the main memory it is necessary to set up some metadata tables (the page tables) as part of its protection mechanism.
This is also true if the CPU needs to access the graphic memory (with possibly a few extra steps, for managing the MMIO if necessary).
The GPU also has page tables, so if the main memory has to be accessed by the GPU these page tables must also have to be created.
You see that it's important to know how the buffer will be used.
Another thing to consider is that the GPUs use highly optimized memory format - for example, the buffer used for a surface can be pictured as a rectangular area of memory.
The same is true for the buffer used by a texture.
However the twos are stored differently: the surface is stored linearly, each row after another, while the texture buffer is tiled (it's like it's made of many, say, 16x16 surfaces stored linearly one after the other).
This makes sampling and filtering faster.
Also, some GPU may need to have texture images on a specific area of memory, vertex buffer in another and so on.
So it's important to give the graphic driver all the information it needs to make the best choice when allocating a buffer.
Once the buffer has been found, the driver (or the D3D runtime) will initialize the buffer if requested.
It can do this by copying the data or by aliasing through the page tables (if the pitch allows for it) and eventually using some form of Copy-On-Write.
However it does that, the source data are not needed anymore (see this).
The COM object returned by CreateBuffer is a convenient proxy, then it is disposed of, thanks to the usual come AddRef/Release mechanism, it also asks the graphic driver to deallocate the buffer.
When memory is allocated in a computer, how does it know which bytes are already occupied and can't be overwritten?
So if these are some bytes of memory that aren't being used:
[0|0|0|0]
How does the computer know whether they are or not? They could just be an integer that equals zero. Or it could be empty memory. How does it know?
That depends on the way the allocation is performed, but it generally involves manipulation of data belonging to the allocation mechanism.
When you allocate some variable in a function, the allocation is performed by decrementing the stack pointer. Via the stack pointer, your program knows that anything below the stack pointer is not allocated to the stack, while anything above the stack pointer is allocated.
When you allocate something via malloc() etc. on the heap, things are similar, but more complicated: all theses allocators have some internal data structures which they never expose to the calling application, but which allow them to select which memory addresses to return on an allocation request. Some malloc() implementation, for instance, use a number of memory pools for small objects of fixed size, and maintain linked lists of free objects for each fixed size which they track. That way, they can quickly pop one memory region of that list, only doing more expensive computations when they run out of regions to satisfy a certain request size.
In any case, each of the allocators have to request memory from the system kernel from time to time. This mechanism always works on complete memory pages (usually 4 kiB), and works via the syscalls brk() and mmap(). Again, the kernel keeps track of which pages are visible in which processes, and at which addresses they are mapped, so there is additional memory allocated inside the kernel for this.
These mappings are made available to the processor via the page tables, which uses them to resolve the virtual memory addresses to the physical addresses. So here, finally, you have some hardware involved in the process, but that is really far, far down in the guts of the mechanics, much below anything that a userspace process is ever able to see. Still, even the page tables are managed by the software of the kernel, not by the hardware, the hardware only interpretes what the software writes into the page tables.
First of all, I have the impression that you believe that there is some unoccupied memory that doesn't holds any value. That's wrong. You can imagine the memory as a very large array when each box contains a value whereas someone put something in it or not. If a memory was never written, then it contains a random value.
Now to answer your question, it's not the computer (meaning the hardware) but the operating system. It holds somewhere in its memory some tables recording which part of the memory are used. Also any byte of memory can be overwriten.
In general, you cannot tell by looking at content of memory at some location whether that portion of memory is used or not. Memory value '0' does not mean the memory is not used.
To tell what portions of memory are used you need some structure to tell you this. For example, you can divide memory into chunks and keep track of which chunks are used and which are not.
There are memory blocks, they have an occupied or not occupied. On the heap, there are very complex data structures which organise it. But the answer to your question is too broad.
I have a C++ application that allocates shared memory on a Linux system via shmget(2). The data that I store in the shared memory grows periodically, and I'd like to resize the shared memory in a way analogous to the way realloc() grows regular memory. Is there a way to do this? I found a doc on IBM's site that mentions a SHM_SIZE command, but the Linux and BSD manpages do not have it, even in the Linux-specific sections.
Simple answer: there is no easy way.
The reasons are pretty logical. Shared memory is being attached to virtual space of every process individually. Each process has it's own virtuall address space. Each process is free to attach the segment at any (not literally, alignment sets some restrictions) arbitrary address. How can system guarantee that, let's say by extending area by 4MiB, every 'user' of this segment will be able to fit bigget block at the same starting address where the smaller segment previously was?
But you should not give up! You can be creative. You can come up with the idea of having one header segment, where you store information about real payload segment. You can make every process to obey some rules as for example: reattach payload segment when its id, as described somewhere in header segment, does not match the known one.
The advice: I suspect you know this, but never keep pointers to data within shared region, only offset.
I hope you'll have some use of my gibberish.
It seems for me that you can write your own memory manager for your purpose. The conception is quite simple:
You have a shared memory block which size is N bytes;
Allocate new block of shared memory with 2*N size;
Copy memory from one block to another;
Free the old shared memory block;
Wrap the #2-4 into some routine and use it;
I'm afraid we have nothing more to do with that. This is how std::vector is implemented. And void *realloc() in most of cases will return you the pointer to the new block of memory (but not to the extended old block).
It seems to me that the function mremap was implemented to perform what you want.
You just have to precise in argument the old size and the new one of the shared memory segment. And if you add the flag MREMAP_MAYMOV, it allows to move the shared memory segment if needed (i.e if not enough free space just after the old shared memory segment).
Look at the manpage : http://man7.org/linux/man-pages/man2/mremap.2.html.
I'll be messing around with memory sharing, and the godcake wants me to do what the title says, because above 20GB I'll allocate 256MB blocks(no other size, ever) to certain users and mmap(share) them with other processes. The 64bit virtual address space is HUGE (16EB). Its just that I need to keep the user manager's code and data somewhere seperated from the users' memory.
Perfect would be some function like this:
void *malloc(size_t n, void *from, void *to);
Supposing you are on a system with POSIX support, you want to create or locate a shared memory segment with shmget, attach to it with shmat. When done, you detach with shmdt and remove the shared memory segment with shmctl. I leave it to you to read the documentation for these routines.
While shmat allows you to request a specific address at which to map the shared memory, there is generally no need to do so. You can pass zero for the address parameter, and shmat will pick an address. Setting a high address does nothing to separate the shared memory from other memory, except possibly to reduce or alter the errors that could be caused by buffer overruns writing to unintended places in memory.
I'm writing a simple program that accesses the memory of another process. I have been using a memory editor to find the addresses of the variables I want my program to retrieve and use with the ReadProcessMemory function. So far, there have been no problems, but I am unsure whether the addresses of the values may change depending on the environment the other program is being run on.
Aside from alterations to the program itself, should I be concerned about this? I have noticed that my memory editor saves the addresses relative to the location of the .exe (such as program.exe+198F6C), and I would love to implement my program like this, but I could not find any method for retrieving the current address of program.exe in C++.
Yes, they change.
The OS loads the process into different offsets each time it launches, and anything allocated with new or malloc is very likely to get different addresses each time the code is run.
There are two issues here: location of variables inside a process's memory space, and the location of a process in physical memory. The first should concern you, the second should not.
Local variables (as well as global/static variables) will have the same address relative to the program location in memory. Dynamically allocated variables (new/malloc) will have different addresses each time.
When I say "memory", I mean the virtual memory space of a specific process: the address 0x100 in one process doesn't equal 0x100 in another process, and in general is different than cell number 0x100 in your RAM.
The actual address isn't usually interesting, because both ReadProcessMemory and your memory editor only work with those relative addresses. You don't need the location of program.exe.
If you're interested in local variables, you can count on ReadProcessMemory returning a meaningful result each time. If you need memory which has been dynamically allocated, you need to find a local pointer, get the address of the allocated memory from it, and call ReadProcessMemory again.
Yes, they will change. Write a program that outputs the memory address of a few variables and run it a few times. Your output should differ each time, especially on other machines.
You are also going to run into concurrency problems with multiple accesses of the same memory area.
Correct order - W1a, W1b,R1a,R1b,W2a,W2b,R2a,R2b
Incorrect order - W1a,W1b,R1a,W2a,W2b,R1b,R2a,R2b
To solve this problem you need to look at IPC, Inter Processor Communication:
http://en.wikipedia.org/wiki/Inter-process_communication