I am learning working with shared memory in C++. I found that under Windows I need to use CreateFileMapping and MapViewOfFile functions. I want to share array of char so part of my code is:
HANDLE hBuffer = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0, size, bufferName);
char * buffer = (char *) MapViewOfFile(hBuffer, FILE_MAP_ALL_ACCESS, 0, 0, size);
(there is checking for NULLs of course) and at the end of using shared memory I call:
UnmapViewOfFile(buffer); // returned true
CloseHandle(hBuffer); // returned true also
But in Resource monitor I can see that there was no memory released. When it is called several times, the allocated memory of application is increasing but there is no releasing. What am I doing wrong? Or is there another function to release shared memory?
Thanks for answers.
Problem solved thanks to marcin_j:
Your code looks find (well,... you mispelled HANDLE), you can use
procexp.exe from sysinternals to find your HANDLE by name (if it is
not found then it was closed), also observe on Performance tab how
Virtual Size of you app changes, there is also Handles count that
should change accordingly.
Also, observe what will happen after you execute
memset(buffer,0,size); after MapViewOfFile - this is actually when
system will commit memory and when your workin set will rise.
My above comment is wrong, CreateFileMapping by default applies
SEC_COMMIT which commits memory. But I suppose your memory is paged
until memset is called, after that call paged pages are moved to
physical memory, which rises working set.... if I am not wrong...
Related
To improve the security of my application, I am trying to delete string data from the process memory, but since there is little information about this on the Internet, I could not write a working code.
Can anyone help me?
My pasted code:
void MemoryStringsClear() {
HANDLE hProc = OpenProcess(PROCESS_ALL_ACCESS, FALSE, GetCurrentProcessId());
MEMORY_BASIC_INFORMATION mbi;
char* addr = 0;
while (VirtualQueryEx(hProc, addr, &mbi, sizeof(mbi)))
{
if (mbi.State != MEM_COMMIT || mbi.Protect == PAGE_NOACCESS)
{
//char* buffer = new char[mbi.RegionSize];
//ReadProcessMemory(hProc, addr, buffer, mbi.RegionSize, nullptr);
if (addr) {
cout << "Addr: " << &addr << " is cleared!" << endl;
memset(addr, '0', mbi.RegionSize);
}
}
addr += mbi.RegionSize;
}
CloseHandle(hProc);
}
EDITED:
I chose this way of solving the problem because my application consists of many modules (.exe applications), some of which I cannot change.
There are some problems with your approach (my idea for a solution is further down):
Most of the strings listed are environment variables
All of the programs that run on your computer have access to those. They are copied to the memory space of every program on startup so every program knows where to look for certain files. There is no point in removing them from the memory of your application, since every application running on your computer already knows them.
You can see them by running cmd.exe, typing set and then pressing return.
OpenProcess and VirtualQueryEx are for accessing another process
You could simply use VirtualQuery, since you only want to access your own process.
I guess you are trying to get access to non-committed memory pages by doing this, but memset can only access committed, writable memory pages in your own program's address space. So those two approaches don't mix.
But there is a more important point to this:
Non-committed memory does not exist
If a memory page is not committed, there is no actual memory assigned to that address. That means, that there is nothing you can overwrite with zeroes. The memory containing your strings may already have been assigned to another application. Read some information about virtual memory management for details.
Most calls to free, delete or garbage collection do not always actually decommit the page
For efficiency reasons, when your code allocates and deallocates memory, your runtime library hands you down little scraps of a larger page of memory (called "heap") that is only decommitted if every single piece in it has been freed.
You could find freed blocks of memory by walking over the heap entries, but how that works depends on your C runtime library or other runtime libraries.
The operating system might move your strings around
If the operating systems detects that there is a shortage of memory, it can save your strings to disk to free up memory for other applications, and reloads them when your application again becomes active. It usually does not bother to clean the disk up afterwards. You have no influence on that (unless you format your hard drive).
My ideas for a solution
Before every call to free or delete in your code that frees
memory with sensitive information (and only those), you can call
memset(...) on that single block of memory. In C++, you can wrap that up in a class which clears its memory on destruction, as Alan Birtles pointed out in his comment.
I don't think there is a solution that you can simply pop onto an existing program that clears sensitive information after the memory has been freed.
This approach leaves only the last problem. You can only circumvent that if you never store your sensitive information unencrypted in memory. That is probably not feasible since that would mean that you do not handle it only encrypted.
What will be difficult or impossible
If you want to clear freed memory in other processes (the separate *.exe files you cannot change you refer to in your edit), you have to understand the internal heap layout of those and use WriteProcessMemory instead of memset.
But this does not catch the case where the other program actually decommits a page, since you do not know if the operating system has already reassigned it. When this happens is completely outside of your control.
You might also try to reimplement the free and delete functions in your C runtime library so they first clear the memory and then call the original version, but this only works if they are actually used by those *.exe files and they are dynamically linked. If these conditions are met, you might still have a hard time.
Define the security threats you want to protect against
To improve the security of my application,
What exactly are you trying to guard against? Have you verified that clearing process memory will actually work against the security attacks that you want to defend against?
Know how memory works
Find out how your operating system allocates both virtual and physical memory, otherwise wrong assumptions of how it works might cause you to implement ineffective solutions. Most computers systems use virtual memory, which means some of your memory might actually end up being copied to different places in physical RAM or to disk. On the other hand, if your process exits and a new process starts, most operating systems will clear the RAM used by the first process before assigning it to the second.
Ensure you have full control over the memory you want to clear
As Iziminza already mentioned, your process has virtual memory, but the operating system can choose how to back that virtual memory with physical memory. When it needs RAM for some other process, it can decide to move your data to a swap file on disk until it is needed again. In order to make clearing of memory using memset() meaningful, you must ensure there are no copies stored elsewhere. You can do this by using VirtualLock() on Windows, or mlock() on other operating systems. Even then, if the computer is going into hibernation mode, even locked memory is written to disk.
I am sharing a memory between my driver and application.
After I did Allocating and Mapping, I clean up with Unmapping and Free.
I found that Allocating and Mapping would allocate a brand new virtual memory page in the application, which can be accessed by both driver and application, but it does not disappear after I did the clean up.
I want to know if I should expect the the page to be free after I did clean up.
I saw that MmAllocatePagesForMdlEx and MmMapLockedPagesSpecifyCache will allocate a brand new page of virtual memory in application, which did not appear in the memory list before. After I clean up with MmUnmapLockedPages and MmFreePagesFromMdl, I still see that allocated memory in the list, which is still able to be read and write by the VC++ debugger and application.
I have think of if the clean up failed, but another thing makes me confused. The clean up would zero the memory page, which makes me think it actually is working.
And there is one more thing I did. I use a tool called CE to read the memory but the result is it cannot be read. CE is using ReadProcessMemory to read the memory. So I again get confused.
To summarize, I did
Allocate - MmAllocatePagesForMdlEx
Map - MmMapLockedPagesSpecifyCache
Unmap - MmUnmapLockedPages
Free - MmFreePagesFromMdl
The result is
VC++ debugger can read the cleaned up memory
Application itself can read the cleaned up memory
CE which use standard API cannot read the cleaned up memory
CE which use standard API can see the memory info in the memory list
The method I used to test reading cleaned up memory
VC++ debugger - in memory tab, goto address
Application - value = ^(ULONG64^)addr; //star disabled
The main code are as below:
mdl = MmAllocatePagesForMdlEx(least, most, least, totalBytes, MmCached, 0);
MmGetSystemAddressForMdlSafe(mdl, NormalPagePriority);
MmUnmapLockedPages(userAddr, mdl);
MmFreePagesFromMdl(mdl);
I expected that after I did the last two code mentioned above, the allocated memory would be free because even the mapped physical memory is being free.
So again my question is, should I expect the cleaned up memory become No Access in application / usermode space.
How do I create a file in virtual memory and can I use normal file functions on it as CreateFile, WriteFile etc. ?
I am trying to implement a buffered file writer class, but the problem is managing a buffer that needs dynamic constant reallocation.
You could use a pipe. They are also considered "files" by the os, and you use WriteFile and ReadFile with them:
HANDLE hReadPipe;
HANDLE hWritePipe;
CreatePipe(&hReadPipe, &hWritePipe , 0, 0);
WriteFile(hWritePipe, "hello", 5, 0, 0);
int sz = GetFileSize(hReadPipe, 0);
assert(sz == 5);
char out[5];
ReadFile(hReadPipe, out, 5, 0,0);
You can even convert them to C FILE* handles with something along the lines of _open_osfhandle if desired.
For C++, just use a std::[io]stringstream.
For the Windows API, you can call CreateFile with the FILE_ATTRIBUTE_TEMPORARY flag, and it'll try (but not guarantee) to keep the content in RAM instead of on disk.
As far as I think,you want the whole file inside Virtual Memory? But it feels a little difficult to me, and thus I have got another workaround for it.You can use VirtualAllocExNuma to load your file in memory and work on it.Since Virtual Memory is that part of the memory which is just supposed to be retained by OS but in physically it is scattered around different Non Uniform memory devices besides RAM.It is not a filesystem where you can save your files and how it is managed is also difficult to be traced(at least for me!).
VirtualAllocExNuma
Reserves or commits a region of memory within the virtual address
space of the specified process, and specifies the NUMA node for the
physical memory.
I have 2 questions concerns about using shared memory. I'm using CreateFileMapping to create a shared memory area between two processes.
1) I understand that I need to call CloseHandle on every handle returned from a CreateFileMapping or OpenFileMapping call in order to release the memory. My question is, do all handles get closed appropriately and mem deallocated by Windows XP/7 if the programs using the shared memory exit without calling CloseHandle? IE - is there a possibility of a mem leak after all processes using the mem have been closed?
2) I use MapViewofFile to get a pointer to the mem. In 1 instance I've assumed that the shared memory will always exist in the context of a method. So I've saved the return value of MapViewOfFile as a pointer and closed the handle to the mem and am just using the pointer to the shared mem (but still locking access to it). Is this safe, or should I call MapViewOfFile every time I access the shared mem?
Thanks,
Ian
1) Yes, all handles are closed when a process terminates, no matter if it dies or finishes nicely. No leaks here.
2) As long as you don't call UnmapViewOfFile, the memory will still be accesible to the process, even if the handle has been closed:
Although an application may close the file handle used to create a file mapping object, the system holds the corresponding file open until the last view of the file is unmapped
I currently try to implement some interprocess communication using the Windows CreateFileMapping mechanism. I know that I need to create a file mapping object with CreateFileMapping first and then create a pointer to the actual data with MapViewOfFile. The example then puts data into the mapfile by using CopyMemory.
In my application I have an image buffer (1 MB large) which I want to send to another process. So now I inquire a pointer to the image and then copy the whole image buffer into the mapfile. But I wonder if this is really necessary. Isn't it possible to just copy an actual pointer in the shared memory which points to the image buffer data? I tried a bit but didn't succeed.
Different processes have different address spaces. If you pass a valid pointer in one process to another process, it will probably point to random data in the second process. So you will have to copy all the data.
I strongly recommend you use Boost::interprocess. It has lots of goodies to manage this kind of stuff & even includes some special Windows-only functions in case you need to interoperate w/ other processes that use particular Win32 features.
The most important thing is to use offset pointers rather than regular pointers. Offset pointers are basically relative pointers (they store the difference between where the pointer is and where the thing pointed to is). This means that even if the two pointers are mapped to different address spaces, as long as the mappings are identical in structure then you are fine.
I've used all kinds of complicated data structures with offset smart pointers and it worked like a charm.
Shared Memory doesn't mean sending and receiving of Data. Its a memory created for number of processes without violation. For that you have to follow some mechanisms like locks so that the data will not corrupt.
In process 1 :
CreateFileMapping() : It will create the Shared Memory Block, with the name provided in last parameter, if it is not already present and returns back a handle (you may call it a pointer), if successful.
MapViewOfFile() : It maps (includes) this shared block in the process address space and returns a handle (again u can say a pointer).
With this pointer returned by MapViewOfFile() only you can access that shared block.
In process 2 :
OpenFileMapping() : If the shared memory block is successfully created by CreateFileMapping(), you can use it with the same name (name used to create the shared memory block).
UnmapViewOfFile() : It will unmap (you can remove the shared memory block from that process address space). When you are done using the shared memory (i.e. access, modification etc) call this function .
Closehandle() : finally to detach the shared memory block from process , call this with argument,handle returned by OpenFileMapping() or CreateFileMapping().
Though these functions look simple, the behaviour is tricky if the flags are not selected properly.
If you wish to read or write shared memory, specify PAGE_EXECUTE_READWRITE in CreateFileMapping().
Whenever you wish to access shared memory after creating it successfully, use FILE_MAP_ALL_ACCESS in MapViewOfFile().
It is better to specify FALSE (do not inherit handle from parent process) in OpenFileMapping() as it will avoid confusion.
You CAN get shared memory to use the same address over 2 processes for Windows. It's achieveable with several techniques.
Using MapViewOfFileEx, here's the significant experpt from MSDN.
If a suggested mapping address is
supplied, the file is mapped at the
specified address (rounded down to the
nearest 64K-boundary) if there is
enough address space at the specified
address. If there is not enough
address space, the function fails.
Typically, the suggested address is
used to specify that a file should be
mapped at the same address in multiple
processes. This requires the region of
address space to be available in all
involved processes. No other memory
allocation can take place in the
region that is used for mapping,
including the use of the VirtualAlloc
or VirtualAllocEx function to reserve
memory.
If the lpBaseAddress parameter
specifies a base offset, the function
succeeds if the specified memory
region is not already in use by the
calling process. The system does not
ensure that the same memory region is
available for the memory mapped file
in other 32-bit processes.
Another related technique is to use a DLL with a section marked Read + Write + Shared. In this case, the OS will pretty much do the MapViewOfFileEx call for you and for any other process which loads the DLL.
You may have to mark your DLL to a FIXED load address, not relocateable etc.. naturally.
You can use Marshalling of pointers.
If it's possible, it would be best to have the image data loaded/generated directly into the shared memory area. This eliminates the memory copy and puts it directly where it needs to be. When it's ready you can signal the other process, giving it the offset into your shared memory where the data begins.