How do I create a file in virtual memory and can I use normal file functions on it as CreateFile, WriteFile etc. ?
I am trying to implement a buffered file writer class, but the problem is managing a buffer that needs dynamic constant reallocation.
You could use a pipe. They are also considered "files" by the os, and you use WriteFile and ReadFile with them:
HANDLE hReadPipe;
HANDLE hWritePipe;
CreatePipe(&hReadPipe, &hWritePipe , 0, 0);
WriteFile(hWritePipe, "hello", 5, 0, 0);
int sz = GetFileSize(hReadPipe, 0);
assert(sz == 5);
char out[5];
ReadFile(hReadPipe, out, 5, 0,0);
You can even convert them to C FILE* handles with something along the lines of _open_osfhandle if desired.
For C++, just use a std::[io]stringstream.
For the Windows API, you can call CreateFile with the FILE_ATTRIBUTE_TEMPORARY flag, and it'll try (but not guarantee) to keep the content in RAM instead of on disk.
As far as I think,you want the whole file inside Virtual Memory? But it feels a little difficult to me, and thus I have got another workaround for it.You can use VirtualAllocExNuma to load your file in memory and work on it.Since Virtual Memory is that part of the memory which is just supposed to be retained by OS but in physically it is scattered around different Non Uniform memory devices besides RAM.It is not a filesystem where you can save your files and how it is managed is also difficult to be traced(at least for me!).
VirtualAllocExNuma
Reserves or commits a region of memory within the virtual address
space of the specified process, and specifies the NUMA node for the
physical memory.
Related
To improve the security of my application, I am trying to delete string data from the process memory, but since there is little information about this on the Internet, I could not write a working code.
Can anyone help me?
My pasted code:
void MemoryStringsClear() {
HANDLE hProc = OpenProcess(PROCESS_ALL_ACCESS, FALSE, GetCurrentProcessId());
MEMORY_BASIC_INFORMATION mbi;
char* addr = 0;
while (VirtualQueryEx(hProc, addr, &mbi, sizeof(mbi)))
{
if (mbi.State != MEM_COMMIT || mbi.Protect == PAGE_NOACCESS)
{
//char* buffer = new char[mbi.RegionSize];
//ReadProcessMemory(hProc, addr, buffer, mbi.RegionSize, nullptr);
if (addr) {
cout << "Addr: " << &addr << " is cleared!" << endl;
memset(addr, '0', mbi.RegionSize);
}
}
addr += mbi.RegionSize;
}
CloseHandle(hProc);
}
EDITED:
I chose this way of solving the problem because my application consists of many modules (.exe applications), some of which I cannot change.
There are some problems with your approach (my idea for a solution is further down):
Most of the strings listed are environment variables
All of the programs that run on your computer have access to those. They are copied to the memory space of every program on startup so every program knows where to look for certain files. There is no point in removing them from the memory of your application, since every application running on your computer already knows them.
You can see them by running cmd.exe, typing set and then pressing return.
OpenProcess and VirtualQueryEx are for accessing another process
You could simply use VirtualQuery, since you only want to access your own process.
I guess you are trying to get access to non-committed memory pages by doing this, but memset can only access committed, writable memory pages in your own program's address space. So those two approaches don't mix.
But there is a more important point to this:
Non-committed memory does not exist
If a memory page is not committed, there is no actual memory assigned to that address. That means, that there is nothing you can overwrite with zeroes. The memory containing your strings may already have been assigned to another application. Read some information about virtual memory management for details.
Most calls to free, delete or garbage collection do not always actually decommit the page
For efficiency reasons, when your code allocates and deallocates memory, your runtime library hands you down little scraps of a larger page of memory (called "heap") that is only decommitted if every single piece in it has been freed.
You could find freed blocks of memory by walking over the heap entries, but how that works depends on your C runtime library or other runtime libraries.
The operating system might move your strings around
If the operating systems detects that there is a shortage of memory, it can save your strings to disk to free up memory for other applications, and reloads them when your application again becomes active. It usually does not bother to clean the disk up afterwards. You have no influence on that (unless you format your hard drive).
My ideas for a solution
Before every call to free or delete in your code that frees
memory with sensitive information (and only those), you can call
memset(...) on that single block of memory. In C++, you can wrap that up in a class which clears its memory on destruction, as Alan Birtles pointed out in his comment.
I don't think there is a solution that you can simply pop onto an existing program that clears sensitive information after the memory has been freed.
This approach leaves only the last problem. You can only circumvent that if you never store your sensitive information unencrypted in memory. That is probably not feasible since that would mean that you do not handle it only encrypted.
What will be difficult or impossible
If you want to clear freed memory in other processes (the separate *.exe files you cannot change you refer to in your edit), you have to understand the internal heap layout of those and use WriteProcessMemory instead of memset.
But this does not catch the case where the other program actually decommits a page, since you do not know if the operating system has already reassigned it. When this happens is completely outside of your control.
You might also try to reimplement the free and delete functions in your C runtime library so they first clear the memory and then call the original version, but this only works if they are actually used by those *.exe files and they are dynamically linked. If these conditions are met, you might still have a hard time.
Define the security threats you want to protect against
To improve the security of my application,
What exactly are you trying to guard against? Have you verified that clearing process memory will actually work against the security attacks that you want to defend against?
Know how memory works
Find out how your operating system allocates both virtual and physical memory, otherwise wrong assumptions of how it works might cause you to implement ineffective solutions. Most computers systems use virtual memory, which means some of your memory might actually end up being copied to different places in physical RAM or to disk. On the other hand, if your process exits and a new process starts, most operating systems will clear the RAM used by the first process before assigning it to the second.
Ensure you have full control over the memory you want to clear
As Iziminza already mentioned, your process has virtual memory, but the operating system can choose how to back that virtual memory with physical memory. When it needs RAM for some other process, it can decide to move your data to a swap file on disk until it is needed again. In order to make clearing of memory using memset() meaningful, you must ensure there are no copies stored elsewhere. You can do this by using VirtualLock() on Windows, or mlock() on other operating systems. Even then, if the computer is going into hibernation mode, even locked memory is written to disk.
I am learning working with shared memory in C++. I found that under Windows I need to use CreateFileMapping and MapViewOfFile functions. I want to share array of char so part of my code is:
HANDLE hBuffer = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0, size, bufferName);
char * buffer = (char *) MapViewOfFile(hBuffer, FILE_MAP_ALL_ACCESS, 0, 0, size);
(there is checking for NULLs of course) and at the end of using shared memory I call:
UnmapViewOfFile(buffer); // returned true
CloseHandle(hBuffer); // returned true also
But in Resource monitor I can see that there was no memory released. When it is called several times, the allocated memory of application is increasing but there is no releasing. What am I doing wrong? Or is there another function to release shared memory?
Thanks for answers.
Problem solved thanks to marcin_j:
Your code looks find (well,... you mispelled HANDLE), you can use
procexp.exe from sysinternals to find your HANDLE by name (if it is
not found then it was closed), also observe on Performance tab how
Virtual Size of you app changes, there is also Handles count that
should change accordingly.
Also, observe what will happen after you execute
memset(buffer,0,size); after MapViewOfFile - this is actually when
system will commit memory and when your workin set will rise.
My above comment is wrong, CreateFileMapping by default applies
SEC_COMMIT which commits memory. But I suppose your memory is paged
until memset is called, after that call paged pages are moved to
physical memory, which rises working set.... if I am not wrong...
I have a block of memory which I have allocated using mmap. I then want to write this block of memory to a file on the file system. I am wondering if there is a way to point the existing block of memory directly to a file descriptor using mmap() or will I have to copy the whole block of memory to a new mmap() area which has been created with the file descriptor?
You can use vmsplice with SPLICE_F_GIFT to place the memory into a pipe, after that splice or copy_file_range may do what is required.
You will have top copy it, unless you can create the mmap with a temporary file in the first place, and then just rename the file when you want to "save" it.
I'm not familiar with all implementations, I have looked quite closely at the Linux implementation, but I believe this applies in all cases: When you do not have a file when you create a mmap region, the OS will use the "swap" as a file for the mmap region. Since there is a mapping between the file-content and the memory content, you can't trivially just change what that mapping is (although it could, perhaps, be technically possible to achieve, it could be rather "expensive" if you have a 4GB mmap region, where only a small part of it is in physical RAM, the rest is "on disk").
Assuming we're talking pure POSIX, there's no way to attach an mmap'd region to a new fd. I'm also not aware of any way to do this in Linux or OS X.
You'll have to either copy, or mmap the file that you want to write to in the first place. If you don't want the file to appear immediately for some reason, there's the following workaround:
open a temporary file in the filesystem where you want the final file (the same directory is a safe bet)
mmap it, modify it, write out changes
rename the file to its final location.
I'm very new to Memory Mapped Files, and I'm a little lost on something.
I know that if I had a file, I could load it and access it from various processes at once using MMaps.
But in my situation, I'm creating a DLL attached to Process A, and that DLL has been given a pointer to a cSurface which Process A has prepared. I need to share that cSurface's data with Process B. I really don't want to have to call up a blank MMap and copy my Process A's surface into it, only to copy it out again in process B.
Is it possible to map my surface as if it were a file so the MMap already points to the surface data when it's created (as it would were I loading SomeTextFile.txt)?
My plan, in theory, would be to receive a pointer to the surface in Proc A, tell windows to share that surface's memory with a given name, and use Mutexes to coordinate access - the idea being that both processes read the same physical copy of the surface with no cumbersome copying.
Is that possible?
Yes, it's all there in the documentation of CreateFileMapping.
You can give your memory mapped file a name. If another process open a mmap with the same name it will point to the same memory. It the handle in CreateFileMapping is put to INVALID_HANDLE_VALUE it keeps it purely in memory. Check the documentation of CreateFileMapping
Thanks for all your comments.
I did some further research and found the answer.
You can use Memory Mapped Files to share either a file, or a blank memory space. If you want to share data already initialised in memory prior to setting up the map, you have to create a blank map and subsequently copy your data into it.
I currently try to implement some interprocess communication using the Windows CreateFileMapping mechanism. I know that I need to create a file mapping object with CreateFileMapping first and then create a pointer to the actual data with MapViewOfFile. The example then puts data into the mapfile by using CopyMemory.
In my application I have an image buffer (1 MB large) which I want to send to another process. So now I inquire a pointer to the image and then copy the whole image buffer into the mapfile. But I wonder if this is really necessary. Isn't it possible to just copy an actual pointer in the shared memory which points to the image buffer data? I tried a bit but didn't succeed.
Different processes have different address spaces. If you pass a valid pointer in one process to another process, it will probably point to random data in the second process. So you will have to copy all the data.
I strongly recommend you use Boost::interprocess. It has lots of goodies to manage this kind of stuff & even includes some special Windows-only functions in case you need to interoperate w/ other processes that use particular Win32 features.
The most important thing is to use offset pointers rather than regular pointers. Offset pointers are basically relative pointers (they store the difference between where the pointer is and where the thing pointed to is). This means that even if the two pointers are mapped to different address spaces, as long as the mappings are identical in structure then you are fine.
I've used all kinds of complicated data structures with offset smart pointers and it worked like a charm.
Shared Memory doesn't mean sending and receiving of Data. Its a memory created for number of processes without violation. For that you have to follow some mechanisms like locks so that the data will not corrupt.
In process 1 :
CreateFileMapping() : It will create the Shared Memory Block, with the name provided in last parameter, if it is not already present and returns back a handle (you may call it a pointer), if successful.
MapViewOfFile() : It maps (includes) this shared block in the process address space and returns a handle (again u can say a pointer).
With this pointer returned by MapViewOfFile() only you can access that shared block.
In process 2 :
OpenFileMapping() : If the shared memory block is successfully created by CreateFileMapping(), you can use it with the same name (name used to create the shared memory block).
UnmapViewOfFile() : It will unmap (you can remove the shared memory block from that process address space). When you are done using the shared memory (i.e. access, modification etc) call this function .
Closehandle() : finally to detach the shared memory block from process , call this with argument,handle returned by OpenFileMapping() or CreateFileMapping().
Though these functions look simple, the behaviour is tricky if the flags are not selected properly.
If you wish to read or write shared memory, specify PAGE_EXECUTE_READWRITE in CreateFileMapping().
Whenever you wish to access shared memory after creating it successfully, use FILE_MAP_ALL_ACCESS in MapViewOfFile().
It is better to specify FALSE (do not inherit handle from parent process) in OpenFileMapping() as it will avoid confusion.
You CAN get shared memory to use the same address over 2 processes for Windows. It's achieveable with several techniques.
Using MapViewOfFileEx, here's the significant experpt from MSDN.
If a suggested mapping address is
supplied, the file is mapped at the
specified address (rounded down to the
nearest 64K-boundary) if there is
enough address space at the specified
address. If there is not enough
address space, the function fails.
Typically, the suggested address is
used to specify that a file should be
mapped at the same address in multiple
processes. This requires the region of
address space to be available in all
involved processes. No other memory
allocation can take place in the
region that is used for mapping,
including the use of the VirtualAlloc
or VirtualAllocEx function to reserve
memory.
If the lpBaseAddress parameter
specifies a base offset, the function
succeeds if the specified memory
region is not already in use by the
calling process. The system does not
ensure that the same memory region is
available for the memory mapped file
in other 32-bit processes.
Another related technique is to use a DLL with a section marked Read + Write + Shared. In this case, the OS will pretty much do the MapViewOfFileEx call for you and for any other process which loads the DLL.
You may have to mark your DLL to a FIXED load address, not relocateable etc.. naturally.
You can use Marshalling of pointers.
If it's possible, it would be best to have the image data loaded/generated directly into the shared memory area. This eliminates the memory copy and puts it directly where it needs to be. When it's ready you can signal the other process, giving it the offset into your shared memory where the data begins.