I have some Visual C++ code that receives a pointer to a buffer with data that needs to be processed by my code and the length of that buffer. Due to a bug outside my control, sometimes this pointer comes into my code uninitialized or otherwise unsuitable for reading (i.e. it causes a crash when I try to access the data in the buffer.)
So, I need to verify this pointer before I use it. I don't want to use IsBadReadPtr or IsBadWritePtr because everyone agrees that they're buggy. (Google them for examples.) They're also not thread-safe -- that's probably not a concern in this case, though a thread-safe solution would be nice.
I've seen suggestions on the net of accomplishing this by using VirtualQuery, or by just doing a memcpy inside an exception handler. However, the code where this check needs to be done is time sensitive so I need the most efficient check possible that is also 100% effective. Any ideas would be appreciated.
Just to be clear: I know that the best practice would be to just read the bad pointer, let it cause an exception, then trace that back to the source and fix the actual problem. However, in this case the bad pointers are coming from Microsoft code that I don't have control over so I have to verify them.
Note also that I don't care if the data pointed at is valid. My code is looking for specific data patterns and will ignore the data if it doesn't find them. I'm just trying to prevent the crash that occurs when running memcpy on this data, and handling the exception at the point memcpy is attempted would require changing a dozen places in legacy code (but if I had something like IsBadReadPtr to call I would only have to change code in one place).
bool IsBadReadPtr(void* p)
{
MEMORY_BASIC_INFORMATION mbi = {0};
if (::VirtualQuery(p, &mbi, sizeof(mbi)))
{
DWORD mask = (PAGE_READONLY|PAGE_READWRITE|PAGE_WRITECOPY|PAGE_EXECUTE_READ|PAGE_EXECUTE_READWRITE|PAGE_EXECUTE_WRITECOPY);
bool b = !(mbi.Protect & mask);
// check the page is not a guard page
if (mbi.Protect & (PAGE_GUARD|PAGE_NOACCESS)) b = true;
return b;
}
return true;
}
a thread-safe solution would be nice
I'm guessing it's only IsBadWritePtr that isn't thread-safe.
just doing a memcpy inside an exception handler
This is effectively what IsBadReadPtr is doing ... and if you did it in your code, then your code would have the same bug as the IsBadReadPtr implementation: http://blogs.msdn.com/oldnewthing/archive/2006/09/27/773741.aspx
--Edit:--
The only problem with IsBadReadPtr that I've read about is that the bad pointer might be pointing to (and so you might accidentally touch) a stack's guard page. Perhaps you could avoid this problem (and therefore use IsBadReadPtr safely), by:
Know what threads are running in your process
Know where the threads' stacks are, and how big they are
Walk down each stack, delberately touching each page of the stack at least once, before you begin to call isBadReadPtr
Also, the some of the comments associated with the URL above also suggest using VirtualQuery.
The reason these functions are bad to use is that the problem can't be solved reliably.
What if the function you're calling returns a pointer to memory that is allocated, so it looks valid, but it's pointing to other, unrelated data, and will corrupt your application if you use it.
Most likely, the function you're calling actually behaves correctly, and you are misusing it. (Not guaranteed, but that is often the case.)
Which function is it?
The fastest solution I can think of is to consult the virtual memory manager using VirtualQuery to see if there is a readable page at the given address, and cache the results (however any caching will reduce the accuracy of the check).
Example (without caching):
BOOL CanRead(LPVOID p)
{
MEMORY_BASIC_INFORMATION mbi;
mbi.Protect = 0;
::VirtualQuery(((LPCSTR)p) + len - 1, &mbi, sizeof(mbi));
return ((mbi.Protect & 0xE6) != 0 && (mbi.Protect & PAGE_GUARD) == 0);
}
Why can't you call the api
AfxIsValidAddress((p), sizeof(type), FALSE));
If the variable is uninitialized you are hosed. Sooner or later it's going to be an address for something you don't want to play with (like your own stack).
If you think you need this, and (uintptr_t)var < 65536 does not suffice (Windows does not allow allocating the bottom 64k), there is no real solution. VirtualQuery, etc. appear to "work" but sooner or later will burn you.
I am afraid you are out of luck - there is no way to reliably check the validity of a pointer. What Microsoft code is giving you bad pointers?
Any implementation of checking the validity of memory is subject to the same constriants that make IsBadReadPtr fail. Can you post an example callstack of where you want to check the validity of memory of a pointer passed to you from Windows? That might help other people (including me) diagnose why you need to do this in the first place.
If you have to resort to checking patterns in data, here are a few tips:
If you mention using IsBadReadPtr, you are probably developing for Windows x86 or x64.
You may be able to range check the pointer. Pointers to objects will be word aligned. In 32-bit windows, user-space pointers are in the range of 0x00401000-0x7FFFFFFF, or for large-address-aware applications, 0x00401000-0xBFFFFFFF instead (edit: 0x00401000-0xFFFF0000 for a 32-bit program on 64-bit windows). The upper 2GB/1GB is reserved for kernel-space pointers.
The object itself will live in Read/Write memory which is not executable. It may live in the heap, or it may be a global variable. If it is a global variable, you can validate that it lives in the correct module.
If your object has a VTable, and you are not using other classes, compare its VTable pointer with another VTable pointer from a known good object.
Range check the variables to see if they are possibly valid. For example, bools can only be 1 or 0, so if you see one with a value of 242, that's obviously wrong. Pointers can also be range checked and checked for alignment as well.
If there are objects contained within, check their VTables and data as well.
If there are pointers to other objects, you can check that the object lives in memory that is Read/Write and not executable, check the VTable if applicable, and range check the data as well.
If you do not have a good object with a known VTable address, you can use these rules to check if a VTable is valid:
While the object lives in Read/Write memory, and the VTable pointer is part of the object, the VTable itself will live in memory that is Read Only and not executable, and will be aligned to a word boundary. It will also belong to the module.
The entries of the VTable are pointers to code, which will be Read Only and Executable, and not writable. There is no alignment restrictions for code addresses. Code will belong to the module.
Here is what I use this just replaces the official microsoft ones by using #define's this way you can use the microsoft ones and not worry about them failing you.
// Check memory address access
const DWORD dwForbiddenArea = PAGE_GUARD | PAGE_NOACCESS;
const DWORD dwReadRights = PAGE_READONLY | PAGE_READWRITE | PAGE_WRITECOPY | PAGE_EXECUTE_READ | PAGE_EXECUTE_READWRITE | PAGE_EXECUTE_WRITECOPY;
const DWORD dwWriteRights = PAGE_READWRITE | PAGE_WRITECOPY | PAGE_EXECUTE_READWRITE | PAGE_EXECUTE_WRITECOPY;
template<DWORD dwAccessRights>
bool CheckAccess(void* pAddress, size_t nSize)
{
if (!pAddress || !nSize)
{
return false;
}
MEMORY_BASIC_INFORMATION sMBI;
bool bRet = false;
UINT_PTR pCurrentAddress = UINT_PTR(pAddress);
UINT_PTR pEndAdress = pCurrentAddress + (nSize - 1);
do
{
ZeroMemory(&sMBI, sizeof(sMBI));
VirtualQuery(LPCVOID(pCurrentAddress), &sMBI, sizeof(sMBI));
bRet = (sMBI.State & MEM_COMMIT) // memory allocated and
&& !(sMBI.Protect & dwForbiddenArea) // access to page allowed and
&& (sMBI.Protect & dwAccessRights); // the required rights
pCurrentAddress = (UINT_PTR(sMBI.BaseAddress) + sMBI.RegionSize);
} while (bRet && pCurrentAddress <= pEndAdress);
return bRet;
}
#define IsBadWritePtr(p,n) (!CheckAccess<dwWriteRights>(p,n))
#define IsBadReadPtr(p,n) (!CheckAccess<dwReadRights>(p,n))
#define IsBadStringPtrW(p,n) (!CheckAccess<dwReadRights>(p,n*2))
This approach is based on my understanding of Raymond Chen's blog post, If I'm not supposed to call IsBadXxxPtr, how can I check if a pointer is bad?
This is an old question but this part:
the code where this check needs to be done is time sensitive so I need
the most efficient check possible that is also 100% effective
VirtualQuery() takes a kernel call, so the simple memcpy() in an exception handler will be faster for the case where the memory is okay to read most of the time.
__try
{
memcpy(dest, src, size);
}__except(1){}
All stays in user mode when there is no exception. Maybe a bit slower for the use case where the memory is bad to read more than it is good (since it fires off a exception which is a round trip through the kernel and back).
You could also extend it with a custom memcpy loop and *size so you could return exactly how many bytes were actually read.
if you are using VC++ then I suggest to use microsoft specific keywords __try __except
to and catch HW exceptions
Related
I have a c++ node add on that uses the Nan library. I have a function that needs to return a buffer. The simplest version of it is as follows (code edited as per comments):
NAN_METHOD(Test) {
char * retVal = (char*)malloc(100 * sizeof(char));
info.GetReturnValue().Set(Nan::NewBuffer(retVal, 100 *sizeof(char)).ToLocalChecked());
}
where the union is just used as an easy way to reinterpret the bytes. According to the documentation, Nan::NewBuffer assumes ownership of the memory so there is no need to free the memory manually. However when I run my node code that uses this function, my memory skyrockets, even when I force the garbage collector to run via global.gc(); The node code to produce the error is extremely simple:
const addon = require("addon");
for (let i = 0; i < 100000000; i++) {
if(i % (1000000) === 0){
console.log(i);
try {
global.gc();
} catch (e) {
console.log("error garbage collecting");
process.exit();
}
}
const buf = addon.Test();
}
Any help would be appreciated.
After much experimentation and research, I discovered this postenter link description here which basically states that the promise to free the memory that is passed into Nan::NewBuffer is just a lie. Using Nan::CopyBuffer instead of Nan::NewBuffer solves the problem at the cost of a memcpy. So essentially, the answer is that Nan::NewBuffer is broken and you shouldn't use it. Use Nan::CopyBuffer instead.
In IT, we tend to call a lie in the documentation a bug.
This question has been the subject of a Node.js issue - https://github.com/nodejs/node/issues/40936
The answer is that the deallocation of the underlying buffer happens in the C++ equivalent of a JS SetImmediate handler and if your code is completely synchronous - which is your case - the buffers won't be freed until the program end.
You correctly found that Nan::CopyBuffer does not suffer from this problem.
Alas, this cannot be easily fixed, as Node.js does not know how this buffer was allocated and cannot call its callback from the garbage collector context.
If you are using Nan::NewBuffer I also suggest to look at this issue: https://github.com/nodejs/node/issues/40926 in which another closely related pitfall is discussed.
I have a pointer (void *) to a function and I want to know which process this function belongs to. I have no idea which way to go about it, but I think it's possible by using some form of VirtualQuery trickery. Any help would be appreciated.
Thanks in advance,
CLARIFICATION: By "belong to process" I mean what process the function is in. For example:
say there was an executable (test.exe) loaded in memory. This executable contains a function named SayHello, which is located at 0xDEADBEEF in memory. In an entirely different process, how would I know 0xDEADBEEF is in test.exe's memory space.
Hope that clears things up.
CLARIFICATION 2: I'm sure you're familiar with "VTable hooking", where an external module changes a VTable pointer in a seperate process to point to a different function. Thereby whenever the hooked member is called, it is passed to the external module.
To prevent this (anti-cheat), I want to be able to check whether all methods of a VTable point to the module they reside in.
SOLUTION CODE:
template<class T>
inline void **GetVTableArray(T *pClass, int *pSize)
{
void **ppVTable = *(void ***)pClass;
if(pSize)
{
*pSize = 0;
while(!IsBadReadPtr(ppVTable[*pSize], sizeof(UINT_PTR)))
(*pSize)++;
}
return ppVTable;
}
bool AllVTableMembersPointToCurrentModule(void *pClass)
{
DWORD dwOldProtect;
HANDLE hModuleSnap = INVALID_HANDLE_VALUE;
MODULEENTRY32 moduleEntry;
// Take a snapshot of all modules in the specified process
hModuleSnap = CreateToolhelp32Snapshot(TH32CS_SNAPMODULE, GetCurrentProcessId());
if(hModuleSnap == INVALID_HANDLE_VALUE)
return false;
// Set the size of the structure before using it
moduleEntry.dwSize = sizeof(MODULEENTRY32);
// Retrieve information about the first module (current process)
if(!Module32First(hModuleSnap, &moduleEntry))
{
CloseHandle(hModuleSnap);
return false;
}
// Grab the base address and size of our module (the address range where
// the VTable can validly point to)
UINT_PTR ulBaseAddress = reinterpret_cast<UINT_PTR>(moduleEntry.modBaseAddr);
UINT_PTR ulBaseSize = moduleEntry.modBaseSize;
// Get the VTable array and VTable member count
int nMethods;
void **ppVTable = GetVTableArray(pClass, &nMethods);
#ifdef VTABLE_FAKING
// Allow patching
VirtualProtect(ppVTable, nMethods * sizeof(UINT_PTR), PAGE_EXECUTE_READWRITE, &dwOldProtect);
// Now take the next module and set the first VTable pointer to point to an
// invalid address, outside of the current module's address range
Module32Next(hModuleSnap, &moduleEntry);
ppVTable[0] = moduleEntry.modBaseAddr;
#endif
// Don't allow people to overwrite VTables (can easily be bypassed, so make
// sure you check the VirtualProtect status of the VTable regularly with
// VirtualQuery)
VirtualProtect(ppVTable, nMethods * sizeof(UINT_PTR), PAGE_EXECUTE, &dwOldProtect);
// Clean up the snapshot object
CloseHandle(hModuleSnap);
// Ensure all VTable pointers are in our current module's address range
for(int i = 0; i < nMethods; ++i)
{
// Get address of the method this VTable pointer points to
UINT_PTR ulFuncAddress = reinterpret_cast<UINT_PTR>(ppVTable[i]);
// Check the address is within our current module range
if(ulFuncAddress < ulBaseAddress || ulFuncAddress > ulBaseAddress + ulBaseSize)
return false;
}
return true;
}
Each process has its own address space. This means that the same address will contain different things for different processes, so there is no way to do what you're asking.
If this pointer is to a function in the current program (i.e. a function that you can currently call), then the answer is simple: it belongs to the current process.
To further clarify: A pointer by itself is meaningless unless you already know which process it belongs to. Process #1001 may have a function sayHello at address 0x12345678, while process #1002 has the function sayGoodbye at address 0x12345678, and process #1003 contains some data at the same address. There is no way to know which process the pointer came from.
The hijacked function pointer in the VTable can only be inside your process, as the other folks have already answered. The memory address only makes sense for your process. If someone is going to overwrite one of your VTable spots, then they would first have to hook something your process, which means running code inside your process. There exists plenty of win API that provides hooking.
See EnumProcessModule to go through all of the modules in your process. See this about modules info including base address of your module. You would then have to check your VTables to make sure those addressed exist inside of your module's address range.
To prevent VTable hijacking in the first place? I don't know how to do this, other than trying Microsoft's Detours library, which can in theory be used to detour any hook API call inside your process.
If you have the module handle, you can inspect the image header to ensure that the vtable pointers are in that module's virtual address space.
In any of the Windows operating systems that are descended from Windows NT (so, for all intents and purposes anything including and after XP, and before that NT 4 and NT 3.51) each process has it's own address space. Within reason, any pointer address can be different in every process in the system as they all have an 0xDEADBEEF address and it may, or may not contain the same thing as other processes. This was not the same with Windows 3.0, 3.1, 95, 98 and ME (they had one address space which all processes shared) where your question MAY have made more sense.
So, without a handle to a process to go with your pointer address the address is pretty much useless to you. With a handle to a process you can (possibly) work out what you want by walking the import tables for the DLLs that you import... If the function isn't an imported function then it's unlikely that you could work out what you want to know.
Note that if the address is to a function which is from a 'standard' system DLL then you MAY be able to work out where it lives by finding out what function it represents in the address space of your process as there is a strong chance that the DLL will be mapped to the same base address in your process as it is in every other process.
Why not tell us a little more about what it is, exactly, that you're actually trying to do?
Edit:
Well, as I described above, what you're suggesting is not possible except on very old versions of Windows. What IS possible is that you can inject code into a process to replace that code that should be executed. The address of this injected code is valid in the target process's address space and contains code that you (the hacking process) have created. You do this by a combination of allocating memory in the remote process with VirtualAllocEx() (1) and then writing your code to it with WriteProcessMemory() (2). You now have code that you wrote in the target process. You can then patch it in so that it's called instead of the code that should be called.
The common way to do this is IAT hooking (Import Address Table hooking) and this lets you replace imported functions from DLLs. To detect this you need to scan the DLL's Import Address Table from the DLL image on disk, work out where the functions are in memory and then scan the in memory IAT to check that the functions are where they should be; if they're not then they've likely been patched.
You're suggesting that someone is replacing an arbitrary C++ vTable entry. This is possible with the same technique but it's harder as there's no convenient table of names to addresses that you can use to work out where to patch. Anyway, assuming the bad guy can find the correct address to patch he can use the same technique as above to create his own function in your process.
Detecting the vTable problem is made more complex by the lack of name to address lookup, but if you're in the process that's being hacked you can simply have written code that takes the address of the function in question at start up. Store that somewhere and compare it later. However, you'd probably best take a copy of the whole function itself in memory and compare with that as you might find that the bad guys simply look for some recognisable function signature bytes and patch a jump into them somewhere to their own code or simply to skip yours.
Good luck and grab yourself a good book, such as one by Jeffrey Richter which will explain much of this far better than I can.
I don't really understand your question, so I'm going to take a stab in the dark and answer what I think you're asking.
You're asking how you can find out, from a function pointer, to which module it belongs.
The solution is rather simple in theory, scan backwards in memory to find the header, and then enjoy using this function GetModuleFileName.
Since your question isn't well worded, you don't get a well worded answer.
I have a C/C++ program that might be hanging when it runs out of memory. We discovered this by running many copies at the same time. I want to debug the program without completely destroying performance on the development machine. Is there a way to limit the memory available so that a new or malloc will return a NULL pointer after, say, 500K of memory has been requested?
Try turning the question on its head and asking how to limit the amount of memory an OS will allow your process to use.
Try looking into http://ss64.com/bash/ulimit.html
Try say:
ulimit -v
Here is another link that's a little old but gives a little more back ground:
http://www.network-theory.co.uk/docs/gccintro/gccintro_77.html
One way is to write a wrapper around malloc().
static unsigned int requested =0;
void* my_malloc(size_tamount){
if (requested + amount < LIMIT){
requested+=amount;
return malloc(amount);
}
return NULL
}
Your could use a #define to overload your malloc.
As GMan states, you could overload new / delete operators as well (for the C++ case).
Not sure if that's the best way, or what you are looking for
Which OS? For Unix, see ulimit -d/limit datasize depending on your shell (sh/csh).
You can write a wrapper for malloc which returns an error in the circonstance you want. Depending on your OS, you may be able to substitute it for the implementation's one.
An other way of doing it is to use failmalloc which is a shared library that overrides malloc etc. and then fail :-). It gives you control over when to fail and can be made to fail randomly, every nth time etc.
I havent used it my self but have heard good things.
That depends on your platform. For example, this can be achieved programmatically on Unix-like platforms using setrlimit(RLIMIT_DATA, ...).
EDIT:
The RLIMIT_AS resource may also be useful in this case as well.
Override new and new[].
void* operator new(size_t s)
{
}
void* operator new[](size_t s)
{
}
Put your own code in the braces to selectively die after X number of calls to new. Normally you would call malloc to allocate the memory and return it.
I once had a student in CS 1 (in C, yeah, yeah, not my fault) try this, and ran out of memory:
int array[42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42][42]..... (42 dimensions);
and then he wanted to know why it gave errors...
If you want to spend money, there's a tool called Holodeck by SecurityInnovations, which lets you inject faults into your program (including low memory). Nice thing is you can turn stuff on and off at will. I haven't really used it, much, so I don't know if it's possible to program in faults at certain points with the tool. I also don't know what platforms are supported...
As far as I know, on Linux, malloc will never return a null pointer. Instead, the OOM Killer will get called. This is, of course, unless you've disabled the OOM Killer. Some googling should come up with a result.
I know this isn't your actual question, but it does have to do with where you're coming from.
This question already has answers here:
Closed 13 years ago.
Possible Duplicates:
How can I get the size of an array from a pointer in C?
Is there any way to determine the size of a C++ array programmatically? And if not, why?
I get a pointer to a chunk of allocated memory out of a C style function.
Now, it would be really interesting for debugging purposes to know how
big the allocated memory block that this pointer points is.
Is there anything more elegant than provoking an exception by blindly running over its boundaries?
Thanks in advance,
Andreas
EDIT:
I use VC++2005 on Windows, and GCC 4.3 on Linux
EDIT2:
I have _msize under VC++2005
Unfortunately it results in an exception in debug mode....
EDIT3:
Well. I have tried the way I described above with the exception, and it works.
At least while I am debugging and ensuring that immediately after the call
to the library exits I run over the buffer boundaries. Works like a charm.
It just isn't elegant and in no way usable in production code.
It's not standard but if your library has a msize() function that will give you the size.
A common solution is to wrap malloc with your own function that logs each request along with the size and resulting memory range, in the release build you can switch back to the 'real' malloc.
If you don't mind sleazy violence for the sake of debugging, you can #define macros to hook calls to malloc and free and pad the first 4 bytes with the size.
To the tune of
void *malloc_hook(size_t size) {
size += sizeof (size_t);
void *ptr = malloc(size);
*(size_t *) ptr = size;
return ((size_t *) ptr) + 1;
}
void free_hook (void *ptr) {
ptr = (void *) (((size_t *) ptr) - 1);
free(ptr);
}
size_t report_size(ptr) {
return * (((size_t *) ptr) - 1);
}
then
#define malloc(x) malloc_hook(x)
and so on
The C runtime library does not provide such a function. Furthermore, deliberately provoking an exception will not tell you how big the block is either.
Usually the way this problem is solved in C is to maintain a separate variable which keeps track of the size of the allocated block. Of course, this is sometimes inconvenient but there's generally no other way to know.
Your C runtime library may provide some heap debug functions that can query allocated blocks (after all, free() needs to know how big the block is), but any of this sort of thing will be nonportable.
With gcc and the GNU linker, you can easily wrap malloc
#include <stdlib.h>
#include <stdio.h>
void* __real_malloc(size_t sz);
void* __wrap_malloc(size_t sz)
{
void *ptr;
ptr = __real_malloc(sz);
fprintf(stderr, "malloc of size %d yields pointer %p\n", sz, ptr);
/* if you wish to save the pointer and the size to a data structure,
then remember to add wrap code for calloc, realloc and free */
return ptr;
}
int main()
{
char *x;
x = malloc(103);
return 0;
}
and compile with
gcc a.c -o a -Wall -Werror -Wl,--wrap=malloc
(Of course, this will also work with c++ code compiled with g++, and with the new operator (through it's mangled name) if you wish.)
In effect, the statically/dynamically loaded library will also use your __wrap_malloc.
No, and you can't rely on an exception when overrunning its boundaries, unless it's in your implementation's documentation. It's part of the stuff you really don't need to know about to write programs. Dig into your compiler's documentation or source code if you really want to know.
There is no standard C function to do this. Depending on your platform, there may be a non-portable method - what OS and C library are you using?
Note that provoking an exception is unreliable - there may be other allocations immediately after the chunk you have, and so you might not get an exception until long after you exceed the limits of your current chunk.
Memory checkers like Valgrind's memcheck and Google's TCMalloc (the heap checker part) keep track of this sort of thing.
You can use TCMalloc to dump a heap profile that shows where things got allocated, or you can just have it check to make sure your heap is the same at two points in program execution using SameHeap().
Partial solution: on Windows you can use the PageHeap to catch a memory access outside the allocated block.
PageHeap is an alternate memory manager present in the Windows kernel (in the NT varieties but nobody should be using any other version nowadays). It takes every allocation in a process and returns a memory block that has its end aligned with the end of a memory page, then it makes the following page unaccessible (no read, no write access). If the program tries to read or write past the end of the block, you'll get an access violation you can catch with your favorite debugger.
How to get it: Download and install the package Debugging Tools for Windows from Microsoft: http://www.microsoft.com/whdc/devtools/debugging/default.mspx
then launch the GFlags utility, go to the 3rd tab and enter the name of your executable, then Hit the key. Check the PageHeap checkbox, click OK and you're good to go.
The last thing: when you're done with debugging, don't ever forget to launch GFlags again, and disable PageHeap for the application. GFlags enters this setting into the Registry (under HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\), so it is persistent, even across reboots.
Also, be aware that using PageHeap can increase the memory needs of your application tremendously.
The way to do what you want is to BE the allocator. If you filter all requests, and then record them for debugging purposes, then you can find out what you want when the memory is free'd.
Additionally, you can check at the end of the program to see if all allocated blocks were freed, and if not, list them. An ambitious library of this sort could even take FUNCTION and LINE parameters via a macro to let you know exactly where you are leaking memory.
Finally, Microsoft's MSVCRT provides a a debuggable heap that has many useful tools that you can use in your debug version to find memory problems: http://msdn.microsoft.com/en-us/library/bebs9zyz.aspx
On Linux, you can use valgrind to find many errors. http://valgrind.org/
I want to create an allocator which provides memory with the following attributes:
cannot be paged to disk.
is incredibly hard to access through an attached debugger
The idea is that this will contain sensitive information (like licence information) which should be inaccessible to the user. I have done the usual research online and asked a few other people about this, but I cannot find a good place start on this problem.
Updates
Josh mentions using VirtualAlloc to set protection on the memory space. I have created a custom allocator ( shown below ) I have found the using the VirtualLock function it limits the amount of memory I can allocate. This seems to be by design though. Since I am using it for small objects this is not a problem.
//
template<class _Ty>
class LockedVirtualMemAllocator : public std::allocator<_Ty>
{
public:
template<class _Other>
LockedVirtualMemAllocator<_Ty>& operator=(const LockedVirtualMemAllocator<_Other>&)
{ // assign from a related LockedVirtualMemAllocator (do nothing)
return (*this);
}
template<class Other>
struct rebind {
typedef LockedVirtualMemAllocator<Other> other;
};
pointer allocate( size_type _n )
{
SIZE_T allocLen = (_n * sizeof(_Ty));
DWORD allocType = MEM_COMMIT;
DWORD allocProtect = PAGE_READWRITE;
LPVOID pMem = ::VirtualAlloc( NULL, allocLen, allocType, allocProtect );
if ( pMem != NULL ) {
::VirtualLock( pMem, allocLen );
}
return reinterpret_cast<pointer>( pMem );
}
pointer allocate( size_type _n, const void* )
{
return allocate( _n );
}
void deallocate(void* _pPtr, size_type _n )
{
if ( _pPtr != NULL ) {
SIZE_T allocLen = (_n * sizeof(_Ty));
::SecureZeroMemory( _pPtr, allocLen );
::VirtualUnlock( _pPtr, allocLen );
::VirtualFree( _pPtr, 0, MEM_RELEASE );
}
}
};
and is used
//a memory safe std::string
typedef std::basic_string<char, std::char_traits<char>,
LockedVirtualMemAllocato<char> > modulestring_t;
Ted Percival mentions mlock, but I have no implementation of that yet.
I found Practical Cryptography by Neil Furguson and Bruce Schneier quite helpful as well.
You can't really protect against memory access. You can probably prevent paging if you are running as an admin or as the system, but you cannot prevent the admin or system from reading your memory. Even if you could somehow completely block other processes from reading your memory (which you can't), another process could still actually inject a new thread into your process and read the memory that way.
Even if you could somehow completely lock down your process and guarantee that the OS would never allow anyone else to access your process, you still don't have full protection. The entire OS could be running in a virtual machine, which could be paused and inspected at any time.
You cannot protect memory contents from the owner of the system. Hollywood and the music industry have been aching for this for years. If it were possible, they'd already be doing it.
On Unix systems you can use mlock(2) to lock memory pages into RAM, preventing them being paged.
mlock() and mlockall() respectively lock part or all of the calling
process’s virtual address space into RAM, preventing that memory from
being paged to the swap area.
There is a limit to how much memory each process can lock, it can be shown with ulimit -l and is measured in kilobytes. On my system, the default limit is 32 kiB per process.
If you're developing for Windows, there are ways you can restrict access to memory, but absolutely blocking out others is not doable. If you're hoping to keep a secret secret, read Writing Secure Code - which addresses this problem at some length, but be aware that you have no way of knowing if your code is running on a real machine or a virtual machine. There's a bunch of Win32 API stuff to deal with crypto that handles this kind of thing, including safe storage of secrets - the book talks about that. You can look at the online Microsoft CyproAPI for details; the OS designers recognise this very problem and the need to keep the cleartext secure (again, read Writing Secure Code).
The Win32 API function VirtualAlloc is the OS level memory allocator. It allows you to set access protection; what you could do is set access to PAGE_GUARD or PAGE_NOACCESS, and flip the access to something friendlier while your program reads, and reset it afterward, but that's merely a speed hump if someone is trying really hard to peek at your secret.
In summary, look at the crypto APIs on your platform, they'll address the problem better than something you hack up yourself.
Let's take this a bit at a time:
I want to create an allocator which
provides memory with the following
attributes:
That's fair enough.
* cannot be paged to disk.
That's going to be hard. As far as I am aware, you cannot disable Virtual Paging as it is handled by the OS. If there is a way, then you'll be spelunking in the bowels of the OS.
* is incredibly hard to access through an attached debugger
You could run it through PGP and store it encrypted in memory and unencrypt it as needed. Massive performance hit.
The idea is that this will contain
sensitive information (like licence
information) which should be
inaccessible to the user. I have done
the usual research online and asked a
few other people about this, but I
cannot find a good place start on this
problem.
Keep all sensitive information off the machine. Seriously. Don't store sensitive information in memory. Write a custom delete routine that will automatically remove all data from any allocations you perform. Never allow general access to a machine with sensitive material on it. If you perform db access, make sure all access is sanitized before firing. Only people with specific log-ins are allowed to access. No general group access.
On a side note, what other methods are
there of accessing the memory of a
process other than attaching a
debugger?
Taking a dump of the memory.
install Libsodium, use allocation mechanisms by #including <sodium.h>
Guarded heap allocations
Slower than malloc() and friends, they require 3 or 4 extra pages of virtual memory.
void *sodium_malloc(size_t size);
Allocate memory to store sensitive data using sodium_malloc() and sodium_allocarray(). You'll need to first call sodium_init() before using these heap guards.
void *sodium_allocarray(size_t count, size_t size);
The sodium_allocarray() function returns a pointer from which count objects that are size bytes of memory each can be accessed. It provides the same guarantees as sodium_malloc() but also protects against arithmetic overflows when count * size exceeds SIZE_MAX.
These functions add guard pages around the protected data to make it less likely to be accessible in a heartbleed-like scenario.
In addition, the protection for memory regions allocated that way can be changed using the locking memory operations: sodium_mprotect_noaccess(), sodium_mprotect_readonly() and sodium_mprotect_readwrite().
After sodium_malloc you can use sodium_free() to unlock and deallocate memory. At this point in your implementation consider zeroing the memory after use.
zero the memory after use
void sodium_memzero(void * const pnt, const size_t len);
After use, sensitive data should be overwritten, but memset() and hand-written code can be silently stripped out by an optimizing compiler or by the linker.
The sodium_memzero() function tries to effectively zero len bytes starting at pnt, even if optimizations are being applied to the code.
locking the memory allocation
int sodium_mlock(void * const addr, const size_t len);
The sodium_mlock() function locks at least len bytes of memory starting at addr. This can help avoid swapping sensitive data to disk.
int sodium_mprotect_noaccess(void *ptr);
The sodium_mprotect_noaccess() function makes a region allocated using sodium_malloc() or sodium_allocarray() inaccessible. It cannot be read or written, but the data are preserved. This function can be used to make confidential data inaccessible except when actually needed for a specific operation.
int sodium_mprotect_readonly(void *ptr);
The sodium_mprotect_readonly() function marks a region allocated using sodium_malloc() or sodium_allocarray() as read-only. Attempting to modify the data will cause the process to terminate.
int sodium_mprotect_readwrite(void *ptr);
The sodium_mprotect_readwrite() function marks a region allocated using sodium_malloc() or sodium_allocarray() as readable and writable, after having been protected using sodium_mprotect_readonly() or sodium_mprotect_noaccess().
What you are asking for is handled at the OS level. Once the data is in your program, it is liable to be paged out.
For accessing the memory, a motivated individual can attach a hardware debugger.
#graham
You could run it through PGP and store it encrypted in memory and unencrypt it as needed. Massive performance hit.
Then you'd have to hold the key in memory. That would make it a little harder, but definitely not impossible. Anyone motivated will still manage to get the data from memory.
Your best bet is to implement something similar to .NET's SecureString class, and be very careful to zero out any plaintext copies of your data as soon as you are done (don't forget to cleanup even when exceptions are thrown). A good way to do this with std::string and such is to use a custom allocator.
On Windows, if you use CryptProtectMemory (or RtlEncryptMemory for older systems), the encryption password is stored in non-pageable (kernel?) memory. In my testing, these functions are pretty darn fast, esp. taking into account the protection they are giving you.
On other systems, I like to use Blowfish since it's a good mix between speed and strength. In the latter case, you will have to randomly generate your own password (16+ bytes of entropy for Blowfish) at program startup. Unfortunately, there's not a whole lot you can do to protect that password without OS support, although you might use general obfuscation techniques to embed a hard-coded salt value into your executable that you can combine with the password (every little bit helps).
Overall, this strategy is only one part of a broader defense-in-depth approach. Also keep in mind that simple bugs such as buffer overflows and not sanitizing program input remain by far the most common attack vectors.
You cannot protect memory contents from the owner of the system.
Hollywood and the music industry have been aching for this for years.
If it were possible, they'd already be doing it.
Have you had a look at Vista (and above) Protected Processes (direct .doc download). I believe the operating system-enforced protection is courtesy of the entertainment industry.
#Derek: Oh, but with trusted computing, you can use memory curtaining! :-P</devils-advocate>
#roo
I was really hoping that is was possible, and that I just hadn't found it yet. Your example just made me realise that that is exactly what we are trying to do - only allow access to files in the context of our program and so preserve the IP.
I guess I have to accept that there is no truly secure way to store someone’s files on another computer, especially if at some point access is allowed to that file by the owner.
That's definitely the problem. You can store something securely so long as you never grant access, but as soon as you grant access, your control is gone. You can make it a little bit more difficult, but that's all.
#Chris
Oh, but with trusted computing, you can use memory curtaining! :-P
But then you have to actually be willing to pay for a computer someone else owns. :p
#Derek Park
He only said harder, not impossible. PGP would make it harder, not impossible.