Is there any way to determine (programatically, of course) if a given pointer is "valid"? Checking for NULL is easy, but what about things like 0x00001234? When trying to dereference this kind of pointer an exception/crash occurs.
A cross-platform method is preferred, but platform-specific (for Windows and Linux) is also ok.
Update for clarification:
The problem is not with stale/freed/uninitialized pointers; instead, I'm implementing an API that takes pointers from the caller (like a pointer to a string, a file handle, etc.). The caller can send (in purpose or by mistake) an invalid value as the pointer. How do I prevent a crash?
Update for clarification: The problem is not with stale, freed or uninitialized pointers; instead, I'm implementing an API that takes pointers from the caller (like a pointer to a string, a file handle, etc.). The caller can send (in purpose or by mistake) an invalid value as the pointer. How do I prevent a crash?
You can't make that check. There is simply no way you can check whether a pointer is "valid". You have to trust that when people use a function that takes a pointer, those people know what they are doing. If they pass you 0x4211 as a pointer value, then you have to trust it points to address 0x4211. And if they "accidentally" hit an object, then even if you would use some scary operation system function (IsValidPtr or whatever), you would still slip into a bug and not fail fast.
Start using null pointers for signaling this kind of thing and tell the user of your library that they should not use pointers if they tend to accidentally pass invalid pointers, seriously :)
Here are three easy ways for a C program under Linux to get introspective about the status of the memory in which it is running, and why the question has appropriate sophisticated answers in some contexts.
After calling getpagesize() and rounding the pointer to a page
boundary, you can call mincore() to find out if a page is valid and
if it happens to be part of the process working set. Note that this requires
some kernel resources, so you should benchmark it and determine if
calling this function is really appropriate in your api. If your api
is going to be handling interrupts, or reading from serial ports
into memory, it is appropriate to call this to avoid unpredictable
behaviors.
After calling stat() to determine if there is a /proc/self directory available, you can fopen and read through /proc/self/maps
to find information about the region in which a pointer resides.
Study the man page for proc, the process information pseudo-file
system. Obviously this is relatively expensive, but you might be
able to get away with caching the result of the parse into an array
you can efficiently lookup using a binary search. Also consider the
/proc/self/smaps. If your api is for high-performance computing then
the program will want to know about the /proc/self/numa which is
documented under the man page for numa, the non-uniform memory
architecture.
The get_mempolicy(MPOL_F_ADDR) call is appropriate for high performance computing api work where there are multiple threads of
execution and you are managing your work to have affinity for non-uniform memory
as it relates to the cpu cores and socket resources. Such an api
will of course also tell you if a pointer is valid.
Under Microsoft Windows there is the function QueryWorkingSetEx that is documented under the Process Status API (also in the NUMA API).
As a corollary to sophisticated NUMA API programming this function will also let you do simple "testing pointers for validity (C/C++)" work, as such it is unlikely to be deprecated for at least 15 years.
Preventing a crash caused by the caller sending in an invalid pointer is a good way to make silent bugs that are hard to find.
Isn't it better for the programmer using your API to get a clear message that his code is bogus by crashing it rather than hiding it?
On Win32/64 there is a way to do this. Attempt to read the pointer and catch the resulting SEH exeception that will be thrown on failure. If it doesn't throw, then it's a valid pointer.
The problem with this method though is that it just returns whether or not you can read data from the pointer. It makes no guarantee about type safety or any number of other invariants. In general this method is good for little else other than to say "yes, I can read that particular place in memory at a time that has now passed".
In short, Don't do this ;)
Raymond Chen has a blog post on this subject: http://blogs.msdn.com/oldnewthing/archive/2007/06/25/3507294.aspx
AFAIK there is no way. You should try to avoid this situation by always setting pointers to NULL after freeing memory.
On Unix you should be able to utilize a kernel syscall that does pointer checking and returns EFAULT, such as:
#include <unistd.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdbool.h>
bool isPointerBad( void * p )
{
int fh = open( p, 0, 0 );
int e = errno;
if ( -1 == fh && e == EFAULT )
{
printf( "bad pointer: %p\n", p );
return true;
}
else if ( fh != -1 )
{
close( fh );
}
printf( "good pointer: %p\n", p );
return false;
}
int main()
{
int good = 4;
isPointerBad( (void *)3 );
isPointerBad( &good );
isPointerBad( "/tmp/blah" );
return 0;
}
returning:
bad pointer: 0x3
good pointer: 0x7fff375fd49c
good pointer: 0x400793
There's probably a better syscall to use than open() [perhaps access], since there's a chance that this could lead to actual file creation codepath, and a subsequent close requirement.
Regarding the answer a bit up in this thread:
IsBadReadPtr(), IsBadWritePtr(), IsBadCodePtr(), IsBadStringPtr() for Windows.
My advice is to stay away from them, someone has already posted this one:
http://blogs.msdn.com/oldnewthing/archive/2007/06/25/3507294.aspx
Another post on the same topic and by the same author (I think) is this one:
http://blogs.msdn.com/oldnewthing/archive/2006/09/27/773741.aspx ("IsBadXxxPtr should really be called CrashProgramRandomly").
If the users of your API sends in bad data, let it crash. If the problem is that the data passed isn't used until later (and that makes it harder to find the cause), add a debug mode where the strings etc. are logged at entry. If they are bad it will be obvious (and probably crash). If it is happening way to often, it might be worth moving your API out of process and let them crash the API process instead of the main process.
Firstly, I don't see any point in trying to protect yourself from the caller deliberately trying to cause a crash. They could easily do this by trying to access through an invalid pointer themselves. There are many other ways - they could just overwrite your memory or the stack. If you need to protect against this sort of thing then you need to be running in a separate process using sockets or some other IPC for communication.
We write quite a lot of software that allows partners/customers/users to extend functionality. Inevitably any bug gets reported to us first so it is useful to be able to easily show that the problem is in the plug-in code. Additionally there are security concerns and some users are more trusted than others.
We use a number of different methods depending on performance/throughput requirements and trustworthyness. From most preferred:
separate processes using sockets (often passing data as text).
separate processes using shared memory (if large amounts of data to pass).
same process separate threads via message queue (if frequent short messages).
same process separate threads all passed data allocated from a memory pool.
same process via direct procedure call - all passed data allocated from a memory pool.
We try never to resort to what you are trying to do when dealing with third party software - especially when we are given the plug-ins/library as binary rather than source code.
Use of a memory pool is quite easy in most circumstances and needn't be inefficient. If YOU allocate the data in the first place then it is trivial to check the pointers against the values you allocated. You could also store the length allocated and add "magic" values before and after the data to check for valid data type and data overruns.
I've got a lot of sympathy with your question, as I'm in an almost identical position myself. I appreciate what a lot of the replies are saying, and they are correct - the routine supplying the pointer should be providing a valid pointer. In my case, it is almost inconceivable that they could have corrupted the pointer - but if they had managed, it would be MY software that crashes, and ME that would get the blame :-(
My requirement isn't that I continue after a segmentation fault - that would be dangerous - I just want to report what happened to the customer before terminating so that they can fix their code rather than blaming me!
This is how I've found to do it (on Windows): http://www.cplusplus.com/reference/clibrary/csignal/signal/
To give a synopsis:
#include <signal.h>
using namespace std;
void terminate(int param)
/// Function executed if a segmentation fault is encountered during the cast to an instance.
{
cerr << "\nThe function received a corrupted reference - please check the user-supplied dll.\n";
cerr << "Terminating program...\n";
exit(1);
}
...
void MyFunction()
{
void (*previous_sigsegv_function)(int);
previous_sigsegv_function = signal(SIGSEGV, terminate);
<-- insert risky stuff here -->
signal(SIGSEGV, previous_sigsegv_function);
}
Now this appears to behave as I would hope (it prints the error message, then terminates the program) - but if someone can spot a flaw, please let me know!
There are no provisions in C++ to test for the validity of a pointer as a general case. One can obviously assume that NULL (0x00000000) is bad, and various compilers and libraries like to use "special values" here and there to make debugging easier (For example, if I ever see a pointer show up as 0xCECECECE in visual studio I know I did something wrong) but the truth is that since a pointer is just an index into memory it's near impossible to tell just by looking at the pointer if it's the "right" index.
There are various tricks that you can do with dynamic_cast and RTTI such to ensure that the object pointed to is of the type that you want, but they all require that you are pointing to something valid in the first place.
If you want to ensure that you program can detect "invalid" pointers then my advice is this: Set every pointer you declare either to NULL or a valid address immediately upon creation and set it to NULL immediately after freeing the memory that it points to. If you are diligent about this practice, then checking for NULL is all you ever need.
Setting the pointer to NULL before and after using is a good technique. This is easy to do in C++ if you manage pointers within a class for example (a string):
class SomeClass
{
public:
SomeClass();
~SomeClass();
void SetText( const char *text);
char *GetText() const { return MyText; }
void Clear();
private:
char * MyText;
};
SomeClass::SomeClass()
{
MyText = NULL;
}
SomeClass::~SomeClass()
{
Clear();
}
void SomeClass::Clear()
{
if (MyText)
free( MyText);
MyText = NULL;
}
void SomeClass::Settext( const char *text)
{
Clear();
MyText = malloc( strlen(text));
if (MyText)
strcpy( MyText, text);
}
Indeed, something could be done under specific occasion: for example if you want to check whether a string pointer string is valid, using write(fd, buf, szie) syscall can help you do the magic: let fd be a file descriptor of temporary file you create for test, and buf pointing to the string you are tesing, if the pointer is invalid write() would return -1 and errno set to EFAULT which indicating that buf is outside your accessible address space.
Peeter Joos answer is pretty good. Here is an "official" way to do it:
#include <sys/mman.h>
#include <stdbool.h>
#include <unistd.h>
bool is_pointer_valid(void *p) {
/* get the page size */
size_t page_size = sysconf(_SC_PAGESIZE);
/* find the address of the page that contains p */
void *base = (void *)((((size_t)p) / page_size) * page_size);
/* call msync, if it returns non-zero, return false */
int ret = msync(base, page_size, MS_ASYNC) != -1;
return ret ? ret : errno != ENOMEM;
}
There isn't any portable way of doing this, and doing it for specific platforms can be anywhere between hard and impossible. In any case, you should never write code that depends on such a check - don't let the pointers take on invalid values in the first place.
As others have said, you can't reliably detect an invalid pointer. Consider some of the forms an invalid pointer might take:
You could have a null pointer. That's one you could easily check for and do something about.
You could have a pointer to somewhere outside of valid memory. What constitutes valid memory varies depending on how the run-time environment of your system sets up the address space. On Unix systems, it is usually a virtual address space starting at 0 and going to some large number of megabytes. On embedded systems, it could be quite small. It might not start at 0, in any case. If your app happens to be running in supervisor mode or the equivalent, then your pointer might reference a real address, which may or may not be backed up with real memory.
You could have a pointer to somewhere inside your valid memory, even inside your data segment, bss, stack or heap, but not pointing at a valid object. A variant of this is a pointer that used to point to a valid object, before something bad happened to the object. Bad things in this context include deallocation, memory corruption, or pointer corruption.
You could have a flat-out illegal pointer, such as a pointer with illegal alignment for the thing being referenced.
The problem gets even worse when you consider segment/offset based architectures and other odd pointer implementations. This sort of thing is normally hidden from the developer by good compilers and judicious use of types, but if you want to pierce the veil and try to outsmart the operating system and compiler developers, well, you can, but there is not one generic way to do it that will handle all of the issues you might run into.
The best thing you can do is allow the crash and put out some good diagnostic information.
In general, it's impossible to do. Here's one particularly nasty case:
struct Point2d {
int x;
int y;
};
struct Point3d {
int x;
int y;
int z;
};
void dump(Point3 *p)
{
printf("[%d %d %d]\n", p->x, p->y, p->z);
}
Point2d points[2] = { {0, 1}, {2, 3} };
Point3d *p3 = reinterpret_cast<Point3d *>(&points[0]);
dump(p3);
On many platforms, this will print out:
[0 1 2]
You're forcing the runtime system to incorrectly interpret bits of memory, but in this case it's not going to crash, because the bits all make sense. This is part of the design of the language (look at C-style polymorphism with struct inaddr, inaddr_in, inaddr_in6), so you can't reliably protect against it on any platform.
It's unbelievable how much misleading information you can read in articles above...
And even in microsoft msdn documentation IsBadPtr is claimed to be banned. Oh well - I prefer working application rather than crashing. Even if term working might be working incorrectly (as long as end-user can continue with application).
By googling I haven't found any useful example for windows - found a solution for 32-bit apps,
http://www.codeproject.com/script/Content/ViewAssociatedFile.aspx?rzp=%2FKB%2Fsystem%2Fdetect-driver%2F%2FDetectDriverSrc.zip&zep=DetectDriverSrc%2FDetectDriver%2Fsrc%2FdrvCppLib%2Frtti.cpp&obid=58895&obtid=2&ovid=2
but I need also to support 64-bit apps, so this solution did not work for me.
But I've harvested wine's source codes, and managed to cook similar kind of code which would work for 64-bit apps as well - attaching code here:
#include <typeinfo.h>
typedef void (*v_table_ptr)();
typedef struct _cpp_object
{
v_table_ptr* vtable;
} cpp_object;
#ifndef _WIN64
typedef struct _rtti_object_locator
{
unsigned int signature;
int base_class_offset;
unsigned int flags;
const type_info *type_descriptor;
//const rtti_object_hierarchy *type_hierarchy;
} rtti_object_locator;
#else
typedef struct
{
unsigned int signature;
int base_class_offset;
unsigned int flags;
unsigned int type_descriptor;
unsigned int type_hierarchy;
unsigned int object_locator;
} rtti_object_locator;
#endif
/* Get type info from an object (internal) */
static const rtti_object_locator* RTTI_GetObjectLocator(void* inptr)
{
cpp_object* cppobj = (cpp_object*) inptr;
const rtti_object_locator* obj_locator = 0;
if (!IsBadReadPtr(cppobj, sizeof(void*)) &&
!IsBadReadPtr(cppobj->vtable - 1, sizeof(void*)) &&
!IsBadReadPtr((void*)cppobj->vtable[-1], sizeof(rtti_object_locator)))
{
obj_locator = (rtti_object_locator*) cppobj->vtable[-1];
}
return obj_locator;
}
And following code can detect whether pointer is valid or not, you need probably to add some NULL checking:
CTest* t = new CTest();
//t = (CTest*) 0;
//t = (CTest*) 0x12345678;
const rtti_object_locator* ptr = RTTI_GetObjectLocator(t);
#ifdef _WIN64
char *base = ptr->signature == 0 ? (char*)RtlPcToFileHeader((void*)ptr, (void**)&base) : (char*)ptr - ptr->object_locator;
const type_info *td = (const type_info*)(base + ptr->type_descriptor);
#else
const type_info *td = ptr->type_descriptor;
#endif
const char* n =td->name();
This gets class name from pointer - I think it should be enough for your needs.
One thing which I'm still afraid is performance of pointer checking - in code snipet above there is already 3-4 API calls being made - might be overkill for time critical applications.
It would be good if someone could measure overhead of pointer checking compared for example to C#/managed c++ calls.
It is not a very good policy to accept arbitrary pointers as input parameters in a public API. It's better to have "plain data" types like an integer, a string or a struct (I mean a classical struct with plain data inside, of course; officially anything can be a struct).
Why? Well because as others say there is no standard way to know whether you've been given a valid pointer or one that points to junk.
But sometimes you don't have the choice - your API must accept a pointer.
In these cases, it is the duty of the caller to pass a good pointer. NULL may be accepted as a value, but not a pointer to junk.
Can you double-check in any way? Well, what I did in a case like that was to define an invariant for the type the pointer points to, and call it when you get it (in debug mode). At least if the invariant fails (or crashes) you know that you were passed a bad value.
// API that does not allow NULL
void PublicApiFunction1(Person* in_person)
{
assert(in_person != NULL);
assert(in_person->Invariant());
// Actual code...
}
// API that allows NULL
void PublicApiFunction2(Person* in_person)
{
assert(in_person == NULL || in_person->Invariant());
// Actual code (must keep in mind that in_person may be NULL)
}
Following does work in Windows (somebody suggested it before):
static void copy(void * target, const void* source, int size)
{
__try
{
CopyMemory(target, source, size);
}
__except(EXCEPTION_EXECUTE_HANDLER)
{
doSomething(--whatever--);
}
}
The function has to be static, standalone or static method of some class.
To test on read-only, copy data in the local buffer.
To test on write without modifying contents, write them over.
You can test first/last addresses only.
If pointer is invalid, control will be passed to 'doSomething',
and then outside the brackets.
Just do not use anything requiring destructors, like CString.
On Windows I use this code:
void * G_pPointer = NULL;
const char * G_szPointerName = NULL;
void CheckPointerIternal()
{
char cTest = *((char *)G_pPointer);
}
bool CheckPointerIternalExt()
{
bool bRet = false;
__try
{
CheckPointerIternal();
bRet = true;
}
__except (EXCEPTION_EXECUTE_HANDLER)
{
}
return bRet;
}
void CheckPointer(void * A_pPointer, const char * A_szPointerName)
{
G_pPointer = A_pPointer;
G_szPointerName = A_szPointerName;
if (!CheckPointerIternalExt())
throw std::runtime_error("Invalid pointer " + std::string(G_szPointerName) + "!");
}
Usage:
unsigned long * pTest = (unsigned long *) 0x12345;
CheckPointer(pTest, "pTest"); //throws exception
On macOS, you can do this with mach_vm_region, which as well as telling you if a pointer is valid, also lets you validate what access you have to the memory to which the pointer points (read/write/execute). I provided sample code to do this in my answer to another question:
#include <mach/mach.h>
#include <mach/mach_vm.h>
#include <stdio.h>
#include <stdbool.h>
bool ptr_is_valid(void *ptr, vm_prot_t needs_access) {
vm_map_t task = mach_task_self();
mach_vm_address_t address = (mach_vm_address_t)ptr;
mach_vm_size_t size = 0;
vm_region_basic_info_data_64_t info;
mach_msg_type_number_t count = VM_REGION_BASIC_INFO_COUNT_64;
mach_port_t object_name;
kern_return_t ret = mach_vm_region(task, &address, &size, VM_REGION_BASIC_INFO_64, (vm_region_info_t)&info, &count, &object_name);
if (ret != KERN_SUCCESS) return false;
return ((mach_vm_address_t)ptr) >= address && ((info.protection & needs_access) == needs_access);
}
#define TEST(ptr,acc) printf("ptr_is_valid(%p,access=%d)=%d\n", (void*)(ptr), (acc), ptr_is_valid((void*)(ptr),(acc)))
int main(int argc, char**argv) {
TEST(0,0);
TEST(0,VM_PROT_READ);
TEST(123456789,VM_PROT_READ);
TEST(main,0);
TEST(main,VM_PROT_READ);
TEST(main,VM_PROT_READ|VM_PROT_EXECUTE);
TEST(main,VM_PROT_EXECUTE);
TEST(main,VM_PROT_WRITE);
TEST((void*)(-1),0);
return 0;
}
The SEI CERT C Coding Standard recommendation MEM10-C. Define and use a pointer validation function says it is possible to do a check to some degree, especially under Linux OS.
The method described in the link is to keep track of the highest memory address returned by malloc and add a function that tests if someone tries to use a pointer greater than that value. It is probably of limited use.
IsBadReadPtr(), IsBadWritePtr(), IsBadCodePtr(), IsBadStringPtr() for Windows.
These take time proportional to the length of the block, so for sanity check I just check the starting address.
I have seen various libraries use some method to check for unreferenced memory and such. I believe they simply "override" the memory allocation and deallocation methods (malloc/free), which has some logic that keeps track of the pointers. I suppose this is overkill for your use case, but it would be one way to do it.
Technically you can override operator new (and delete) and collect information about all allocated memory, so you can have a method to check if heap memory is valid.
but:
you still need a way to check if pointer is allocated on stack ()
you will need to define what is 'valid' pointer:
a) memory on that address is
allocated
b) memory at that address
is start address of object (e.g.
address not in the middle of huge
array)
c) memory at that address
is start address of object of expected type
Bottom line: approach in question is not C++ way, you need to define some rules which ensure that function receives valid pointers.
There is no way to make that check in C++. What should you do if other code passes you an invalid pointer? You should crash. Why? Check out this link: http://blogs.msdn.com/oldnewthing/archive/2006/09/27/773741.aspx
Addendum to the accpeted answer(s):
Assume that your pointer could hold only three values -- 0, 1 and -1 where 1 signifies a valid pointer, -1 an invalid one and 0 another invalid one. What is the probability that your pointer is NULL, all values being equally likely? 1/3. Now, take the valid case out, so for every invalid case, you have a 50:50 ratio to catch all errors. Looks good right? Scale this for a 4-byte pointer. There are 2^32 or 4294967294 possible values. Of these, only ONE value is correct, one is NULL, and you are still left with 4294967292 other invalid cases. Recalculate: you have a test for 1 out of (4294967292+ 1) invalid cases. A probability of 2.xe-10 or 0 for most practical purposes. Such is the futility of the NULL check.
You know, a new driver (at least on Linux) that is capable of this probably wouldn't be that hard to write.
On the other hand, it would be folly to build your programs like this. Unless you have some really specific and single use for such a thing, I wouldn't recommend it. If you built a large application loaded with constant pointer validity checks it would likely be horrendously slow.
you should avoid these methods because they do not work. blogs.msdn.com/oldnewthing/archive/2006/09/27/773741.aspx – JaredPar Feb 15 '09 at 16:02
If they don't work - next windows update will fix it ?
If they don't work on concept level - function will be probably removed from windows api completely.
MSDN documentation claim that they are banned, and reason for this is probably flaw of further design of application (e.g. generally you should not eat invalid pointers silently - if you're in charge of design of whole application of course), and performance/time of pointer checking.
But you should not claim that they does not work because of some blog.
In my test application I've verified that they do work.
these links may be helpful
_CrtIsValidPointer
Verifies that a specified memory range is valid for reading and writing (debug version only).
http://msdn.microsoft.com/en-us/library/0w1ekd5e.aspx
_CrtCheckMemory
Confirms the integrity of the memory blocks allocated in the debug heap (debug version only).
http://msdn.microsoft.com/en-us/library/e73x0s4b.aspx
I know there are already many similar questions and answers exist, but I am not able to solve my problem.
In my big application heap is getting corrupted somewhere and I am not able to locate it. I used tool like gflags also but no luck.
I tried gflags on the following sample which corrupts the heap by purpose:
char* pBuffer = new char[256];
memset(pBuffer, 0, 256 + 1);
delete[] pBuffer;
At line#2 heap is overwritten but how to find it via tools like gflags, windbg etc. May be I am not using the gflags properly.
If automated tools (like electric fence or valgrind) don't do the trick, and staring intently at your code to try and figure out where it might have gone wrong doesn't help, and disabling/enabling various operations (until you get a correlation between the presence of heap-corruption and what operations did or didn't execute beforehand) to narrow it doesn't seem to work, you can always try this technique, which attempts to find the corruption sooner rather than later, so as to make it easier to track down the source:
Create your own custom new and delete operators that put corruption-evident guard areas around the allocated memory regions, something like this:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <new>
// make this however big you feel is "big enough" so that corrupted bytes will be seen in the guard bands
static int GUARD_BAND_SIZE_BYTES = 64;
static void * MyCustomAlloc(size_t userNumBytes)
{
// We'll allocate space for a guard-band, then space to store the user's allocation-size-value,
// then space for the user's actual data bytes, then finally space for a second guard-band at the end.
char * buf = (char *) malloc(GUARD_BAND_SIZE_BYTES+sizeof(userNumBytes)+userNumBytes+GUARD_BAND_SIZE_BYTES);
if (buf)
{
char * w = buf;
memset(w, 'B', GUARD_BAND_SIZE_BYTES); w += GUARD_BAND_SIZE_BYTES;
memcpy(w, &userNumBytes, sizeof(userNumBytes)); w += sizeof(userNumBytes);
char * userRetVal = w; w += userNumBytes;
memset(w, 'E', GUARD_BAND_SIZE_BYTES); w += GUARD_BAND_SIZE_BYTES;
return userRetVal;
}
else throw std::bad_alloc();
}
static void MyCustomDelete(void * p)
{
if (p == NULL) return; // since delete NULL is a safe no-op
// Convert the user's pointer back to a pointer to the top of our header bytes
char * internalCP = ((char *) p)-(GUARD_BAND_SIZE_BYTES+sizeof(size_t));
char * cp = internalCP;
for (int i=0; i<GUARD_BAND_SIZE_BYTES; i++)
{
if (*cp++ != 'B')
{
printf("CORRUPTION DETECTED at BEGIN GUARD BAND POSITION %i of allocation %p\n", i, p);
abort();
}
}
// At this point, (cp) should be pointing to the stored (userNumBytes) field
size_t userNumBytes = *((const size_t *)cp);
cp += sizeof(userNumBytes); // skip past the user's data
cp += userNumBytes;
// At this point, (cp) should be pointing to the second guard band
for (int i=0; i<GUARD_BAND_SIZE_BYTES; i++)
{
if (*cp++ != 'E')
{
printf("CORRUPTION DETECTED at END GUARD BAND POSITION %i of allocation %p\n", i, p);
abort();
}
}
// If we got here, no corruption was detected, so free the memory and carry on
free(internalCP);
}
// override the global C++ new/delete operators to call our
// instrumented functions rather than their normal behavior
void * operator new(size_t s) throw(std::bad_alloc) {return MyCustomAlloc(s);}
void * operator new[](size_t s) throw(std::bad_alloc) {return MyCustomAlloc(s);}
void operator delete(void * p) throw() {MyCustomDelete(p);}
void operator delete[](void * p) throw() {MyCustomDelete(p);}
... the above will be enough to get you Electric-Fence style functionality, in that if anything writes into either of the two 64-byte "guard bands" at the beginning or end of any new/delete memory-allocation, then when the allocation is deleted, MyCustomDelete() will notice the corruption and crash the program.
If that's not good enough (e.g. because by the time the deletion occurs, so much has happened since the corruption that it's difficult to tell what caused the corruption), you can go even further by having MyCustomAlloc() add the allocated buffer into a singleton/global doubly-linked list of allocations, and have MyCustomDelete() remove it from that same list (make sure to serialize these operations if your program is multithreaded!). The advantage of doing that is that you can then add another function called e.g. CheckForHeapCorruption() that will iterate over that linked list and check the guard-bands of every allocation in the linked list, and report if any of them have been corrupted. Then you can sprinkle calls to CheckForHeapCorruption() throughout your code, so that when heap corruption occurs it will be detected at the next call to CheckForHeapCorruption() rather than some time later on. Eventually you will find that one call to CheckForHeapCorruption() passed with flying colors, and then the next call to CheckForHeapCorruption(), just a few lines later, detected corruption, at which point you know that the corruption was caused by whatever code executed between the two calls to CheckForHeapCorruption(), and you can then study that particular code to figure out what it's doing wrong, and/or add more calls to CheckForHeapCorruption() into that code as necessary.
Repeat until the bug becomes obvious. Good luck!
If the same variable is consistently being corrupted, data break points are a quick and simple way to find the code responsible for the change (if your IDE supports them). (Debug->New Break Point->New Data Breakpoint... in MS Visual Studio 2008). They won't help if your heap corruption is more random (but figured I'd share the simple answer in case it helps).
There's a tool called electric fence that I think is supported also on Windows.
Essentially, what it does is hijack malloc and co to make every allocation end at page boundary and mark the next page inaccessible.
The effect is that you get a seg fault on buffer overrun.
It probably also have an option for buffer underrun.
Please read this link
Visual Studio - how to find source of heap corruption errors
Is there a good Valgrind substitute for Windows?
It tells technique for finding heap issues on windows.
But on the other hand you can always write (if you are writing new code) memory managers.
The way to do is: use your wrapper apis which will call malloc/calloc etc.
Suppose you have api myMalloc(size_t len);
then inside your function, you can try allocationg HEADER + len + FOOTER.
On your header save info like size of allocation or may be more info. At the footer, add some magic number like deadbeef. And return ptr(from malloc) + HEADER from myMalloc.
When freeing it up using myfree(void *ptr), then just do ptr -HEADER, check the len, then jump at the FOOTER = ptr-HEADER + really allcated len. At this offset, you should find deadbeef, and if you dont find, then you know, its been corrupted.
I have been debugging a crash for days now, that occurs in the depths of OpenSSL (discussion with the maintainers here). I took some time investigating so I'll try to make this question interesting and informative.
First and to give some context, my minimal-sample that reproduces the crash is as follow:
#include <openssl/crypto.h>
#include <openssl/ec.h>
#include <openssl/objects.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <openssl/engine.h>
int main()
{
ERR_load_crypto_strings(); OpenSSL_add_all_algorithms();
ENGINE_load_builtin_engines();
EC_GROUP* group = EC_GROUP_new_by_curve_name(NID_sect571k1);
EC_GROUP_set_point_conversion_form(group, POINT_CONVERSION_UNCOMPRESSED);
EC_KEY* eckey = EC_KEY_new();
EC_KEY_set_group(eckey, group);
EC_KEY_generate_key(eckey);
BIO* out = BIO_new(BIO_s_file());
BIO_set_fp(out, stdout, BIO_NOCLOSE);
PEM_write_bio_ECPrivateKey(out, eckey, NULL, NULL, 0, NULL, NULL); // <= CRASH.
}
Basically, this code generates an Elliptic Curve key and tries to output it to stdout. Similar code can be found in openssl.exe ecparam and on Wikis online. It works fine on Linux (valgrind reports no error at all). It only crashes on Windows (Visual Studio 2013 - x64). I made sure the proper runtimes were linked-to (/MD in my case, for all dependencies).
Fearing no evil, I recompiled OpenSSL in x64-debug (this time linking everything in /MDd), and stepped through the code to find the offending set of instructions. My search led me to this code (in OpenSSL's tasn_fre.c file):
static void asn1_item_combine_free(ASN1_VALUE **pval, const ASN1_ITEM *it, int combine)
{
// ... some code, not really relevant.
tt = it->templates + it->tcount - 1;
for (i = 0; i < it->tcount; tt--, i++) {
ASN1_VALUE **pseqval;
seqtt = asn1_do_adb(pval, tt, 0);
if (!seqtt) continue;
pseqval = asn1_get_field_ptr(pval, seqtt);
ASN1_template_free(pseqval, seqtt);
}
if (asn1_cb)
asn1_cb(ASN1_OP_FREE_POST, pval, it, NULL);
if (!combine) {
OPENSSL_free(*pval); // <= CRASH OCCURS ON free()
*pval = NULL;
}
// Some more code...
}
For those not too familiar with OpenSSL and it's ASN.1 routines, basically what this for-loop does is that it goes trough all the elements of a sequence (starting with the last element) and "deletes" them (more on that later).
Right before the crash happens, a sequence of 3 elements is deleted (at *pval, which is 0x00000053379575E0). Looking at the memory, one can see the following things happen:
The sequence is 12 bytes long, each element being 4-bytes long (in this case, 2, 5, and 10). On each loop iteration, elements are "deleted" by OpenSSL (in this context, neither delete or free are called: they are just set to a specific value). Here is how the memory looks after one iteration:
The last element here was set to ff ff ff 7f which I assume is OpenSSL's way of ensuring no key information leaks when the memory is unallocated later.
Right after the loop (and before the call to OPENSSL_free()), the memory is as follow:
All elements were set to ff ff ff 7f, asn1_cb is NULL so no call is made. The next thing that goes on is the call to OPENSSL_free(*pval).
This call to free() on what seems to be a valid & allocated memory fails and cause the execution to abort with a message: "HEAP CORRUPTION DETECTED".
Curious about this, I hooked into malloc, realloc and free (as OpenSSL permits) to ensure this wasn't a double-free or a free on never-allocated memory. It turns out the memory at 0x00000053379575E0 really is a 12 bytes block that was indeed allocated (and never freed before).
I'm at loss figuring out what happens here: from my research, it seems that free() fails on a pointer that was normally returned by malloc(). In addition to that, this memory location was being written to a couple of instructions before without any problem which confirms the hypothesis that the memory be correctly allocated.
I know it's hard, if not impossible, to debug remotely without all the information but I have no idea what my next steps should be.
So my question is: how exactly is this "HEAP CORRUPTION" detected by Visual Studio's debugger ? What are all the possible causes for it when originating from a call to free() ?
In general the possibilities include:
Duplicate free.
Prior duplicate free.
(Most probable) Your code wrote beyond the limits of the allocated chunk of memory, either before the beginning or after the end. malloc() and friends put extra bookkeeping information in here, such as the size, and probably a sanity-check, which you will fail by overwriting.
Freeing something that hadn't been malloc()-ed.
Continuing to write to a chunk that had already been free()-d.
I could finally find the problem and solve it.
Turned out some instruction was writing bytes past the allocated heap buffer (hence the 0x00000000 instead of the expected 0xfdfdfdfd).
In debug mode this overwrite of the memory guards remains undetected until the memory is freed with free() or reallocated with realloc(). This is what caused the HEAP CORRUPTION message I faced.
I expect that in release mode, this could have had dramatic effects, like overwritting a valid memory block used somewhere else in the application.
For future reference to people facing similar issues, here is how I did:
OpenSSL provides a CRYPTO_set_mem_ex_functions() function, defined like so:
int CRYPTO_set_mem_ex_functions(void *(*m) (size_t, const char *, int),
void *(*r) (void *, size_t, const char *,
int), void (*f) (void *))
This function allows you to hook in and replace memory allocation/freeing functions within OpenSSL. The nice thing is the addition of the const char *, int parameters which are basically filled for you by OpenSSL and contain the filename and line number of the allocation.
Armed with this information, it was easy to find out the place where the memory block was allocated. I could then step through the code while looking at the memory inspector waiting for the memory block to be corrupted.
In my case what happenned was:
if (!combine) {
*pval = OPENSSL_malloc(it->size); // <== The allocation is here.
if (!*pval) goto memerr;
memset(*pval, 0, it->size);
asn1_do_lock(pval, 0, it);
asn1_enc_init(pval, it);
}
for (i = 0, tt = it->templates; i < it->tcount; tt++, i++) {
pseqval = asn1_get_field_ptr(pval, tt);
if (!ASN1_template_new(pseqval, tt))
goto memerr;
}
ASN1_template_new() is called on the 3 sequence elements to initialize them.
Turns out ASN1_template_new() calls in turn asn1_item_ex_combine_new() which does this:
if (!combine)
*pval = NULL;
pval being a ASN1_VALUE**, this instruction sets 8 bytes on Windows x64 systems instead of the intended 4 bytes, leading to memory corruption for the last element of the list.
For the full discussion on how this problem was solved upstream, see this thread.
I can't sleep! :)
I have a reasonably large project on Windows and encountered some heap corruption issues. I have read all SO, including this nice topic: How to debug heap corruption errors?, however nothing was suitable to help me out-of-the-box. Debug CRT and BoundsChecker detected heap corruptions, but addresses were always different and detections point were always far away from the actual memory overwrites. I have not slept till the middle of the night and crafted the following hack:
DWORD PageSize = 0;
inline void SetPageSize()
{
if ( !PageSize )
{
SYSTEM_INFO sysInfo;
GetSystemInfo(&sysInfo);
PageSize = sysInfo.dwPageSize;
}
}
void* operator new (size_t nSize)
{
SetPageSize();
size_t Extra = nSize % PageSize;
nSize = nSize + ( PageSize - Extra );
return Ptr = VirtualAlloc( 0, nSize, MEM_COMMIT, PAGE_READWRITE);
}
void operator delete (void* pPtr)
{
MEMORY_BASIC_INFORMATION mbi;
VirtualQuery(pPtr, &mbi, sizeof(mbi));
// leave pages in reserved state, but free the physical memory
VirtualFree(pPtr, 0, MEM_DECOMMIT);
DWORD OldProtect;
// protect the address space, so noone can access those pages
VirtualProtect(pPtr, mbi.RegionSize, PAGE_NOACCESS, &OldProtect);
}
Some heap corruption errors became obvious and i was able to fix them. There were no more Debug CRT warnings on exit. However, i have some questions regarding this hack:
1. Can it produce any false positives?
2. Can it miss some of the heap corruptions? (even if we replace malloc/realloc/free?)
3. It fails to run on 32-bits with OUT_OF_MEMORY, only on 64-bits. Am I right we simply run out of the virtual address space on 32-bits?
Can it produce any false positives?
So, this will only catch bugs of the class "use after free()". For that purpose, I think, it's reasonably good.
If you try to delete something that wasn't new'ed, that's a different type of bug. In delete you should first check if the memory has been indeed allocated. You shouldn't be blindly freeing the memory and marking it as inaccessible. I'd try to avoid that and report (by, say, doing a debug break) when there's an attempt to delete something that shouldn't be deleted because it was never new'ed.
Can it miss some of the heap corruptions? (even if we replace malloc/realloc/free?)
Obviously, this won't catch all corruptions of heap data between new and and the respective delete. It will only catch those attempted after delete.
E.g.:
myObj* = new MyObj(1,2,3);
// corruption of *myObj happens here and may go unnoticed
delete myObj;
It fails to run on 32-bit target with OUT_OF_MEMORY error, only on 64-bit. Am I right that we simply run out of the virtual address space on 32-bits?
Typically you have available about ~2GB of the virtual address space on a 32-bit Windows. That's good for at most ~524288 new's like in the provided code. But with objects bigger than 4KB, you'll be able to successfully allocate fewer instances than that. And then address space fragmentation will reduce that number further.
It's a perfectly expected outcome if you create many object instances during the life cycle of your program.
This won't catch:
use of uninitialized memory (once your pointer is allocated, you can read garbage from it at will)
buffer overruns (unless you overrun the PageSize boundary)
Ideally, you should write a well-known bit pattern before and after your allocated blocks, so that operator delete can check whether they were overwritten (indicated buffer over- or under-run).
Currently this would be allowed silently in your scheme, and switching back to malloc etc. would allow it to silently damage the heap, and show up as an error later on (eg. when freeing the block after the over-run one).
You can't catch everything though: note for example that if the underlying problem is (valid) pointer somewhere getting overwritten with garbage, you can't detect this until the damaged pointer is de-referenced.
Yes, your current answer can miss heap corruptions of buffer under- and overruns.
Your delete() function is pretty good!
I implemented a new() function in similar manner, that adds guard pages both for under- and overruns. From GFlags documentation I conclude that it protects only against overruns.
Note that when returning simply a pointer next to the underrun guard page then guard page for overruns is likely to be located away from the allocated object and immediate vicinity after the allocated object is NOT guarded.
To compensate for this one would need to return such a pointer that the object is located immediately before overrun guard page (in this case again an underrun is less likely to be detected).
The below code does one or the other alternately for each call of new(). Or one might want to modify it to use threadsafe random generator instead to prevent any interferences with code calling the new().
Considering all this one should be aware that detecting under- and overruns by the below code is still probabilistic to a degree - this is especially relevant in the case when some objects are allocated only once for the entire duration of the program.
NB! Because new() returns a modified aadress, the delete() function also had to be adjusted a bit, so it now uses mbi.AllocationBase instead of ptr for VirtualFree() and VirtualProtect().
PS. Driver Verifier's Special Pool uses similar tricks.
volatile LONG priorityForUnderrun = rand(); //NB! init with rand so that the pattern is different across program runs and different checks are applied to global singleton objects
void ProtectMemRegion(void* region_ptr, size_t sizeWithGuardPages)
{
size_t preRegionGuardPageAddress = (size_t)region_ptr;
size_t postRegionGuardPageAddress = (size_t)(region_ptr) + sizeWithGuardPages - PageSize;
DWORD flOldProtect1;
BOOL preRegionProtectSuccess = VirtualProtect(
(void*)(preRegionGuardPageAddress),
pageSize,
PAGE_NOACCESS,
&flOldProtect1
);
DWORD flOldProtect2;
BOOL postRegionProtectSuccess = VirtualProtect(
(void*)(postRegionGuardPageAddress),
PageSize,
PAGE_NOACCESS,
&flOldProtect2
);
}
void* operator new (size_t size)
{
size_t sizeWithGuardPages = (size + PageSize - 1) / PageSize * PageSize + 2 * PageSize;
void* ptr = VirtualAlloc
(
NULL,
sizeWithGuardPages,
MEM_COMMIT | MEM_RESERVE,
PAGE_READWRITE
);
if (ptr == NULL) //NB! check for allocation failures
{
return NULL;
}
ProtectMemRegion(ptr, sizeWithGuardPages);
void* result;
if (InterlockedIncrement(&priorityForUnderrun) % 2)
result = (void*)((size_t)(ptr) + pageSize);
else
result = (void*)(((size_t)(ptr) + sizeWithGuardPages - pageSize - size) / sizeof(size_t) * sizeof(size_t));
return result;
}
void operator delete (void* ptr)
{
MEMORY_BASIC_INFORMATION mbi;
DWORD OldProtect;
VirtualQuery(ptr, &mbi, sizeof(mbi));
// leave pages in reserved state, but free the physical memory
VirtualFree(mbi.AllocationBase, 0, MEM_DECOMMIT);
// protect the address space, so noone can access those pages
VirtualProtect(mbi.AllocationBase, mbi.RegionSize, PAGE_NOACCESS, &OldProtect);
}