Creating a Crash Log - c++

When I want to find a segfault or any other error which leads to a crash of a program, i would always inspect the core dump with gdb. This is quite painful when such an application runs on a computer without gdb installed.
So a few days ago I used a program (JDownloader) wich wrote a crash log file, and this file contained a stack trace. I thought this would be a great enhancement to my application. But I haven't found any information on how to write a file which contains the stacktrace just before the crash.
Is it even possible? How would I do this on Linux/Windows?
I'm using C/C++.

I believe JDownloader is written in Java. I think the language allows you to retrieve a full plain text stack trace at any point. C++ is unable to do this, because the compiled executable usually doesn't keep any information about the code used to generate it.
Windows API does allows you to catch fatal exceptions and create a dump of the process (or parts of the process, if you don't want to deal with a huge file). This dump can then be inspected with windbg, Visual Studio, or your debugger of choice.
The downside to this is that you must have the exact source code that was used to build the dumped executable, as well as the symbol database (PDB file) that was generated during the build. On top of that, some code can be optimized in ways that makes it impossible for the debugger to give you an accurate stack trace, even with the symbol data.
See MiniDumpWriteDump for details. If you're going to take this route, the best practice is to not generate the dump in the crashing process, but spawn a child process to take a dump of the parent.
There are also C and C++ libraries that can 'manually' record the call stack and give you a textual representation of it at run time, but I haven't encountered any of these that I would suggest.

Related

A program I support is crashing with SIGSEGV but I can't from the .dmp file

As per the title, I can't locate any dump files when this program I support is crashing.
The application's logs clearly mention its a SIGSEGV exception, but I have searched my entire hard drive, and there are no .dmp files anywhere to be found.
The developers of the program have seen similar issues elsewhere but have so far been unable to explain why this is happening - and we're kind of a bit stuck at the moment.
The last section in the application logs reads as :
Received signal SIGSEGV, segmentation violation.
OurApplication::sigHandler 11.
Removing signal handlers.
OurApplication::signalCatched.
OurApplication::sigHandler: exiting application.
Removing signal handlers.
My limited understanding of this is that our application's signal handler might be 'neutralising' the SIGSEGV exception that got thrown. And therefore no core dump is getting generated... I did raise this idea with the developers but they never really seemed have investigated if this might be the reason. The theory they raised in counter was that they think the reason the dmp isn't getting generated is because the program may be crashing twice very close together.
So the questions I have at this point are:
Are there any Windows7 parameters that control the creation of a .dmp file?
Are there any requirements/flags that need to be compiled into a program in order for it (or windows) to create a core dump file if it crashes?
I'm 99% sure it must be windows that is responsible for creating the core file, since the program itself would be dead/terminated when it crashed, correct?
Are there any other things I should be aware of, or check for, or 'evidence' I can collect and then show our developers?
Many thanks in advance
Are there any Windows7 parameters that control the creation of a .dmp file?
There are parameters which control the creation of a crash dump: see MSDN on Collecting user-mode dumps.
Are there any requirements/flags that need to be compiled into a program in order for it (or windows) to create a core dump file if it crashes?
You don't need to compile anything in for the previous answer to work. However, the program needs to terminate due to an unhandled exception, which means you need to let the exception bubble up and not being handled by the unhandled exception handler.
I'm 99% sure it must be Windows that is responsible for creating the core file, since the program itself would be dead/terminated when it crashed, correct?
As stated above, Windows can handle that and it's a good idea to have Windows handle the crash. Imagine that your program is broken due to a memory leak. Some memory has been overwritten. In that case, your unhandled exception handler can be destroyed. Windows, however, still has full control over the process, can suspend it and create a dump from outside (rather from inside).
Are there any other things I should be aware of, or check for, or 'evidence' I can collect and then show our developers?
Well, suggest letting the dump be created by Windows due to above reasons. Then they also don't need to implement a configuration setting (you don't want the crash dump file to be always created, do you?). You don't need to implement a limiting number for the files. You don't need to implement a check for disk space, etc.
And you can suggest to read the Windows Internals 6 books.
Consider creating your own minidump file programatically. Should be plenty of code around showing how to do it. You can try here:
https://stackoverflow.com/search?q=minidump
This way, you're not relying on Dr. Watson or any other settings to create a dump file. Instead you will be calling the functions in DBGHELP.DLL to create the dump file.

Win32: Is there a difference between Dr. Watson's full/mini dumps and writing my own?

I have an application that is occasionally crashing in the release build; unfortunately, it looks like it is crashing in a 3rd party DLL. While trying to get a handle on it I've been swimming in a sea of HOW TOs and descriptions of how Windows creates crash dumps.
I was thinking about using this suggested mini-dump:
Getting a dump of a process that crashes on startup
I was planning on leaving this functionality in the code so the dump is always created for my application without having to have the PC set up beforehand. BTW, this application is not for distribution; it will be paired with our own hardware so I'm not concerned about random users having dump files building on their machines if the application happens to crash.
Additional note: all of the code is C/C++.
Is there a difference between what Dr. Watson (drwtsn32.exe) and this code will produce for me?
With Dr. Watson you'll only get the dumps when the Dr. sees you 'crashed'. Using the dumper API you'll be able to invoke it from any point in the app. Eg. you can trampoline the ordinary asserts to dump instead of showing a dialog. In my experience once you have dump support in your app you'll find it easier down the road to investigate, troubleshoot and fix various problems, simply because you can produce a full dump (or even a minidump) at any place you see fit in code.
There isn't much difference except that if you create your own minidump you have more control over the level of detail in it. By default Minidumps have the stack and some local variables, but creating your own gives you the option of creating a full memory dump also which may prove to be more useful (though this then may make collection of these dumps more problematic if the memory image is large).
If the crash happens reasonably frequently it may be worth just collecting some minidumps that drwatson (or werfault in Vista onwards) produces for you, as that may give you enough information. If it doesn't then you have the option of adding your own unhandled exception filter. Another thing that can happen is that the minidump you receive is the site of the crash rather than a first chance exception that may have arisen. Creating your own minidumps means that you're more likely to get a stack trace closer to where the problem is.
Another option, if you have a machine which exhibits the problem more often is to run ADPlus in the background -- it will sit and wait until your app crashes or throws exceptions then produce some useful log files. It does a similar same thing as the unhandled exception filter except it requires no changes to your app.
The biggest thing to watch out for is that MiniDumpWriteDump has to do memory allocation and file I/O. Calling it from inside the failed process may fail if e.g. heap structures are corrupted.
Calling MiniDumpWriteDump from a helper process works the same as using Dr. Watson, except that you have control over the dump options.
Recommended reading: loader lock deadlock in MiniDumpWriteDump
I don't think so. Although Dr Watson will generate full or mini dumps, you could use the ntsd debugger instead to get a lot more control of what data is included in the dumps.
Dr Watson's minidumps are good enough for most things, you get a call stack and variables. IF you need more, ntsd has a load of options.
The only benefit to using DrWatson is that is comes pre-installed on Windows.

Analyzing a crash in Windows: what does the error message tell us?

A small utility of mine that I made for personal use (written in C++) crashed randomly yesterday (I've used it roughly 100+ hours with no issues so far) and while I don't normally do this, I was feeling a bit adventurous and wanted to try and learn more about the problem. I decided to go into the Event Viewer and see what Windows had logged about the crash:
Faulting application StraightToM.exe, version 0.0.0.0, time stamp 0x4a873d19
Faulting module name : StraightToM.exe, version 0.0.0.0, time stamp 0x4a873d19
Exception code : 0xc0000005
Fault offset : 0x0002d160,
Faulting process id: 0x17b4
Faulting application start time: time 0x01ca238d9e6b48b9.
My question is, what do each of these things mean, and how would I use these to debug my program? Here's what I know so far: exception code describes the error, and 0xc0000005 is a memory access violation (tried to access memory it didn't own). I'm specifically interested in knowing more about the following:
What does the fault offset mean? Does that represent the location in the file where the error occured, or does it mean the assembly 'line' where the error occured? Knowing the fault offset, how would I use a program like OllyDbg to find the corresponding assembly code that caused the error? Or -- even better -- would it be possible to (easily) determine what line of code in the C++ source caused this error?
It's obvious that the time stamp corresponds to the 32-bit UNIX time at the time of the crash, but what does the 64-bit application start time mean? Why would it be 64-bits if the time stamp is 32?
Note that I'm primarily a C++ programmer, so while I know something about assembly, my knowledge of it is very limited. Additionally, this really isn't a serious problem that needs fixing (and is also not easily reproduced, given the nature of the program), I'm just using this more as an excuse to learn more about what these error messages mean. Most of the information about these crash logs that I've found online are usually aimed at the end-user, so they haven't helped me (as the programmer) very much.
Thanks in advance
The 64-bit time stamp is the time application's primary thread was created in 100-nanosecond intervals since January 1, 1601 (UTC) (this is known as FILETIME). The 32-bit timestamp is indeed in time_t format (it tells the time the module was created and is stored in the module's header).
I'd say 0x0002d160 is an offset from the module's load address (it seems too low for an absolute address). Fire up Visual Studio, start the debugger, take a look at the "modules" debug window. Your exe file should be listed there. Find the address where the module is loaded, add 0x0002d160 to that address and take a look at the disassembly at the resulting address. Visual Studio shows source code intermixed with the assembly, you should have no problem figuring out what source line caused the problem.
There isn't much you're going to be able to do postmortem with this information.
The useful bit of information is the exception code, 0xc0000005, which in this case just means an access violation. So you dereferenced null or some other bit of memory you didn't own.
Fault offset, I suspect, is the offset from where your DLL was loaded into memory, so you could in theory add it to your base address and find the offending code, but I'm not sure.
Your best bet for debugging this is to catch it in the debugger the next time this happens. You can use Image File Execution Options to run your app automatically in the debugger. Make sure you have symbols ready (consider building DEBUG if you're currently using RELEASE).
Debugging god John Robbins built a little tool called CrashFinder to help with situations like this:
https://www.wintellect.com/crashfinder-2-8-yes-native-code-still-lives/
It's always a good idea to save PDBs for every build you release to the public (this sounds like a tool you only use in private, but it might be a good idea to keep the PDB symbols around for the latest build).
looks like there is still no good answer here, what if the crash happens outside the development environment.
I think off set is the address where the assembly code crash.
But you need to know where the start of the assembly code of that dll is. or maybe you don't need to know the start address, because you can use assembly tool to open dll, and find the assembly code by adding the offset to start address
My program CrashExplorer will help to analyze such crashes using the Fault offset:
It will work with map and listing files generated with Visual Studio:
The map file lists all functions of the program with addresses. The listing files maps source code to assembler code per translation unit.

How to create binary/hex dump of another process's memory?

I am having trouble finding a reasonable way to dump another process's memory to a file.
After extensive searching, I've been able to find a nice article at CodeProject that has *most* of the functionality I want:
Performing a hex dump of another process's memory. This does a good job of addressing permission issues and sets a good foundation.
However, with this utility I've seen that even a small process, such as an clean Notepad.exe or Calc.exe instance, can generate a dump file over 24MB in size, while the process itself runs under 20KB in memory according to TaskManager.
The article has lead me to believe that perhaps it is also dumping things in shared memory, possibly DLL space and the like. For example, a dump of Calc.exe will include sections that include method names (and presumably memory) from Kernel32.dll:
²³´µKERNEL32.dll ActivateActCtx AddAtomA AddAtomW AddConsoleAliasA AddConsoleAliasW AddLocalAlternateComputerNameA AddLocalAlternateComputerNameW AddRefActCtx AddVectoredExceptionHandler AllocConsole AllocateUserPhysicalPages AreFileApisANSI AssignProcessToJobObject AttachConsole BackupRead BackupSeek BackupWrite BaseCheckAppcompatCache BaseCleanupAppcompatCache
Is there a better way to dump the memory of another process that doesn't lead to this overhead, or perhaps an improvement upon the linked article's code that solves this problem? I want to get the memory that actually belongs to the process itself. I'd be okay with dumping the memory space of functions that are actually used in DLLs, but it seems unnecessary to dump the *entire* contents of multiple DLLs to get the running memory of the process.
I'm looking for a way to get the 30-60KB of a 30KB process, rather than 25MB for a 30KB process. Or at least closer than I can get currently.
Thanks in advance for your suggestions and guidance, it is appreciated.
Note: This is for a console utility, so GUI elements like the ones in the CodeProject article are unimportant.
You're basically asking for a user process minidump. The Windows Debug Helper library has a ready made function for this, MiniDumpWriteDump.
There is a coarse control over the amount of the detail contained in the mini dump from the MINIDUMP_TYPE parameter passed in to the function. The most basic, MiniDumpNormal, will only capture the call stack of each thread in the process. The amount of memory gets progressively more detailed with the other mini dump types.
You can also fine control the amount of information to be written into the mini dump by providing a callback to the MiniDumpWriteDump function and in the callback set the flags on the MINIDUMP_CALLBACK_OUTPUT structure.
The resulted mini dumps can be read with a debugger like Windbg or Visual Studio, or they can be processed by the various functions in the dbghelp.dll library.
Not really a "how to program it" answer, but I just found your question while looking for a tool that could do that, when I ran into PMDump:
http://ntsecurity.nu/toolbox/pmdump/
It's dead easy and simple to use, and creates correct dumps (I just tried it with some programs).

Program only crashes as release build -- how to debug?

I've got a "Schroedinger's Cat" type of problem here -- my program (actually the test suite for my program, but a program nonetheless) is crashing, but only when built in release mode, and only when launched from the command line. Through caveman debugging (ie, nasty printf() messages all over the place), I have determined the test method where the code is crashing, though unfortunately the actual crash seems to happen in some destructor, since the last trace messages I see are in other destructors which execute cleanly.
When I attempt to run this program inside of Visual Studio, it doesn't crash. Same goes when launching from WinDbg.exe. The crash only occurs when launching from the command line. This is happening under Windows Vista, btw, and unfortunately I don't have access to an XP machine right now to test on.
It would be really nice if I could get Windows to print out a stack trace, or something other than simply terminating the program as if it had exited cleanly. Does anyone have any advice as to how I could get some more meaningful information here and hopefully fix this bug?
Edit: The problem was indeed caused by an out-of-bounds array, which I describe more in this post. Thanks everybody for your help in finding this problem!
In 100% of the cases I've seen or heard of, where a C or C++ program runs fine in the debugger but fails when run outside, the cause has been writing past the end of a function local array. (The debugger puts more on the stack, so you're less likely to overwrite something important.)
When I have encountered problems like this before it has generally been due to variable initialization. In debug mode, variables and pointers get initialized to zero automatically but in release mode they do not. Therefore, if you have code like this
int* p;
....
if (p == 0) { // do stuff }
In debug mode the code in the if is not executed but in release mode p contains an undefined value, which is unlikely to be 0, so the code is executed often causing a crash.
I would check your code for uninitialized variables. This can also apply to the contents of arrays.
No answer so far has tried to give a serious overview about the available techniques for debugging release applications:
Release and Debug builds behave differently for many reasons. Here is an excellent overview. Each of these differences might cause a bug in the Release build that doesn't exist in the Debug build.
The presence of a debugger may change the behavior of a program too, both for release and debug builds. See this answer. In short, at least the Visual Studio Debugger uses the Debug Heap automatically when attached to a program. You can turn the debug heap off by using environment variable _NO_DEBUG_HEAP . You can specify this either in your computer properties, or in the Project Settings in Visual Studio. That might make the crash reproducible with the debugger attached.
More on debugging heap corruption here.
If the previous solution doesn't work, you need to catch the unhandled exception and attach a post-mortem debugger the instance the crash occurs. You can use e.g. WinDbg for this, details about the avaiable post-mortem debuggers and their installation at MSDN
You can improve your exception handling code and if this is a production application, you should:
a. Install a custom termination handler using std::set_terminate
If you want to debug this problem locally, you could run an endless loop inside the termination handler and output some text to the console to notify you that std::terminate has been called. Then attach the debugger and check the call stack. Or you print the stack trace as described in this answer.
In a production application you might want to send an error report back home, ideally together with a small memory dump that allows you to analyze the problem as described here.
b. Use Microsoft's structured exception handling mechanism that allows you to catch both hardware and software exceptions. See MSDN. You could guard parts of your code using SEH and use the same approach as in a) to debug the problem. SEH gives more information about the exception that occurred that you could use when sending an error report from a production app.
Things to look out for:
Array overruns - the visual studio debugger inserts padding which may stop crashes.
Race conditions - do you have multiple threads involved if so a race condition many only show up when an application is executed directly.
Linking - is your release build pulling in the correct libraries.
Things to try:
Minidump - really easy to use (just look it up in msdn) will give you a full crash dump for each thread. You just load the output into visual studio and it is as if you were debugging at the time of the crash.
You can set WinDbg as your postmortem debugger. This will launch the debugger and attach it to the process when the crash occurs. To install WinDbg for postmortem debugging, use the /I option (note it is capitalized):
windbg /I
More details here.
As to the cause, it's most probably an unitialized variable as the other answers suggest.
After many hours of debugging, I finally found the cause of the problem, which was indeed caused by a buffer overflow, caused a single byte difference:
char *end = static_cast<char*>(attr->data) + attr->dataSize;
This is a fencepost error (off-by-one error) and was fixed by:
char *end = static_cast<char*>(attr->data) + attr->dataSize - 1;
The weird thing was, I put several calls to _CrtCheckMemory() around various parts of my code, and they always returned 1. I was able to find the source of the problem by placing "return false;" calls in the test case, and then eventually determining through trial-and-error where the fault was.
Thanks everybody for your comments -- I learned a lot about windbg.exe today! :)
Even though you have built your exe as a release one, you can still generate PDB (Program database) files that will allow you to stack trace, and do a limited amount of variable inspection.
In your build settings there is an option to create the PDB files. Turn this on and relink. Then try running from the IDE first to see if you get the crash. If so, then great - you're all set to look at things. If not, then when running from the command line you can do one of two things:
Run EXE, and before the crash do an Attach To Process (Tools menu on Visual Studio).
After the crash, select the option to launch debugger.
When asked to point to PDB files, browse to find them. If the PDB's were put in the same output folder as your EXE or DLL's they will probably be picked up automatically.
The PDB's provide a link to the source with enough symbol information to make it possible to see stack traces, variables etc. You can inspect the values as normal, but do be aware that you can get false readings as the optimisation pass may mean things only appear in registers, or things happen in a different order than you expect.
NB: I'm assuming a Windows/Visual Studio environment here.
Crashes like this are almost always caused because an IDE will usually set the contents of uninitialized variable to zeros, null or some other such 'sensible' value, whereas when running natively you'll get whatever random rubbish that the system picks up.
Your error is therefore almost certainly that you are using something like you are using a pointer before it has been properly initialized and you're getting away with it in the IDE because it doesn't point anywhere dangerous - or the value is handled by your error checking - but in release mode it does something nasty.
In order to have a crash dump that you can analyze:
Generate pdb files for your code.
You rebase to have your exe and dlls loaded in the same address.
Enable post mortem debugger such as Dr. Watson
Check the crash failures address using a tool such as crash finder.
You should also check out the tools in Debugging tools for windows.
You can monitor the application and see all the first chance exceptions that were prior to your second chance exception.
Hope it helps...
Sometimes this happens because you have wrapped important operation inside "assert" macro. As you may know, "assert" evaluates expressions only on debug mode.
A great way to debug an error like this is to enable optimizations for your debug build.
Once i had a problem when app behaved similarily to yours. It turned out to be a nasty buffer overrun in sprintf. Naturally, it worked when run with a debugger attached. What i did, was to install an unhandled exception filter (SetUnhandledExceptionFilter) in which i simply blocked infinitely (using WaitForSingleObject on a bogus handle with a timeout value of INFINITE).
So you could something along the lines of:
long __stdcall MyFilter(EXCEPTION_POINTERS *)
{
HANDLE hEvt=::CreateEventW(0,1,0,0);
if(hEvt)
{
if(WAIT_FAILED==::WaitForSingleObject(hEvt, INFINITE))
{
//log failure
}
}
}
// somewhere in your wmain/WinMain:
SetUnhandledExceptionFilter(MyFilter);
I then attached the debugger after the bug had manifested itself (gui program stopped responding).
Then you can either take a dump and work with it later:
.dump /ma path_to_dump_file
Or debug it right away. The simplest way is to track where processor context has been saved by the runtime exception handling machinery:
s-d esp Range 1003f
Command will search stack address space for CONTEXT record(s) provided the length of search. I usually use something like 'l?10000'. Note, do not use unsually large numbers as the record you're after usually near to the unhanded exception filter frame.
1003f is the combination of flags (i believe it corresponds to CONTEXT_FULL) used to capture the processor state.
Your search would look similar to this:
0:000> s-d esp l1000 1003f
0012c160 0001003f 00000000 00000000 00000000 ?...............
Once you get results back, use the address in the cxr command:
.cxr 0012c160
This will take you to this new CONTEXT, exactly at the time of crash (you will get exactly the stack trace at the time your app crashed).
Additionally, use:
.exr -1
to find out exactly which exception had occurred.
Hope it helps.
With regard to your problems getting diagnostic information, have you tried using adplus.vbs as an alternative to WinDbg.exe? To attach to a running process, use
adplus.vbs -crash -p <process_id>
Or to start the application in the event that the crash happens quickly:
adplus.vbs -crash -sc your_app.exe
Full info on adplus.vbs can be found at: http://support.microsoft.com/kb/286350
Ntdll.dll with debugger attached
One little know difference between launching a program from the IDE or WinDbg as opposed to launching it from command line / desktop is that when launching with a debugger attached (i.e. IDE or WinDbg) ntdll.dll uses a different heap implementation which performs some little validation on the memory allocation/freeing.
You may read some relevant information in unexpected user breakpoint in ntdll.dll. One tool which might be able to help you identifying the problem is PageHeap.exe.
Crash analysis
You did not write what is the "crash" you are experiencing. Once the program crashes and offers you to send the error information to the Microsoft, you should be able to click on the technical information and to check at least the exception code, and with some effort you can even perform post-mortem analysis (see Heisenbug: WinApi program crashes on some computers) for instructions)
Vista SP1 actually has a really nice crash dump generator built into the system. Unfortunately, it isn't turned on by default!
See this article:
http://msdn.microsoft.com/en-us/library/bb787181(VS.85).aspx
The benefit of this approach is that no extra software needs to be installed on the affected system. Grip it and rip it, baby!
As my experience, that are most being memory corruption issues.
For example :
char a[8];
memset(&a[0], 0, 16);
: /*use array a doing some thing */
it is very possible to be normal in debug mode when one runs the code.
But in release, that would/might be crash.
For me, to rummage where the memory is out of bound is too toilsome.
Use some tools like Visual Leak Detector (windows) or valgrind (linux) are more wise choise.
I've seen a lot of right answers. However, there is none that helped me. In my case, there was a wrong usage of the SSE instructions with the unaligned memory. Take a look at your math library (if you use one), and try to disable SIMD support, recompile and reproduce the crash.
Example:
A project includes mathfu, and uses the classes with STL vector: std::vector< mathfu::vec2 >. Such usage will probably cause a crash at the time of the construction of mathfu::vec2 item since the STL default allocator does not guarantee required 16-byte alignment. In this case to prove the idea, one can define #define MATHFU_COMPILE_WITHOUT_SIMD_SUPPORT 1 before each include of the mathfu, recompile in Release configuration and check again.
The Debug and RelWithDebInfo configurations worked well for my project, but not the Release one. The reason behind this behavior is probably because debugger processes allocation/deallocation requests and does some memory bookkeeping to check and verify the accesses to the memory.
I experienced the situation in Visual Studio 2015 and 2017 environments.
Something similar happend to me once with GCC. It turned out to be a too aggressive optimization that was enabled only when creating the final release and not during the development process.
Well, to tell the truth it was my fault, not gcc's, as I didn't noticed that my code was relying on the fact that that particular optimization wouldn't have been done.
It took me a lot of time to trace it and I only came to it because I asked on a newsgroup and somebody made me think about it. So, let me return the favour just in case this is happening to you as well.
I've found this this article useful for your scenario. ISTR the compiler options were a little out of date. Look around your Visual Studio project options to see how to generate pdb files for your release build, etc.
It's suspicious that it would happen outside the debugger and not inside; running in the debugger does not normally change the application behavior. I would check the environment differences between the console and the IDE. Also, obviously, compile release without optimizations and with debug information, and see if that affects the behavior. Finally, check out the post-mortem debugging tools other people have suggested here, usually you can get some clue from them.
Debugging release builds can be a pain due to optimizations changing the order in which lines of your code appear to be executed. It can really get confusing!
One technique to at least narrow down the problem is to use MessageBox() to display quick statements stating what part of the program your code has got to ("Starting Foo()", "Starting Foo2()"); start putting them at the top of functions in the area of your code that you suspect (what were you doing at the time when it crashed?). When you can tell which function, change the message boxes to blocks of code or even individual lines within that function until you narrow it down to a few lines. Then you can start printing out the value of variables to see what state they are in at the point of crashing.
Try using _CrtCheckMemory() to see what state the allocated memory is in .
If everything goes well , _CrtCheckMemory returns TRUE , else FALSE .
You might run your software with Global Flags enabled (Look in Debugging Tools for Windows). It will very often help to nail the problem.
Make your program generate a mini dump when the exception occurs, then open it up in a debugger (for example, in WinDbg). The key functions to look at: MiniDumpWriteDump, SetUnhandledExceptionFilter
Here's a case I had that somebody might find instructive. It only crashed in release in Qt Creator - not in debug. I was using .ini files (as I prefer apps that can be copied to other drives, vs. ones that lose their settings if the Registry gets corrupted). This applies to any apps that store their settings under the apps' directory tree. If the debug and release builds are under different directories, you can have a setting that's different between them, too. I had preference checked in one that wasn't checked in the other. It turned out to be the source of my crash. Good thing I found it.
I hate to say it, but I only diagnosed the crash in MS Visual Studio Community Edition; after having VS installed, letting my app crash in Qt Creator, and choosing to open it in Visual Studio's debugger. While my Qt app had no symbol info, it turns out that the Qt libraries had some. It led me to the offending line; since I could see what method was being called. (Still, I think Qt is a convenient, powerful, & cross-platform LGPL framework.)
I had this problem too. In my case, the RELEASE mode was having msvscrtd.dll in the linker definition. We removed it and the issue resolved.
Alternatively, adding /NODEFAULTLIB to the linker command line arguments also resolved the issue.
I'll add another possibility for future readers: Check if you're logging to stderr or stdout from an application with no console window (ie you linked with /SUBSYSTEM:WINDOWS). This can crash.
I had a GUI application where I logged to both stderr and a file in both debug and release, so logging was always enabled. I created a console window in debug for easy viewing of the logs, but not in release. However, if the VS debugger is attached to the release build, it'll automatically pipe stderr to the VS output window. So only in release with no debugger did it actually crash when I wrote to stderr.
To make things worse, printf debugging obviously didn't work, which I didn't understand why until I'd tracked down the root cause (by painfully bisecting the codebase by inserting an infinite loop in various spots).
I had this error and vs crashed even when trying to !clean! my project. So I deleted the obj files manually from the Release directory, and after that it built just fine.
I agree with Rolf. Because reproducibility is so important, you shouldn't have a non-debug mode. All your builds should be debuggable. Having two targets to debug more than doubles your debugging load. Just ship the "debug mode" version, unless it is unusable. In which case, make it usable.