If my C++ app crashes on Windows I want to send useful debugging information to our server.
On Linux I would use the GNU backtrace() function - is there an equivalent for Windows?
Is there a way to extract useful debugging information after a program has crashed? Or only from within the process?
(Advice along the lines of "test you app so it doesn't crash" is not helpful! - all non-trivial programs will have bugs)
The function Stackwalk64 can be used to snap a stack trace on Windows.
If you intend to use this function, you should be sure to compile your code with FPO disabled - without symbols, StackWalk64 won't be able to properly walk FPO'd frames.
You can get some code running in process at the time of the crash via a top-level __try/__except block by calling SetUnhandledExceptionFilter. This is a bit unreliable since it requires you to have code running inside a crashed process.
Alternatively, you can just the built-in Windows Error Reporting to collect crash data. This is more reliable, since it doesn't require you to add code running inside the compromised, crashed process. The only cost is to get a code-signing certificate, since you must submit a signed binary to the service. https://sysdev.microsoft.com/en-US/Hardware/signup/ has more details.
You can use the Windows API call MiniDumpWriteDump if you wish to roll your own code. Both Windows XP and Vist automate this process and you can sign up at https://winqual.microsoft.com to gain access to the error reports.
Also check out http://kb.mozillazine.org/Breakpad and http://www.codeproject.com/KB/debug/crash_report.aspx for other solutions.
This website provides quite a detailed overview of stack retrieval on Win32 after a C++ exception:
http://www.eptacom.net/pubblicazioni/pub_eng/except.html
Of course, this will only work from within the process, so if the process gets terminated or crashes to the point where it terminates before that code is run, it won't work.
Generate a minidump file. You can then load it up in windbg or Visual Studio and inspect the entire stack where the crash occurred.
Here's a good place to start reading.
Its quite simple to dump the current stackframe addresses into a log file. All you have to do is get such a function called on program faults (i.e. a interrupt handler in Windows) or asserts. This can be done at released versions as well. The log file then can be matched with a map file resulting in a call stack with function names.
I published a article about this some years ago.
See http://www.ddj.com/architect/185300443
Let me describe how I handle crashes in my C++/WTL application.
First, in the main function, I call _set_se_translator, and pass in a function that will throw a C++ exception instead of using structured windows exceptions. This function gets an error code, for which you can get a Windows error message via FormatMessage, and a PEXCEPTION_POINTERS argument, which you can use to write a minidump (code here). You can also check the exception code for certain "meltdown" errors that you should just bail from, like EXCEPTION_NONCONTINUABLE_EXCEPTION or EXCEPTION_STACK_OVERFLOW :) (If it's recoverable, I prompt the user to email me this minidump file.)
The minidump file itself can be opened in Visual Studio like a normal project, and providing you've created a .pdb file for your executable, you can run the project and it'll jump to the exact location of the crash, together with the call stack and registers, which can be examined from the debugger.
If you want to grab a callstack (plus other good info) for a runtime crash, on a release build even on site, then you need to set up Dr Watson (run DrWtsn32.exe). If you check the 'generate crash dumps' option, when an app crashes, it'll write a mini dump file to the path specified (called user.dmp).
You can take this, combine it with the symbols you created when you built your server (set this in your compiler/linker to generate pdb files - keep these safe at home, you use them to match the dump so they can work out the source where the crash occurred)
Get yourself windbg, open it and use the menu option to 'load crash dump'. Once it's loaded everything you can type '~#kp' to get a callstack for every thread (or click the button at the top for the current thread).
There's good articles to know how to do this all over the web, This one is my favourite, and you'll want to read this to get an understanding of how to helpyourself manage the symbols really easily.
You will have to set up a dump generation framework in your application, here is how you may do it.
You may then upload the dump file to the server for further analysis using dump analyzers like windbg.
You may want to use adplus to capture the crash callstack.
You can download and install Debugging tools for Windows.
Usage of adplus is mentioned here:
Adplus usage
This creates the complete crash or hang dump. Once you have the dump, Windbg comes to the rescue. Map the correct pdbs and symbols and you are all set to analyze the dump. To start with use the command "!analyze -v"
Related
I've developed a program by a customer who's experiencing when he do a certain operation. This isn't happening always on the same place and on the same data and, moreover, it is not happening nor in my local developing machine nor in my test Virtual Machine (which is free of all developing equipment).
Given these conditions, I've decided to compile with MAP (enabled in Configuring Properties-> Linker->Debugger with option /MAP) to see which function is causing crash.
If I've correctly understood, when the program crash I've to check down the offset error and then, search in my MAP under the column RVA+BASE:
Address Publics by Value Rva+Base Lib:Object
0001:00037af0 ?PersonalizzaPlancia#CDlgGestioneDatiProgetto#MosaicoDialogs##IAEXXZ 00438af0 f DlgGestioneDatiProgetto.obj
0001:00038000 ?SalvaTemporanei#CDlgGestioneDatiProgetto#MosaicoDialogs##IAEXXZ 00439000 f DlgGestioneDatiProgetto.obj
Actually, my crash happens at offset: 00038C90 So I should think that it's somewhere in the method:
MosaicoDialogs::CDlgGestioneDatiProgetto::PersonalizzaPlancia
but this is not absolutely possible, so assuming that the computer can't be wrong, I'm the one who's doing it bad.
Can someone explain me how to read MAP in correct way?
don't bother - instead, build the project with symbols enabled and strip them into a pdb file.
Modify the program a little, to write a minidump when it crashes using a unhandled exception handler
Give the newly compiled program to the customer, and when it crashes call MiniDumpWriteDump.
Ask the customer to send this .dmp file to you, and you then simply load it up in Visual Studio (or WinDbg) and it will match up the symbols to the program, and will also match up the code. You should be able to see the exact line of code and some of the variables involved. (if using VS, when you load the .dmp file, top right corner will be an option to "start debugging" click that as it will 'start debugging' at the point of the crash)
Try it first locally - put a div by zero error somewhere in your program and see if you can debug the dump after its been run. Note that you must keep the exact same symbol file for each build of your program - they match exactly. You cannot expect a symbol file for one build to match another build, even if nothing changed.
There are tutorials for this kind of thing, such as this one from CodeProject that looks like it describes what you need.
Reading of MAP files to find out crash location is explained nicely in this code project article.
http://www.codeproject.com/Articles/3472/Finding-crash-information-using-the-MAP-file
Hope helps.
For postmortem debugging, there's an alternative that would not required the use of a map file. Rather, it would require you to create a simple registry script to enable some WER (Windows Error Reporting) flags to trap the crash dump file. First, build your application with debug symbols. Then, follow the instructions for Collecting User-Mode Dumps. Basically, you create a sub key under the "LocalDumps" key. This sub key must be the name of your application, for example, "myapplication.exe". Then, create the "DumpCount", "DumpType", and "DumpFolder" keys/values. Have the user run the registry script. This will enable trapping the dump locally. Then, have the user force the crash to collect the dump file. The user can then send the dump file to you to debug using the symbols you created earlier. Lastly, you'll need to create a registry script that removes the keys/values you added to the registry.
Question: Can I set up VS2010 so it automatically writes debug output to a file?
Motivation: I have a DirectX 9 application that I'm trying to debug. I've noticed that when my application is fullscreen, it may crash under certain conditions. Normally I would just check my logs or DirectX debug output. However, the way my program crashes prevents that. It freezes and does not respond to any my attempts to end it (including "End Process" from task manager). Moreover, it also freezes my VS2010, and so VS doesn't respond to any commands either. The only way out of this whole thing that I've found is to End VS process. This, however, also destroys the output I'd very much like to read.
Now I see two ways out of this. First is to write all the debug info to a file but I have no idea how to do it. Second is to make my application crash in a more friendly way, but this seems like a difficult task.
Probably MiniDumpWriteMiniDump(..) helps on this. You can at any time dump the current state of the process to a file. After that, you can open the dumps with Visual Studio and analyze the state of the process - this includes callstacks of every thread, variable values...
Try to identify conditions in which your process crashes and write one or more dumps.
Another try is to install the Windows Debugging Tools and use WinDbg to debug your application. This is not as comfortable as Visual Studio, but allows a deeper insight.
Edit:
If there are debug statements made with OutputDebugString(..), you can use DebugView (from Microsoft, earlier Sysinternals) to display it.
We have a problem for which I am looking for a direction. We have an old MFC based application, and due to various reasons, the application is crashing sometimes intermittently in some weird scenarios. And even the customers who are using our application and getting these crashes are finding difficulty in identifying the pattern of crash. So, I had a thought that, if we can have a mechanism by which we can generate a log whenever the application crashes. Like for example, the call stack or any other information in that log. I know,, we can use the crash dump in this case, but then I feel like having a log is a better option. So any help or information in this regard would be really helpful.
Thank you.
You can find a good implementation of crash reporter in the link here.
When you compile your release build, make sure that both DEBUG and /MAP are enabled. Store your binaries together with your .map files and let your customer run this version until a crash is produced. In the Event Viewer you will then find a crash log with a crash offset. Then debug step into your code (F10) and use the crash offset together with some nifty tricks and tricks to jump (set the EIP register to... well, you have to google this a bit) to the location where the crash occurred. You should now be able to find the error!
we have a problem with an application we're developing. Very seldom, like once in a hundred, the application crashes at start up. When the crash happens it brings down the whole system, the computer starts to beep and freezes up completely, the only way to recover is to turn off the power (we're using Windows XP). The rarity of the crash combined with the fact that we can't break into the debugger or even generate a stackdump when it occurs makes it extremely hard to debug.
I'm looking for something that logs all function calls to a file. Does such a tool exist? It shouldn't be impossible to implement, profilers like VTune does something very similar.
We're using visual studio 2008 (C++).
Thanks
A.B.
Logging function entries/exits is a low-level approach to your problem. I would suggest using automatic debugger instrumentation (using Debugger key under Image File Execution Options with regedit or using gflags from the package I provide a link to below) and trying to repro the problem until it crashes. Additionally, you can have the debugger log function call history of suspected module(s) using a script or have collect any other information.
But not knowing the details of your application it is very hard to suggest a solution. Is it a user app, service or a driver? What does "crashes at startup" mean - at windows startup or app's startup?
Use this debugger package to troubleshoot.
The only problem with the logging idea is that when the system crashes, the latest log entries might still be in the cache and have no chance to be written to disk...
If it was me I would try running the program on a different PC - it might be flaky hardware or drivers causing the problem. An application program "shouldn't" be able to bring down the system.
A few Ideas-
There is a good chance that just prior to your crash there is some sort of exception in the application. if you set you handler for all unhandled exceptions using SetUnhandledExceptionFilter() and write a stack trace to your log file, you might have a chance to catch the crash in action.
Just remember to flush the file after every write.
Another option is to use a tool such as strace which logs all of the system calls into the kernel (there are multiple flavors and implementations for that so pick your favorite). if you look at the log just before the crash you might find the culprit
Have you considered using a second machine as a remote debugger (via the network)? When the application (and system) crashes, the second machine should still show some useful information, if not the actual point of the problem. I believe VC++ has that ability, at least in some versions.
For Visual C++ _penter() and _pexit() can be used to instrument your code.
See also Method Call Interception in C++.
GCC (including the version MingGW for Windows development) has a code generation switch called -finstrument-functions that tells the compiler to emit special calls to functions called __cyg_profile_func_enter and __cyg_profile_func_exit around every function call. For Visual C++, there are similar options called /GH and /Gh. These cause the compiler to emit calls to __penter and __pexit around function calls.
These instrumentation modes can be used to implement a logging system, with you implementing the calls that the compiler generates to output to your local filesystem or to another computer on your network.
If possible, I'd also try running your system using valgrind or a similar checking tool. This might catch your problem before it gets out-of-hand.
Sometimes my c++ program crashes in debug mode, and what I got is a message box saying that an assertion failed in some of the internal memory management routines (accessing unallocated memory etc.). But I don't know where that was called from, because I didn't get any stack trace. How do I get a stack trace or at least see where it fails in my code (instead of library/ built-in routines)?
If you have a crash, you can get information about where the crash happened whether you have a debug or a release build. And you can see the call stack even if you are on a computer that does not have the source code.
To do this you need to use the PDB file that was built with your EXE. Put the PDB file inside the same directory as the EXE that crashed. Note: Even if you have the same source code, building twice and using the first EXE and the second PDB won't work. You need to use the exact PDB that was built with your EXE.
Then attach a debugger to the process that crashed. Example: windbg or VS.
Then simply checkout your call stack, while also having your threads window open. You will have to select the thread that crashed and check on the callstack for that thread. Each thread has a different call stack.
If you already have your VS debugger attached, it will automatically go to the source code that is causing the crash for you.
If the crash is happening inside a library you are using that you don't have the PDB for. There is nothing you can do.
If you run the debug version on a machine with VS, it should offer to bring it up and let you see the stack trace.
The problem is that the real problem is not on the call stack any more. If you free a pointer twice, that can result in this problem somewhere else unrelated to the program (the next time anything accesses the heap datastructures)
I wrote this blog on some tips for getting the problem to show up in the call stack so you can figure out what is going on.
http://www.atalasoft.com/cs/blogs/loufranco/archive/2007/02/06/6-_2200_Pointers_2200_-on-Debugging-Unmanaged-Code.aspx
The best tip is to use the gflags utility to make pointer issues cause immediate problems.
You can trigger a mini-dump by setting a handler for uncaught exceptions. Here's an article that explains all about minidumps
Google actually implemented their own open source crash handler called BreakPad, which also mozilla use I think (that's if you want something more serious - a rich and robust crash handler).
If I remember correctly that message box should have a button which says 'retry'. This should then break the program (in the debugger) at the point where the assertion happened.
CrashFinder can help you locate the place of the exception given the DLL and the address of the exception reported.
You can take this code and integrate it into your application to have a stack trage automatically generated when there is an uncaught exception. This is generally performed using __try{} __except{} or with a call to SetUnhandledExceptionFilter which allows you to specify a callback to all unhandled exceptions.
You can also have a post-mortem debugger installed on the client system. This is a decent, general way to get information when you do not have dump creation built into your application (maybe for an older version for which you must still get information).
Dr. Watson on Windows can be installed by running: drwtsn32 -i Running drwtsn32 (without any options) will bring up the configuration dialog. This will allow the creation of crash dump files, which you can later analyze with WinDbg or something similar.
You can use Poppy for this. You just sprinkle some macros across your code and it will gather the stack trace, together with the actual parameter values, local variables, loop counters, etc. It is very lightweight so it can be left in the release build to gather this information from crashes on end-user machines