Question: Can I set up VS2010 so it automatically writes debug output to a file?
Motivation: I have a DirectX 9 application that I'm trying to debug. I've noticed that when my application is fullscreen, it may crash under certain conditions. Normally I would just check my logs or DirectX debug output. However, the way my program crashes prevents that. It freezes and does not respond to any my attempts to end it (including "End Process" from task manager). Moreover, it also freezes my VS2010, and so VS doesn't respond to any commands either. The only way out of this whole thing that I've found is to End VS process. This, however, also destroys the output I'd very much like to read.
Now I see two ways out of this. First is to write all the debug info to a file but I have no idea how to do it. Second is to make my application crash in a more friendly way, but this seems like a difficult task.
Probably MiniDumpWriteMiniDump(..) helps on this. You can at any time dump the current state of the process to a file. After that, you can open the dumps with Visual Studio and analyze the state of the process - this includes callstacks of every thread, variable values...
Try to identify conditions in which your process crashes and write one or more dumps.
Another try is to install the Windows Debugging Tools and use WinDbg to debug your application. This is not as comfortable as Visual Studio, but allows a deeper insight.
Edit:
If there are debug statements made with OutputDebugString(..), you can use DebugView (from Microsoft, earlier Sysinternals) to display it.
Related
I'd like to extend my application with a kind of logging, where one can choose to log to a file and/or to log to the "Visual Studio" "output" window.
For the latter, it seems that the function OutputDebugString() is doing the trick, but only for "Debug" builds, and I'd wish to have this also for "Release" builds, so I came up with the idea of using tracepoints (at least, I think that tracepoints are also writing to the "output" window for "release" builds), but here I have a rather frustrating problem:
Tracepoints are a special kind of breakpoints and seem to be written to *.suo files, being present in <project directory>\.vs\<project name>\v15 directory (see Where are Visual Studio breakpoints saved?., thank you, Max).
My idea was to write a postbuild event, adding a tracepoint to every line in the *.suo file where a log() operation was called.
This, however, is not possible, for the simple reason that the *.suo files are binary :-(
Does anybody know of another way to add tracepoints to an application at build-time or how to write (with a "Debug"/"Release" switch) to the "output" window of Visual Studio?
Thanks in advance
OutputDebugString() should work in Release builds - if there is anything to consume the output (such as WinDbg.) Debugging Tools for Windows (WinDbg, KD, CDB, NTSD)
As to why it's not - probably want to look into that, as it's possible your application isn't building quite the way you expect otherwise.
As I understand it, Tracepoints only work in a debugger (Debug or Release) - but the way I believe this works is, the debugger will, on application load/startup, (or during a run) actually replace instructions with a BRK and when the application triggers the break, the debugger catches it, substitutesthe original instruction back in, before performing whatever task it should (whether that be pause, increment a counter, etc.) - but I think Tracepoints will only work with an attached debugger, as otherwise there's nothing to perform the replace, catch and substitute operations, and the code runs uninterrupted.
However, if I read your intentions correctly, you might actually want to use something like Event Tracing* instead. This should output what you want to see to a standard windows log file, will work in debug or release, inside or outside of a debugger. However it won't output to the VS output window. For that, OutputDebugString() should the be correct operation.
*Disclaimer - not used them myself, as we use VS output and WinDbg for now - but it looks like a better way for deployed applications, going forwards.
I've experienced this with every version of Visual Studio starting from 2012 (2012, 2013, 2015 Preview), on multiple computers and multiple projects, but I haven't figured out how to fix it:
Whenever I'm debugging a 64-bit(?) C++ console program, after a few minutes and seemingly completely randomly (when I'm not clicking or typing anything), the console window for the program spontaneously closes and I can no longer debug or step through the program with Visual Studio. When I press Stop and attempt to restart debugging, I usually get ERROR_NETWORK_UNREACHABLE:
// MessageId: ERROR_NETWORK_UNREACHABLE
// MessageText:
// The network location cannot be reached. For information about network troubleshooting, see Windows Help.
#define ERROR_NETWORK_UNREACHABLE 1231L
If I try to attach to the process manually I get the error:
Unable to attach to the process.
The only fix I've found for this is to restart Visual Studio. I can't find any other way to fix it, and I've tried running Process Monitor but haven't found anything.
What causes this problem and how can I fix it?
(?) Upon further checking it seems that this only happens in 64-bit mode, but I'm not 100% sure.
Ok, this is just so wrong
I also have issues with this bug, and in my case it occurred every other debug session. Which meant debug -> stop -> debug -> bug -> restart visual studio -> go to start (repeat every minute during the whole day).
Needless to say I was driven to find a solution. So yesterday I tried procmon, spend hours looking at API monitor differences, looked at plugins, netstat, etc, etc, etc. And found nothing. I gave up.
Today
Until today.
To track down a stupid bug in my program today, I launched appverifier. For my application, I ran the 'basics' tests and clicked save. After a few hours this led me to the bug in my program, which was something like this (extremely simplified version):
void* dst = _aligned_malloc(4096, 32);
memcpy(dst, src, 8192);
Obviously this is a bug and obviously it needed fixing. I noticed the error after putting a breakpoint on the memcpy line, which was not executed.
After a stop and 'debug' again I was surprised to find that I could actually debug the program for the second time. And now, several hours later, this annoying bug here hasn't re-emerged.
So what appears to be going on
So... apparently data from my program is bleeding through into the data or execution space of the debugger, which in turn appears to generate the bug.
I see you thinking: No, this shouldn't happen... you're right; but apparently it does.
So how to fix it? Basically fixing your program (more particular: heap corruption issues) seems to make the VS debugger bug go away. Using appverifier.exe (It's in Debugging tools for Windows) will give you a head start.
Why this works
Since VS2012, VC++ uses a different way to manage the heap. Hans Passant explains it here: Does msvcrt uses a different heap for allocations since (vs2012/2010/2013) .
What basically happens is that heap corruption will break your debugger. The AppVerifier basic settings will ensure that a breakpoint is triggered just before the application does something to corrupt the heap.
So, what happens now is that before the process will break the heap, a breakpoint will trigger instead, which usually means you will terminate the process. The net effect is that the heap will still be in-tact before you terminate your program, which means that your debugger will still function.
"Test"
Before using appverifier -- bug triggered every 2 minutes
While using appverifier -- VS debugger has been stable for 5 days (and counting)
This is an environmental problem of course. Always hard to troubleshoot, SysInternals' utilities like Process Monitor and Process Explorer are your primary weapons of choice. Some non-intuitive ways that a network error can be generated while debugging:
Starting with VS2012, the C runtime library had a pretty drastic modification that can cause very hard to diagnose mis-behavior if your program corrupts the heap. Much like #atlaste describes. Since time memorial, the CRT always created its own heap, underlying call was HeapCreate(). No more, it now uses GetProcessHeap(). This is very convenient, much easier now to deal with DLLs that were built with /MT. But with a pretty sharp edge, you can now easily corrupt the memory owned by Microsoft code. Not strongly indicated if you can't reattach a 64-bit program, you'd have to kill msvsmon.exe to clear up the corruption.
The Microsoft Symbol Server supplies PDBs for Microsoft executables. They normally have their source+line-number info stripped, but not all of them. Notably not for the CRT for example. Those PDBs were built on a build server owned by DevDiv in Redmond that had the source code on the F: drive. A few around that were built from the E: drive, Patterns+Practices uses that (unlikely in a C++ program). Your debugger will go look there to try to find source code. That usually ends well, it gives up quickly, but not if your machine uses those drive letters as well. Diagnose by clearing the symbol cache and disabling the symbol server with Tools + Options, Debugging, Symbols.
The winapi suffers from two nasty viral infections it inherited from another OS that add global state to any process. The PATH environmental variable and the default working directory. Use Control Panel + System + Advanced + Environment to have a look at PATH, copy/paste the content of the intentionally small textboxes into a text editor. Make sure it is squeaky clean, some paralysis at the usual mess is normal btw. Take no prisoners. Having trouble with the default directory is much harder to troubleshoot. Both should pop out when you use Process Monitor.
No slamdunk explanations, it is a tough problem, but dark corners you can look in.
I have the same problem. Thought it was related to 64 bit console apps, where it is very easily triggered with almost any debug session. But, it also happens on 64 bit windows apps too. Now I am seeing it on 32 bit windows apps. I am running Windows 8.1 pro on a single desktop with the latest version of vs 2013 and no remote debugging. My (added) extensions are Visual Assist, Advanced Installer, ClangFormat, Code Alignment, Code Compare, Duplicate Selection, Productivity Power Tools 2013, and Visual SVN.
I discovered that the "Visual Studio 2013\Settings\CurrentSettings.vssettings" file gets corrupted. You can delete this file and recreate it by restarting VS or you can try to edit the XML. I then keep a copy of a good settings file that I use to replace when it gets corrupted again.
In my case, the corrupted line begins with
</ToolsOptionsSubCategory><ToolsOptionsSubCategory name="XAML" RegisteredName="XAML"
... and it is extremely long (I think this is why it is prone to corruption).
I just disabled in the Menu
Tools > Options
Debugging > Edit and Continue
Native-only options > Enable native Edit and Continue
and now it does not give the that stupid error which was preventing the starting of the debuggee application.
I also had the same problem with VS2015. It was so frustrating that a simple Hello World program gave this error when I ran debugger for the second time. Tried uninstall and reinstall and didn't work.
Finally, the solution mentioned in https://social.msdn.microsoft.com/Forums/vstudio/en-US/8dce0952-234f-4c18-a71a-1d614b44f256/visual-studios-2012-cannot-findlaunch-project-exe?forum=vsdebug
worked. Reset all visual studio settings using Tools->Import and Export settings. Now the issue is not occurring.
I have a native C++ program which runs more than 20 times slower when started with Debug (F5), but runs at normal speed when using start without debug (Ctrl + F5).
It does not matter whether I use a debug or release build. Also if I use WinDbg the program is a magnitude slower.
Is there some setting I did choose wrong or something?
Set the _NO_DEBUG_HEAP environment variable to 1 (as seen on http://preshing.com/20110717/the-windows-heap-is-slow-when-launched-from-the-debugger).
This can be done from inside Visual Studio, too.
Now this is just a workaround, and I'm curious to know how to refactor a program which suffers from this kind of problem. Do you have many std::map's, shared_ptr, or any other big indirections by any chance?
This is of course not caused by having the _DEBUG symbol defined or compiling the code in the debug configuration. The added debugging code runs whether or not the debugger is attached to the program.
The debugger doesn't normally affect code execution, it stays out of the way by calling WaitForDebugEvent. Which blocks it, until the operating system tells it that something noteworthy happened. That can trigger a bunch of code in the debugger that can slow down your program. You can see the events listed in the DEBUG_EVENT structure documentation.
Annotating them a bit beyond the documentation: the debugger steps in and can slow down your program when:
The program loads or unloads a DLL. Lots of stuff happens during load, the debugger goes hunting for a debug symbol file (.pdb). It may contact a symbol server to download it. Any breakpoints that were set in the DLL source code will get activated. This can be quite slow, but the effect is temporary and generally only slows down the startup. You can see the load/unload notification in the Output window.
The program raises an exception. This activates the debugger at the moment the exception is raised, a "first chance notification". Which can be very helpful, you can use the Debug + Exception, Thrown checkbox to make the debugger stop when the exception is raised. You can see the notification message in the Output window. This does slow down code that raises and catches exceptions tremendously and is quite likely the source of your slowdown. Never use exceptions for flow control.
A thread starts running or terminates. Again, a notification message is printed to the Output window. You'd have to create a lot of threads to make this slow down your program.
When your program uses OutputDebugString() for tracing purposes. Visible in the Output window. Another good candidate for a slow down, output falls in the bit bucket if no debugger is attached. You shouldn't have any trouble diagnosing this as the cause, the obvious side-effect is seeing a lot of messages in the Output window.
When the program hits a breakpoint. Not a lot of reasons to be stumped by that one. But you can set breakpoints that slow down the program a lot yet don't cause a debugger break. Particularly the Conditional breakpoint, Hit counter, Filter and the When Hit operation will be slow. Use Debug + Windows + Breakpoints to review the breakpoints that are defined.
When a process is created under the debugger, the operating system by default uses the debug heap. The debug heap does more verification of your memory, especially at de-alloc.
There are several possible options in order to disable the use of the Debug Heap:
Attach to the process soon after startup. This will allow you to speed up performance in debug mode knowingly so and being fully aware of the mode in which you are running.
Add the environment variable setting _NO_DEBUG_HEAP=1.
This can be set either globally for the machine or for a specific instance of Visual Studio.
a. Globally you would set an environment variable through the Control Panel → System → Advanced system settings → Environment Variables, and there add the variable _NO_DEBUG_HEAP=1.
Note: This will have an affect on EVERY application you debug.
b. For an instance of Visual Studio you can open a command prompt, setting the environment variable _NO_DEBUG_HEAP=1 and then open visual studio from inside that command prompt. This will influence only the processes created from
that instance of Visual Studio will inherit the environment
variable.
Append the behavior of the debugger, this is possible for VS2015. There are 2 ways to override this:
a. To modify for a specific project, go to the project properties Configuration Properties → Debugging and change the Environment property _NO_DEBUG_HEAP to 1
b. To modify for every project in Visual Studio, go to Tools → Options → Debugging and check the option: “Enable Windows debug heap allocator (Native only)”.
Note: f the _NO_DEBUG_HEAP environment variable mentioned in the 'a' is set in a project level it will override this global setting.
For me the difference in performance between debug mode and release mode is about a factor 40. After some digging, it appears several things contribute to the difference in performance, but there is one compiler option that quadruples my debug performance almost for free.
Namely, changing /ZI into /Zi. For a description, see the MSDN page.
I don't use the edit and continue feature anyway.
No one has mentioned closing unused source windows.
After closing 20+ unused windows, debug source stepping went from ~5s back down to ~.2s. This unusually slow project loads a DLL dynamically and that DLL was also the one being stepped through (and having source windows open) so it seems likely related. However this was C# (headline and tags are non-specific).
There are several things that are different if you run the debug build outside the IDE. One is that the IDE takes a while to load symbols, and if you depend on a lot of libraries then that startup time can be significant.
If you are using a symbol server (including the Microsoft public symbol server) then this may add to the startup time, so ensure you have a local symbol cache in your _NT_SYMBOL_PATH variable if that is the case.
Also the IDE runs with the debug heap enabled, but I don't think this happens if you run outside the IDE.
Debugging Visual C++ comes with lots and lots of overhead, especially in the STL. Try not defining _DEBUG, and define NDEBUG.
If my C++ app crashes on Windows I want to send useful debugging information to our server.
On Linux I would use the GNU backtrace() function - is there an equivalent for Windows?
Is there a way to extract useful debugging information after a program has crashed? Or only from within the process?
(Advice along the lines of "test you app so it doesn't crash" is not helpful! - all non-trivial programs will have bugs)
The function Stackwalk64 can be used to snap a stack trace on Windows.
If you intend to use this function, you should be sure to compile your code with FPO disabled - without symbols, StackWalk64 won't be able to properly walk FPO'd frames.
You can get some code running in process at the time of the crash via a top-level __try/__except block by calling SetUnhandledExceptionFilter. This is a bit unreliable since it requires you to have code running inside a crashed process.
Alternatively, you can just the built-in Windows Error Reporting to collect crash data. This is more reliable, since it doesn't require you to add code running inside the compromised, crashed process. The only cost is to get a code-signing certificate, since you must submit a signed binary to the service. https://sysdev.microsoft.com/en-US/Hardware/signup/ has more details.
You can use the Windows API call MiniDumpWriteDump if you wish to roll your own code. Both Windows XP and Vist automate this process and you can sign up at https://winqual.microsoft.com to gain access to the error reports.
Also check out http://kb.mozillazine.org/Breakpad and http://www.codeproject.com/KB/debug/crash_report.aspx for other solutions.
This website provides quite a detailed overview of stack retrieval on Win32 after a C++ exception:
http://www.eptacom.net/pubblicazioni/pub_eng/except.html
Of course, this will only work from within the process, so if the process gets terminated or crashes to the point where it terminates before that code is run, it won't work.
Generate a minidump file. You can then load it up in windbg or Visual Studio and inspect the entire stack where the crash occurred.
Here's a good place to start reading.
Its quite simple to dump the current stackframe addresses into a log file. All you have to do is get such a function called on program faults (i.e. a interrupt handler in Windows) or asserts. This can be done at released versions as well. The log file then can be matched with a map file resulting in a call stack with function names.
I published a article about this some years ago.
See http://www.ddj.com/architect/185300443
Let me describe how I handle crashes in my C++/WTL application.
First, in the main function, I call _set_se_translator, and pass in a function that will throw a C++ exception instead of using structured windows exceptions. This function gets an error code, for which you can get a Windows error message via FormatMessage, and a PEXCEPTION_POINTERS argument, which you can use to write a minidump (code here). You can also check the exception code for certain "meltdown" errors that you should just bail from, like EXCEPTION_NONCONTINUABLE_EXCEPTION or EXCEPTION_STACK_OVERFLOW :) (If it's recoverable, I prompt the user to email me this minidump file.)
The minidump file itself can be opened in Visual Studio like a normal project, and providing you've created a .pdb file for your executable, you can run the project and it'll jump to the exact location of the crash, together with the call stack and registers, which can be examined from the debugger.
If you want to grab a callstack (plus other good info) for a runtime crash, on a release build even on site, then you need to set up Dr Watson (run DrWtsn32.exe). If you check the 'generate crash dumps' option, when an app crashes, it'll write a mini dump file to the path specified (called user.dmp).
You can take this, combine it with the symbols you created when you built your server (set this in your compiler/linker to generate pdb files - keep these safe at home, you use them to match the dump so they can work out the source where the crash occurred)
Get yourself windbg, open it and use the menu option to 'load crash dump'. Once it's loaded everything you can type '~#kp' to get a callstack for every thread (or click the button at the top for the current thread).
There's good articles to know how to do this all over the web, This one is my favourite, and you'll want to read this to get an understanding of how to helpyourself manage the symbols really easily.
You will have to set up a dump generation framework in your application, here is how you may do it.
You may then upload the dump file to the server for further analysis using dump analyzers like windbg.
You may want to use adplus to capture the crash callstack.
You can download and install Debugging tools for Windows.
Usage of adplus is mentioned here:
Adplus usage
This creates the complete crash or hang dump. Once you have the dump, Windbg comes to the rescue. Map the correct pdbs and symbols and you are all set to analyze the dump. To start with use the command "!analyze -v"
we have a problem with an application we're developing. Very seldom, like once in a hundred, the application crashes at start up. When the crash happens it brings down the whole system, the computer starts to beep and freezes up completely, the only way to recover is to turn off the power (we're using Windows XP). The rarity of the crash combined with the fact that we can't break into the debugger or even generate a stackdump when it occurs makes it extremely hard to debug.
I'm looking for something that logs all function calls to a file. Does such a tool exist? It shouldn't be impossible to implement, profilers like VTune does something very similar.
We're using visual studio 2008 (C++).
Thanks
A.B.
Logging function entries/exits is a low-level approach to your problem. I would suggest using automatic debugger instrumentation (using Debugger key under Image File Execution Options with regedit or using gflags from the package I provide a link to below) and trying to repro the problem until it crashes. Additionally, you can have the debugger log function call history of suspected module(s) using a script or have collect any other information.
But not knowing the details of your application it is very hard to suggest a solution. Is it a user app, service or a driver? What does "crashes at startup" mean - at windows startup or app's startup?
Use this debugger package to troubleshoot.
The only problem with the logging idea is that when the system crashes, the latest log entries might still be in the cache and have no chance to be written to disk...
If it was me I would try running the program on a different PC - it might be flaky hardware or drivers causing the problem. An application program "shouldn't" be able to bring down the system.
A few Ideas-
There is a good chance that just prior to your crash there is some sort of exception in the application. if you set you handler for all unhandled exceptions using SetUnhandledExceptionFilter() and write a stack trace to your log file, you might have a chance to catch the crash in action.
Just remember to flush the file after every write.
Another option is to use a tool such as strace which logs all of the system calls into the kernel (there are multiple flavors and implementations for that so pick your favorite). if you look at the log just before the crash you might find the culprit
Have you considered using a second machine as a remote debugger (via the network)? When the application (and system) crashes, the second machine should still show some useful information, if not the actual point of the problem. I believe VC++ has that ability, at least in some versions.
For Visual C++ _penter() and _pexit() can be used to instrument your code.
See also Method Call Interception in C++.
GCC (including the version MingGW for Windows development) has a code generation switch called -finstrument-functions that tells the compiler to emit special calls to functions called __cyg_profile_func_enter and __cyg_profile_func_exit around every function call. For Visual C++, there are similar options called /GH and /Gh. These cause the compiler to emit calls to __penter and __pexit around function calls.
These instrumentation modes can be used to implement a logging system, with you implementing the calls that the compiler generates to output to your local filesystem or to another computer on your network.
If possible, I'd also try running your system using valgrind or a similar checking tool. This might catch your problem before it gets out-of-hand.