We have a large C++ application, which sometimes we need to run as a debug build in order to investigate bugs. The debug build is much much slower than the release build, to the point of being almost unusable.
What tricks are available for making MSVC Debug builds execute faster without sacrificing too much on the debugability?
Use #pragma optimize("", off) at the top of selected files that you want to debug in release. This gives better stack trace/variable view.
Works well if it's only a few files you need to chase the bug in.
Why don't you just switch on debug information in your release configuration?
We turned off Iterator debugging with the preprocessor symbols:
_HAS_ITERATOR_DEBUGGING=0
_SCL_SECURE=0
It helped a bit, but was still not as fast as we'd like. We also ended up making our debug build more release-like by defining NDEBUG instead of _DEBUG. There were a couple other options that we changed too, but I'm not remembering them.
Its unfortunate that we needed to do all this, but our application has a certain amount of work needed to be done every 50ms or its unusable. VS2008 out of the box would give us ~60ms times for debug and ~6ms times for release. With the tweaks mentioned above we could get debug down to ~20ms or so, which is at least usable.
profile the app and see what ti taking the time. you should then be able to see what debugging need to be tuned.
Are you using MFC?
In my experience, the main thing that can make a debug version slow is the class validation routines, which are usually disabled in release. If the data structure is at all tree-like, it can end up re-validating subtrees hundreds of times.
Regardless, if it is, say, 10 times slower than the release build, that means it is spending 1/10 of its time doing what's necessary, and 9/10 doing something else. If, while you're waiting for it, you just hit the "pause" button and look at the call stack, chances are 9/10 that you will see exactly what the problem is.
It's a quick & dirty, but effective way to find performance problems.
Create a ReleaseWithSymbols configuration, that defines NDEBUG and has no optimisations enabled. This will give you better performance while maintaining accurate symbols for debugging.
there are several difference between debug builds and release builds that influence both debugability and speed. The most important are the _DEBUG/NDEBUG define, the compiler optimizations and the creation of debug information.
You might want to create a third Solution Configuration and play around with these settings. For example, adding debug information to a release build doesn't really decrease performance but you already get a sensible stack trace so you know which function you are in. Only the line information is not reliable because of the compiler optimizations.
If you want reliable line information, go on and turn off optimizations. This will slow down the execution a bit but this will still be faster than normal debug as long as the _DEBUG define is not set yet. Then you can do pretty good debugging, only everything that has "#ifdef _DEBUG" or similar around it won't be there (e.g. calls to assert etc.).
Hope this helps,
Jan
Which VS are you using? We moved from VS.net to VS2008 recently and I experienced same slowness while debugging on high end machine on > 500k LOC project. Turns out, Intellisense base got corrupted and would update itself constantly but get stuck somewhere. Deleting .ncb file solved the problem.
Related
Recently, I'm working on intergrating the physics engine into my graphics engine program. Before this, I always build my program in Debug mode because I feel Debug means safe and more information to let me know where is wrong.
In my program, I built Assimp in Release mode, but I still used it in the Debug mode until now. For now, I build Bullet Physics in Release mode beacuse of the performance is huge different in Debug mode. If you want to know how slow it is, you could see this.
The important thing is that I could not use this *.lib file in Debug mode ever, so I have a question, when or why do you change to Release mode from Debug mode or on the other hand. Or using the Release library in Debug mode either? For now, I think I need to change to Release mode permanently for Bullet Physics, and don't know if this is good or bad.
Edit:
I know the benifit for release and debug mode, because there is a lot of possible duplicated articles in stackoverflow, but what I want to know is when you make a program what the decision for choosing or just because encounter a performace problem so need to rebuild the program into release mode like me. I just want to clarify a little different between the posiible duplicated articles :)
There are four cases for this question.
Case 1. Start a new project and programming in Debug mode, because of... .
Case 2. Start a new project and programming in Release mode, because of... .
Case 3. Work in Debug mode a period of time, but change to Release mode because of ... .
Case 4. Work in Release mode a period of time, but change to Debug mode because of ... .
I really want to know what the decisions in the four cases. I'm in the case 3, because of the performace of the Bullet Physics, I decide to make program to be release mode forever, and also make other debug library to rebuild to release mode. In case 3, the question is, I just use the third party library, so do I need to use Debug mode for any reason? Or I just go to release mode?
For case 1&2, Is there any decision why you let the debug mode or release mode to build your new program.
For case 4, I could not think why make a man to change the Release mode to Debug mode, but maybe there is a actual cases? except just build the library you could make this decision.
If my question is not clear enough, please let me know.
"Debug mode" and "Release mode" are two predefined set of compiler switches. You can however override them, and create intermediate forms. C++ commonly benefits a lot from inlining, so a debug-with-inlining build is a reasonable intermediate form. Another common variation is to use the Release version of the C & C++ libraries even in debug builds.
You can even override switches on individual file level. Your bullet physics are likely correct, so you might want to check if they can be compiled with
That said, it's quite common to flip between debug and release mode, when tackling different problems.
Use debug (unoptimized) builds when you need to debug something. They produce (slow) code that matches your source closely so you can more easily debug.
Use release (optimized) builds for code that you ship. They produce code that run a lot faster (orders of magnitude often), but debugging optimized code is really difficult since the generated code is often quite different from the source you wrote.
Mixing objects built as debug/release in the same program is often a recipie for disaster (crashes) (especially with Microsofts compiler); so just don't do that.
I use std::map in my code, but in debug builds code that uses it runs unacceptably slower than in release. This answer says that this is because
In the Microsoft Visual Studio, there's a global lock when accessing
the Standard C++ Library to protect from multi threading issue in
Debug builds.
But it doesn't elaborate on a way to disable that global mutex lock. Is there any way to do it?
If your problem is generally that your debug build is slow using STL, I would try turning off iterator debugging. As far as I know, that's one of the big slow downs between release and debug builds. I believe the relevant #define is:
#define _HAS_ITERATOR_DEBUGGING 0
As far as I know, there isn't because some of the debug checking inside the runtime library is probably (and most likely can't be made) thread safe and thus in order to give you the additional checks, you have to have the global lock.
From experience I know the debug runtime is considerably slower but in a lot of cases, the additional runtime checks are valuable. I would also point a profiler at the code before blaming the runtime - your debug code is running unoptimized unless you tweaked the flags and as a result any performance problem in the code with be magnified because the compiler did what you told it to do :).
That said, one option that you might be able to use to work around this if you are willing to forego the additional runtime checks would be to try and build your debug version against the release runtime. I haven't tried that in quite a while but the last time I tried this did work and it might make an unacceptably slow piece of code more bearable.
I got a program that has a quite simple 'load up' function that takes about 30 seconds (3M triangles that goes into std containers).
It works perfectly well.
The rest doesn't always (it is not finished) so I debug a lot I make a lot of changes and so on which means restarting quite often.
Is there any secret technique to compile the loader in release (which speeds up everything enormously) and leaving the rest as debug?
ps. I use MSVC 2005
Debug builds tend to be very slow on Visual C++. There are a few reasons for this:
the most obvious is that the code is not optimized
the memory allocation functions in the debug CRT library perform additional checks to detect heap corruption and other issues, so they are slower.
many STL functions perform additional validations and assertions on debug builds, which make use of STL containers very slow
I've had success debugging apps that make heavy use of memory and STL using the following method:
use the release build for debugging (yes, this works fine!).
configure your release builds to include debugging symbols, this should make the compiler write the .pdb file that the debugger uses
only for the files you intend to debug, set the compiler to no optimizations.
Note that the above works great to debug problems in your own logic, but it may not be the best idea if you are debugging memory corruption or other problems, since you are eliminating all the extra debug code that the CRT provides for these types of issues.
I hope this helps!
Mixing debug and release builds tends to go horribly wrong.
But there's no reason why you shouldn't turn on optimisation for some selected source files even in a debug build - and the optimisation should give you the performance improvements.
I'm compiling a very small Win32 command-line application in VS2010 Release-Mode, with all speed optimizations turned on (not memory optimizations).
This application is designed to serve a single purpose - to perform a single pre-defined complex mathematical operation to find a complex solution to a specific problem. The algorithm is completely functional (confirmed) and it compiles and runs fine in Debug-Mode. However, when I compile in Release-Mode (the algorithm is large enough to make use of the optimizations), Link.exe appears to run endlessly, and the code never finishes linking. It sits at 100% CPU usage, with no changes in memory usage (43,232 K).
My application contains only two classes, both of which are pretty short code files. However, the algorithm comprises 20 or so nested loops with inline function calls from within each layer. Is the linker attempting to run though every possible path through these loops? And if so, why is the Debug-Mode linker not having any problems?
This is a tiny command-line application (2KB exe file), and compiling shouldn't take more than a couple minutes. I've waited 30 minutes so far, with no change. I'm thinking about letting it link overnight, but if it really is trying to run through all possible code paths in the algorithm it could end up linking for decades without a SuperComputer handy.
What do I need to do to get the linker out of this endless cycle? Is it possible for such code to create an infinite link-loop without getting a compiler-error prior to the link cycle?
EDIT:
Jerry Coffin pointed out that I should kill the linker and attempt again. I forgot to mention this in the original post, but I have aborted the build, closed and re-opened VS, and attempted to build multiple times. The issue is consistent, but I haven't changed any linker options as of yet.
EDIT2:
I also neglected to mention the fact that I deleted the "Debug" and "Release" folders and re-built from scratch. Same results.
EDIT3:
I just confirmed that turning off function inlining causes the linker to function normally. The problem is I need function inlining, as this is a very performance-sensitive operataion with a minimal memory footprint. This leads me to ask, why would inlining cause such a problem to occur?
EDIT4:
The output that displays during the unending link cycle:
Link:
Generating code
EDIT5:
I confirmed that placing all the code into a single CPP file did not resolve the issue.
Nested loops only affect the linker in terms of Link Time Code Generation. There's tons of options that determine how this works in detail.
For a start I suggest disabling LTCG alltogether to see if there's some other unusual problem.
If it links fine in Release with LTCG disabled you can experiment with inlining limits, intrinsics and optimization level.
Have you done a rebuild? If one of the object files got corrupted, or a .ilk file (though release mode probably isn't using one, I can't be sure about your project settings), then the cleaning pass which rebuild does should cure it.
Closing and re-opening Visual Studio is only helpful in cases when the debugging (or one of the graphical designers) is keeping an open handle to the build product.
Which leads to another suggestion -- check Task Manager and make sure that you don't have a copy of your application still running.
If the linker seems to be the bottle neck and both classes are pretty short, why not put all the code in a single file? Also, there is a multi-processor compilation option in visual studio under C++ general tab in the project settings dialog box. Both of these might help.
Some selected answers:
Debug builds don't inline. Not at all. This makes it possible to put breakpoints in all functions. It wou
It doesn't really matter whether the linker truly runs an infinite loop, or whether it is evaluating 2^20 different combinations of inline/don't inline. The latter still could take 2^20 seconds. (We know the linker is inlining from the "Generating code" message).
Why are you using LTCG anyway? For such a small project, it's entirely feasible to put everything in one .cpp file. Currently, the compiler is just parsing C++, and not generating any code. That also explains why it doesn't have any problem: there's only a single loop in the compiler, over all lines of source code.
Here in my company we discovered something similar.
As far as I know it is no endless loop, just a very long operation.
This means you have to options:
turn off the optimizations
wait until the linker has finished
You have to decide if you really need these optimizations. Here the program is just a little helper, where it does not matter if the program is running 23.4 seconds or 26.8. The gain of program speed compared to the build speed are making the optimizations nearly useless. At least in our special scenario.
I encountered exactly this problem trying to build OSMesa x64 Release mode. My contribution to this discussion is that after letting the linker run overnight, it did complete properly.
I've seen posts talk about what might cause differences between Debug and Release builds, but I don't think anybody has addressed from a development standpoint what is the most efficient way to solve the problem.
The first thing I do when a bug appears in the Release build but not in Debug is I run my program through valgrind in hopes of a better analysis. If that reveals nothing, -- and this has happened to me before -- then I try various inputs in hopes of getting the bug to surface also in the Debug build. If that fails, then I would try to track changes to find the most recent version for which the two builds diverge in behavior. And finally I guess I would resort to print statements.
Are there any best software engineering practices for efficiently debugging when the Debug and Release builds differ? Also, what tools are there that operate at a more fundamental level than valgrind to help debug these cases?
EDIT: I notice a lot of responses suggesting some general good practices such as unit testing and regression testing, which I agree are great for finding any bug. However, is there something specifically tailored to this Release vs. Debug problem? For example, is there such a thing as a static analysis tool that says "Hey, this macro or this code or this programming practice is dangerous because it has the potential to cause differences between your Debug/Release builds?"
One other "Best Practice", or rather a combination of two: Have Automated Unit Tests, and Divide and Conquer.
If you have a modular application, and each module has good unit tests, then you may be able to quickly isolate the errant piece.
The very existence of two configurations is a problem from debugging point of view. Proper engineering would be such that the system on the ground and in the air behave the same way, and achieve this by reducing the number of ways by which the system can tell the difference.
Debug and Release builds differ in 3 aspects:
_DEBUG define
optimizations
different version of the standard library
The best way around, the way I often work, is this:
Disable optimizations where performance is not critical. Debugging is more important. Most important is disable function auto-inlining, keep standard stack frame and variable reuse optimizations. These annoy debug the most.
Monitor code for dependence on DEBUG define. Never use debug-only asserts, or any other tools sensitive to DEBUG define.
By default, compile and work /release.
When I come across a bug that only happens in release, the first thing I always look for is use of an uninitialized stack variable in the code that I am working on. On Windows, the debug C runtime will automatically initialise stack variables to a know bit pattern, 0xcdcdcdcd or something. In release, stack variables will contain the value that was last stored at that memory location, which is going to be an unexpected value.
Secondly, I will try to identify what is different between debug and release builds. I look at the compiler optimization settings that the compiler is passed in Debug and Release configurations. You can see this is the last property page of the compiler settings in Visual Studio. I will start with the release config, and change the command line arguments passed to the compiler one item at a time until they match the command line that is used for compiling in debug. After each change I run the program and reproducing the bug. This will often lead me to the particular setting that causes the bug to happen.
A third technique can be to take a function that is misbehaving and disable optimizations around it using the pre-processor. This will allow you run the program in release with the particular function compiled in debug. The behaviour of the program which has been built in this way will help you learn more about the bug.
#pragma optimize( "", off )
void foo() {
return 1;
}
#pragma optimize( "", on )
From experience, the problems are usually stack initialization, memory scrubbing in the memory allocator, or strange #define directives causing the code to be compiled incorrectly.
The most obvious cause is simply the use of #ifdef and #ifndef directives associated DEBUG or similar symbol that change between the two builds.
Before going down the debugging road (which is not my personal idea of fun), I would inspect both command lines and check which flags are passed in one mode and not the other, then grep my code for this flags and check their uses.
One particular issue that comes to mind are macros:
#ifdef _DEBUG_
#define CHECK(CheckSymbol) { if (!(CheckSymbol)) throw CheckException(); }
#else
#define CHECK(CheckSymbol)
#endif
also known as a soft-assert.
Well, if you call it with a function that has side effect, or rely on it to guard a function (contract enforcement) and somehow catches the exception it throws in debug and ignore it... you will see differences in release :)
When debug and release differ it means:
you code depends on the _DEBUG or similar macros (defined when compiling a debug version - no optimizations)
your compiler has an optimization bug (I seen this few times)
You can easily deal with (1) (code modification) but with (2) you will have to isolate the compiler bug. After isolating the bug you do a little "code rewriting" to get the compiler generate correct binary code (I did this a few times - the most difficult part is to isolate the bug).
I can say that when enabling debug information for release version the debugging process works ... (though because of optimizations you might see some "strange" jumps when running).
You will need to have some "black-box" tests for your application - valgrind is a solution in this case. These solutions help you find differences between release and debug (which is very important).
The best solution is to set up something like automated unit testing to thoroughly test all aspects of the application (not just individual components, but real world tests which use the application the same way a regular user would with all of the dependencies). This allows you to know immediately when a release-only bug has been introduced which should give you a good idea of where the problem is.
Good practice to actively monitor and seek out problems beats any tool to help you fix them long after they happen.
However, when you have one of those cases where it's too late: too many builds have gone by, can't reproduce consistently, etc. then I don't know of any one tool for the job. Sometimes fiddling with your release settings can give a bit of insight as to why the bug is occurring: if you can eliminate optimizations which suddenly make the bug go away, that could give you some useful information about it.
Release-only bugs can fall into various categories, but the most common ones (aside from something like a misuse of assertions) is:
1) Uninitialized memory. I use this term over uninitialized variables as a variable may be initialized but still be pointing to memory which hasn't been initialized properly. For this, memory diagnostic tools like Valgrind can help.
2) Timing (ex: race conditions). These can be a nightmare to debug, but there are some multithreading profilers and diagnostic tools which can help. I can't suggest any off the bat, but there's Coverity Integrity Manager as one example.