One of our application (windows form application C++, MSVS 2010 )crashes after few minutes of usage. Task manager tells that the memory usage grows to 60% of total system RAM in just few seconds of the run.
I used Intel inspector to get any idea of memory leaks. I was expecting I will get a a list of functions that are creating problem. But what I got is only the dlls as can be seen in the following screenshot.
The application is using a couple of third party libraries such as those starting with Pv, OpenCv cdio, CAIO etc. As you can see the last one is an opencv library, and is occupying close to 400MB. (How is this possible ? )
Also the right panel shows different types of leaks which have occurred.
I want to pin point the memory leak code and correct it. What should be my strategy, what functions should I start looking into? Why the inspector is not giving me correct source code and just giving me dlls? I am sure dlls are perfect as these are used by millions of people.
Please advice,
Thanks
Update
I think I have done something wrong in various compiler setting while generating the exe. .
As can be seen above, no symbol information is loaded. That was the reason I was unable to get the source code where memory leaks were happening. Pressing F1 reaches me to the following instructions:
Troubleshooting No Symbolic Information Symptoms
In the Sources window, the Intel Inspector displays no source code for any code locations in the problem set.
Details Intel Inspector cannot display source code for viewing and editing unless the application has symbols available.
If the Intel Inspector detects no symbols for a location, it sets the call stack and code pane to the first location where it can find symbols.
If the Intel Inspector cannot find any location in the call stack with symbols, it provides the module name and relative virtual address (RVA) for the location.
Possible Correction Strategies
1- When you compile and link an application on Windows* systems:
a) Enable the debug information compiler option.
b) Set the linker option to generate debug information.
2- Configure the project to search non-standard directories.
3- Copy the symbol files, such as *.pdb files, to a location where the Intel Inspector can find them.
So now I am focusing on the above correction strategies. My latest question are:
1- how do I set the above three strategies in MSVS 2010.
2- Do I need to use debgug exe or a release exe while using Intel inspector ?
If this is your source code, and you are sure your code is causing the leaks, you can use Visual Leak Detector.
You just need very minimal changes in project - I would say just #include<vld.h> (which you can make conditional). It will report all memory leaks on Debug Output window. This differs from VC++ standard leaking staticitics - it shows where memory was allocated.
Probally it couldn't load symbols for some module / modules and thus the information is a bit incorrect. Is symbol file (like opencv_core240.pdb) opencv_core240.dll available? Check it!
Also I would suggest to try another memory leak detectors to compare their results to each other.
In general when using Inspector the recommendation is to use a debug build of the code. Release builds may optimize away some important pieces of code.
You can also enable just debugging symbols in a release build, which is important when using Amplifier and Advisor. You can do this by going to Project -> [project name] properties... -> Linker -> Debugging -> Generate Debug Info -> Yes and Project->[project name] properties...->C/C++->General->Debug Information Format -> Program Database. Even if you are in debug configuration make sure these settings are set correctly, because they may have been accidentally modified.
WRT what you are seeing in the report:
OpenCV (and others) isn't occupying 350MB, rather there has been a leak of that size (which means the last pointer to dynamically allocated memory was overwritten without releasing that memory). Is it possible that you're misusing the library APIs?
Also, you may find it useful to look at the call stack at the location of the leak. You will possibly find the API leading to the memory leak which can help you pinpoint the issue.
Related
I've experienced this with every version of Visual Studio starting from 2012 (2012, 2013, 2015 Preview), on multiple computers and multiple projects, but I haven't figured out how to fix it:
Whenever I'm debugging a 64-bit(?) C++ console program, after a few minutes and seemingly completely randomly (when I'm not clicking or typing anything), the console window for the program spontaneously closes and I can no longer debug or step through the program with Visual Studio. When I press Stop and attempt to restart debugging, I usually get ERROR_NETWORK_UNREACHABLE:
// MessageId: ERROR_NETWORK_UNREACHABLE
// MessageText:
// The network location cannot be reached. For information about network troubleshooting, see Windows Help.
#define ERROR_NETWORK_UNREACHABLE 1231L
If I try to attach to the process manually I get the error:
Unable to attach to the process.
The only fix I've found for this is to restart Visual Studio. I can't find any other way to fix it, and I've tried running Process Monitor but haven't found anything.
What causes this problem and how can I fix it?
(?) Upon further checking it seems that this only happens in 64-bit mode, but I'm not 100% sure.
Ok, this is just so wrong
I also have issues with this bug, and in my case it occurred every other debug session. Which meant debug -> stop -> debug -> bug -> restart visual studio -> go to start (repeat every minute during the whole day).
Needless to say I was driven to find a solution. So yesterday I tried procmon, spend hours looking at API monitor differences, looked at plugins, netstat, etc, etc, etc. And found nothing. I gave up.
Today
Until today.
To track down a stupid bug in my program today, I launched appverifier. For my application, I ran the 'basics' tests and clicked save. After a few hours this led me to the bug in my program, which was something like this (extremely simplified version):
void* dst = _aligned_malloc(4096, 32);
memcpy(dst, src, 8192);
Obviously this is a bug and obviously it needed fixing. I noticed the error after putting a breakpoint on the memcpy line, which was not executed.
After a stop and 'debug' again I was surprised to find that I could actually debug the program for the second time. And now, several hours later, this annoying bug here hasn't re-emerged.
So what appears to be going on
So... apparently data from my program is bleeding through into the data or execution space of the debugger, which in turn appears to generate the bug.
I see you thinking: No, this shouldn't happen... you're right; but apparently it does.
So how to fix it? Basically fixing your program (more particular: heap corruption issues) seems to make the VS debugger bug go away. Using appverifier.exe (It's in Debugging tools for Windows) will give you a head start.
Why this works
Since VS2012, VC++ uses a different way to manage the heap. Hans Passant explains it here: Does msvcrt uses a different heap for allocations since (vs2012/2010/2013) .
What basically happens is that heap corruption will break your debugger. The AppVerifier basic settings will ensure that a breakpoint is triggered just before the application does something to corrupt the heap.
So, what happens now is that before the process will break the heap, a breakpoint will trigger instead, which usually means you will terminate the process. The net effect is that the heap will still be in-tact before you terminate your program, which means that your debugger will still function.
"Test"
Before using appverifier -- bug triggered every 2 minutes
While using appverifier -- VS debugger has been stable for 5 days (and counting)
This is an environmental problem of course. Always hard to troubleshoot, SysInternals' utilities like Process Monitor and Process Explorer are your primary weapons of choice. Some non-intuitive ways that a network error can be generated while debugging:
Starting with VS2012, the C runtime library had a pretty drastic modification that can cause very hard to diagnose mis-behavior if your program corrupts the heap. Much like #atlaste describes. Since time memorial, the CRT always created its own heap, underlying call was HeapCreate(). No more, it now uses GetProcessHeap(). This is very convenient, much easier now to deal with DLLs that were built with /MT. But with a pretty sharp edge, you can now easily corrupt the memory owned by Microsoft code. Not strongly indicated if you can't reattach a 64-bit program, you'd have to kill msvsmon.exe to clear up the corruption.
The Microsoft Symbol Server supplies PDBs for Microsoft executables. They normally have their source+line-number info stripped, but not all of them. Notably not for the CRT for example. Those PDBs were built on a build server owned by DevDiv in Redmond that had the source code on the F: drive. A few around that were built from the E: drive, Patterns+Practices uses that (unlikely in a C++ program). Your debugger will go look there to try to find source code. That usually ends well, it gives up quickly, but not if your machine uses those drive letters as well. Diagnose by clearing the symbol cache and disabling the symbol server with Tools + Options, Debugging, Symbols.
The winapi suffers from two nasty viral infections it inherited from another OS that add global state to any process. The PATH environmental variable and the default working directory. Use Control Panel + System + Advanced + Environment to have a look at PATH, copy/paste the content of the intentionally small textboxes into a text editor. Make sure it is squeaky clean, some paralysis at the usual mess is normal btw. Take no prisoners. Having trouble with the default directory is much harder to troubleshoot. Both should pop out when you use Process Monitor.
No slamdunk explanations, it is a tough problem, but dark corners you can look in.
I have the same problem. Thought it was related to 64 bit console apps, where it is very easily triggered with almost any debug session. But, it also happens on 64 bit windows apps too. Now I am seeing it on 32 bit windows apps. I am running Windows 8.1 pro on a single desktop with the latest version of vs 2013 and no remote debugging. My (added) extensions are Visual Assist, Advanced Installer, ClangFormat, Code Alignment, Code Compare, Duplicate Selection, Productivity Power Tools 2013, and Visual SVN.
I discovered that the "Visual Studio 2013\Settings\CurrentSettings.vssettings" file gets corrupted. You can delete this file and recreate it by restarting VS or you can try to edit the XML. I then keep a copy of a good settings file that I use to replace when it gets corrupted again.
In my case, the corrupted line begins with
</ToolsOptionsSubCategory><ToolsOptionsSubCategory name="XAML" RegisteredName="XAML"
... and it is extremely long (I think this is why it is prone to corruption).
I just disabled in the Menu
Tools > Options
Debugging > Edit and Continue
Native-only options > Enable native Edit and Continue
and now it does not give the that stupid error which was preventing the starting of the debuggee application.
I also had the same problem with VS2015. It was so frustrating that a simple Hello World program gave this error when I ran debugger for the second time. Tried uninstall and reinstall and didn't work.
Finally, the solution mentioned in https://social.msdn.microsoft.com/Forums/vstudio/en-US/8dce0952-234f-4c18-a71a-1d614b44f256/visual-studios-2012-cannot-findlaunch-project-exe?forum=vsdebug
worked. Reset all visual studio settings using Tools->Import and Export settings. Now the issue is not occurring.
We have a problem for which I am looking for a direction. We have an old MFC based application, and due to various reasons, the application is crashing sometimes intermittently in some weird scenarios. And even the customers who are using our application and getting these crashes are finding difficulty in identifying the pattern of crash. So, I had a thought that, if we can have a mechanism by which we can generate a log whenever the application crashes. Like for example, the call stack or any other information in that log. I know,, we can use the crash dump in this case, but then I feel like having a log is a better option. So any help or information in this regard would be really helpful.
Thank you.
You can find a good implementation of crash reporter in the link here.
When you compile your release build, make sure that both DEBUG and /MAP are enabled. Store your binaries together with your .map files and let your customer run this version until a crash is produced. In the Event Viewer you will then find a crash log with a crash offset. Then debug step into your code (F10) and use the crash offset together with some nifty tricks and tricks to jump (set the EIP register to... well, you have to google this a bit) to the location where the crash occurred. You should now be able to find the error!
Alright, I presented this question on the MSDN forums but have yet to receive any kind of response so I figured I'd give StackOverflow a try.
I'm currently developing a DirectX application using VS2008 on Win7. I recently experienced a nasty memory corruption bug with a memory allocation class that grabbed byte aligned memory. During this bug I could still run the debug and release executables however it would crash due the instructions getting corrupted or whatever, but it would still execute for a bit until the crash.
I then stripped out the entire memory allocation class. The application runs perfectly in the IDE (release and debug builds) but I can't run any of the executables at all. They immediately crash with a non-responding/stop working error. And I don't think it is my environment because I get the same issue on another computer that wasn't having problems before either.
Dependency walker gives a "Warning: At least one delay-load dependency module was not found. Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module." error and indicates that GPSVC.dll and IESHIMS.DLL can't be found. I've read that this can be misleading and just indicates a potential problem somewhere. And Dependency walker wasn't giving me this error the day before.
I haven't tinkered with any of the configuration or project settings or added new code. Any idea of what could be causing this behavior?
Also another note, I installed the Windows 7.1 sdk the same day. Think this could be some kind of compiler related bug?
Just in case some useful information pops up on the MSDN post, here is the link
http://social.msdn.microsoft.com/Forums/en-IE/vsdebug/thread/f692b394-8af2-4453-991c-aa6a443a9019
Thanks!
Edit -
Here is the last couple lines of Dependency Walker's profiling output
GetProcAddress(0x76CD0000 [c:\windows\syswow64\KERNEL32.DLL], "DecodePointer") called from "c:\windows\syswow64\NVWGF2UM.DLL" at address 0x6D8BAE4F and returned 0x77B59D65.
GetProcAddress(0x76CD0000 [c:\windows\syswow64\KERNEL32.DLL], "DecodePointer") called from "c:\windows\syswow64\NVWGF2UM.DLL" at address 0x6D8BAE4F and returned 0x77B59D65.
GetProcAddress(0x76CD0000 [c:\windows\syswow64\KERNEL32.DLL], "EncodePointer") called from "c:\windows\syswow64\NVWGF2UM.DLL" at address 0x6D8BAF60 and returned 0x77B60FDB.
GetProcAddress(0x76CD0000 [c:\windows\syswow64\KERNEL32.DLL], "DecodePointer") called from "c:\windows\syswow64\NVWGF2UM.DLL" at address 0x6D8BAF70 and returned 0x77B59D65.
Second chance exception 0xC0000005 (Access Violation) occurred in "c:\users\joel\desktop\DXAPP.EXE" at address 0x0110152E.
Exited "c:\users\joel\desktop\DXAPP.EXE" (process 0x27D8) with code 255 (0xFF).
Is this referring to a DLL grabbing a null pointer or to my actual instructions? Going to read up on how to use WinDbg real quick and I'll post it's output if this doesn't shed any immediate light on the issue.
Edit 2 -
Simply running the application and hitting debug to bring up Visual Studio consistently brought me to where I'm compiling my shaders. I'm assuming at the moment that the root of the problem lies around this. However, I still don't understand the change of behavior during execution between using the IDE and not.
Solution! -
I was so thrown off by the previous memory corruption bug that I didn't realize my shaders weren't in a local directory to the executables. This in turn was generating a null pointer that wasn't handled properly after calling D3DX11CompileFromFile().
Shoot, sorry I meant to post this as a comment...
I can only suggest more diagnostic attempts.
One would be to profile the app from within Depends, this will also show dynamic DLL loads and might show something new. Also it captures the debug output. It may behave differently than launching in the debugger itself and provide a clue. You don't mention actually profiling so I thought I'd suggest it in case you hadn't. Also, pay very close attention to the paths for the DLL's loaded - you might be surprised at a DLL loading from a location other than you intended.
Another suggestion is to try at attach to the stopped app after the crash (before dismissing the error dialog). See if you can get a stack trace or anything out of it.
Finally try attaching (or even launching from) WindDbg rather than the IDE. Like the Depends profile, the difference in debugger behavior and how it hooks the app may allow the crash to happen, while providing the clues you need.
Good Luck!
I tried to catch the stack trace of an address, but it always show me nothing,
my issue is analyse memory leak, so i have queried all heap statics and then queried highest allocated memory, it returned me lot of allocation,
and i tried to search many of allocation address by “!heap -p –a ####“, #### is adress.
but it never return me any call stack,
but if I search this address/any other adresses in memory, it shows me similar allocation.
also if i try to run this command “dt ntdll!_DPH_HEAP_BLOCK StackTrace ####,” , it return me NULL stack trace.
is it because of page heap for application is not enabled????
For native memory issues, you cannot easily use dumps to determine the root cause.
Microsoft has a tool DebugDiag for 32 bit processes,
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en
You can use it to track native memory leaks in some typical scenarios,
http://support.microsoft.com/kb/919790
Besides, involving Microsoft support team can speed up the root cause analysis,
http://support.microsoft.com
There are a number of things here, firstly you will not get a complete stack trace for symbols you do not have access to, for this you need the pdbs without the private symbols stripped.
For the microsoft symbols entering '.symfix;.reload' in WinDbg should fix that problem.
Secondly although you will not get a stack trace for 3rd party dlls you will for your own application dlls, you will need to make sure your pdbs with public symbols match the version used to generate the dump.
You could ask the customer to generate the user stack trace in gflags and reproduce the error or to send you the pdbs with full symbol information.
It is also possible to open the dump file in Visual studio:
File->Open->Project/Solution navigate to dump file and click ‘OK’
In Solution Explorer right click project->Debug->Start new instance
You may find this approach more familiar but you will not have access to WinDbg tools but as this is a post mortem analysis you can open the dump in visual studio and WinDbg as you are not invasively debugging anyway.
I have a native C++ program which runs more than 20 times slower when started with Debug (F5), but runs at normal speed when using start without debug (Ctrl + F5).
It does not matter whether I use a debug or release build. Also if I use WinDbg the program is a magnitude slower.
Is there some setting I did choose wrong or something?
Set the _NO_DEBUG_HEAP environment variable to 1 (as seen on http://preshing.com/20110717/the-windows-heap-is-slow-when-launched-from-the-debugger).
This can be done from inside Visual Studio, too.
Now this is just a workaround, and I'm curious to know how to refactor a program which suffers from this kind of problem. Do you have many std::map's, shared_ptr, or any other big indirections by any chance?
This is of course not caused by having the _DEBUG symbol defined or compiling the code in the debug configuration. The added debugging code runs whether or not the debugger is attached to the program.
The debugger doesn't normally affect code execution, it stays out of the way by calling WaitForDebugEvent. Which blocks it, until the operating system tells it that something noteworthy happened. That can trigger a bunch of code in the debugger that can slow down your program. You can see the events listed in the DEBUG_EVENT structure documentation.
Annotating them a bit beyond the documentation: the debugger steps in and can slow down your program when:
The program loads or unloads a DLL. Lots of stuff happens during load, the debugger goes hunting for a debug symbol file (.pdb). It may contact a symbol server to download it. Any breakpoints that were set in the DLL source code will get activated. This can be quite slow, but the effect is temporary and generally only slows down the startup. You can see the load/unload notification in the Output window.
The program raises an exception. This activates the debugger at the moment the exception is raised, a "first chance notification". Which can be very helpful, you can use the Debug + Exception, Thrown checkbox to make the debugger stop when the exception is raised. You can see the notification message in the Output window. This does slow down code that raises and catches exceptions tremendously and is quite likely the source of your slowdown. Never use exceptions for flow control.
A thread starts running or terminates. Again, a notification message is printed to the Output window. You'd have to create a lot of threads to make this slow down your program.
When your program uses OutputDebugString() for tracing purposes. Visible in the Output window. Another good candidate for a slow down, output falls in the bit bucket if no debugger is attached. You shouldn't have any trouble diagnosing this as the cause, the obvious side-effect is seeing a lot of messages in the Output window.
When the program hits a breakpoint. Not a lot of reasons to be stumped by that one. But you can set breakpoints that slow down the program a lot yet don't cause a debugger break. Particularly the Conditional breakpoint, Hit counter, Filter and the When Hit operation will be slow. Use Debug + Windows + Breakpoints to review the breakpoints that are defined.
When a process is created under the debugger, the operating system by default uses the debug heap. The debug heap does more verification of your memory, especially at de-alloc.
There are several possible options in order to disable the use of the Debug Heap:
Attach to the process soon after startup. This will allow you to speed up performance in debug mode knowingly so and being fully aware of the mode in which you are running.
Add the environment variable setting _NO_DEBUG_HEAP=1.
This can be set either globally for the machine or for a specific instance of Visual Studio.
a. Globally you would set an environment variable through the Control Panel → System → Advanced system settings → Environment Variables, and there add the variable _NO_DEBUG_HEAP=1.
Note: This will have an affect on EVERY application you debug.
b. For an instance of Visual Studio you can open a command prompt, setting the environment variable _NO_DEBUG_HEAP=1 and then open visual studio from inside that command prompt. This will influence only the processes created from
that instance of Visual Studio will inherit the environment
variable.
Append the behavior of the debugger, this is possible for VS2015. There are 2 ways to override this:
a. To modify for a specific project, go to the project properties Configuration Properties → Debugging and change the Environment property _NO_DEBUG_HEAP to 1
b. To modify for every project in Visual Studio, go to Tools → Options → Debugging and check the option: “Enable Windows debug heap allocator (Native only)”.
Note: f the _NO_DEBUG_HEAP environment variable mentioned in the 'a' is set in a project level it will override this global setting.
For me the difference in performance between debug mode and release mode is about a factor 40. After some digging, it appears several things contribute to the difference in performance, but there is one compiler option that quadruples my debug performance almost for free.
Namely, changing /ZI into /Zi. For a description, see the MSDN page.
I don't use the edit and continue feature anyway.
No one has mentioned closing unused source windows.
After closing 20+ unused windows, debug source stepping went from ~5s back down to ~.2s. This unusually slow project loads a DLL dynamically and that DLL was also the one being stepped through (and having source windows open) so it seems likely related. However this was C# (headline and tags are non-specific).
There are several things that are different if you run the debug build outside the IDE. One is that the IDE takes a while to load symbols, and if you depend on a lot of libraries then that startup time can be significant.
If you are using a symbol server (including the Microsoft public symbol server) then this may add to the startup time, so ensure you have a local symbol cache in your _NT_SYMBOL_PATH variable if that is the case.
Also the IDE runs with the debug heap enabled, but I don't think this happens if you run outside the IDE.
Debugging Visual C++ comes with lots and lots of overhead, especially in the STL. Try not defining _DEBUG, and define NDEBUG.