I got a memory corruption problem in my program. I've figured out there is an class pointer wrote to an incorret place. I want to trace all assignment of this kind of objects, check them one by one.
There are probably half million lines in my codes. I can't do that manually.Are there any tools or compile options will help me?
I'm not sure what platform you're using. But most debuggers have the idea of a watch point, where you can watch specific variables.
For GDB you just type watch foo and run the code
See more here:
http://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.html
How can I use "watch" GDB?
Alternatively, just use valgrind to find your memory corruption, it is very good at it.
Related
First of all, thank you for taking the time to view my question and help. I noticed that a lot of questioners here show little or no appreciation, but I'm sincerely appreciative for the help and the community here :)
I wrote a C++ plugin (compromised of hundreds of source files) for an application I do not have the source code for (it's a video game). In other words, I only have the source code for my plugin, but not the game. Now, somewhere in those thousands of lines in my plugin, something causes the game engine to throw (probably an access violation) and I don't know where. By the time the debugger breaks, the stack is corrupted and all I get are hex addresses for DLLs I do not have the source for (but the exception occurs in my DLL for sure). I tried everything... I just can't seem to find where the exception occurs. Sometimes the debugger points to a "memory relocation" function (which I never used in my plugin), sometimes it points to the engine's GameFrame(), and other times it points to a damage callback (all these are just different member functions of a class).
I tried practically everything... I googled for hours trying to find out how to use other debuggers like WinDbg and Microsoft Application Verifier. I tried to comment out one or the other, or both, where the debugger points, but it still crashes. I even inserted OUTPUT("The name of the last executed function is: %s", __FUNCTION__) into EVERY function in my application hoping to painstakingly catch the last function but it seems any kind of I/O prevents the exception from occurring for some reason... And 10 minutes of debugging and the crash happens at some random last executed function.
I can't find out where this access violation is happening or where some temporary object is removed to cause these bad pointers (I check every pointer before using it), but damn, I'm reaching my limit's end here.
So, how does one debug the impossible... a random crash with a crappy debugger call stack? Thanks in advance for your patient and kind help!
My suggestion: try different debuggers (non MS), they catch different things.
My experience: a program I have source code and full debugging symbols corrupt the stack, VS nor WinDbg can help but Ollydbg comments a non-string var with the value "r for pattern.", so I had overwrote some string buffer onto this var. Also Ollydbg have option to walk the stack the hard way (not using dbghelp.dll)
From my experience, the old adage "Prevention is better than cure" is very relevant. It is best to prevent the bugs from creeping in, by following good software development practices (unit tests, regressions, code review, etc.) than to work it out later once the bugs show up.
Of course, real world is not perfect, and bugs do show up. To debug memory corruption, you have some nice tools like valgrind, which at least narrow down the problem sections for you to take a closer look at. Debugging a complex program is not easy, and if your debugger throws up, it requires a lot of persistence on your part. One technique I find useful is to selectively enable or disable certain modules, to narrow down the module has the problem.
Sometimes you need to use "referential transparency" to unload some modules. To give you a stripped down example, consider:
int foo = factorial(3);
If I suspect there's a problem in this code (and the debugger crashes before I can see the call stack), I have to try by removing this code, and see if the problem persists. However, foo may be used later, so I cannot just remove it. Instead I can replace it with int foo = 6; and continue.
Another important point is to always maintain a trace file, where your code keeps logging what it is doing. When a program crashes, the trace file can often help narrow down the problem. Of course, you disable the tracing by default, so that it doesn't cause a performance bottleneck.
I am learning TDD and using CppUTest in eclipse.
Is there any way to debug my code getting a nagging segmentation fault.
Thanks
I don't know anything special in CppUTest or Eclipse to help you, but some generic segfault debugging ideas seem appropriate here:
Add flushing print statements (e.g. printf(...) + fflush(stdout) or fprintf(stderr, ...)) to your code and see what gets printed. Do this in a binary search fashion with just a few prints at a time until you narrow down exactly where it is crashing. This sounds old fashioned but is extremely effective. Here is a guide I found googling that talks about this well-known technique: http://www.floccinaucinihilipilification.net/blog/2011/3/24/debugging-via-binary-search.html
Compile your code with debugging symbols and run it in a debugger. When you hit your segfault, ask for a backtrace and see if you can figure out what happened. When doing this it can be especially helpful to use a graphical debugger.
Run your code with a debugging tool like a debug malloc library or something from the valgrind suite. This may catch problems that are root causes of your segfaults but aren't occuring at the exact place where the segfault is generated (e.g. double frees, out of bound array access clobbering pointers used later, etc).
It would be helpful if you could add some code to your question, to give us a better idea of what you are up against. Not knowing any of the details, I would suggest the following:
Add -vto your executable's arguments in the Debug dialog. This will print the names of your test cases as they are executed. The last name that prints is likely the test where the segmentation fault occurs.
Put a breakpoint in that test case, where you call your code under test
Step into your code until the segfault occurs.
Trace back the value that caused the segfault (most often, a dangling pointer) and find out, why it was NULL or uninitialized.
This is utterly mystifying me. I have, in my class declaration, two lines:
std::multimap<int, int> commands;
std::multimap<std::string, std::string> config;
The code compiles without issue, but when I run it, I get the following error:
*** glibc detected *** ./antares: free(): invalid pointer: 0xb5ac1b64 ***
Seems simple enough, except that it has nothing to do with how the two variables are later handled. I removed all references in the rest of the code to the variables - still crashed. I commented out one of the lines - either one, and the program ran without issue. How can the error not be with either particular variable? I'm working under the assumption that there isn't a bug in STL, but I've run out of ideas on how my code could possibly be doing this.
This one has me stumped, so I'd appreciate any help you can provide.
Wyatt
EDIT: I'm not suggesting there's a problem with STL, that was just me being a bit glib. I know the bug is in my code, what I want to know is - what could possibly be wrong that declaring an unreferenced variable would cause it to crash? Why would that affect my code at all?
My code is a few thousand lines long, so it's not really worth anyone's time reading through it, I'm just looking for someone to point me in the right direction.
You're correct to assume the problem isn't in GCC or the STL. However, if the maps are causing free errors, your other code is likely stack smashing (or heap smashing). A truly terrible bug to chase down. The worse part about stack smashing is the object that breaks is not the object with the bug.
Here are some debugging tips.
Run the app under valgrind.
define _GLIBCXX_DEBUG to enable stl debugging
add MALLOC_CHECK_=1 as an environment variable. This will give you better malloc error messages. More info here.
On rare occasions I have been able to add a memory watch to the location that will be smashed. But it is rare when you can predict where the smashing will occur.
You are right: the crash is not from these two lines - they just make it visible.
Here's how to diagnose this problem:
first, leave your variables defined (make your program crash)
second, remove or disable other parts of your code until the crash stops happening. Then you will know an approximate area that corrupts your memory.
third (once you have an area that when disabled stops the crash) start enabling parts of it until the crash happens again.
Edit: I'd say your problem is with code that contains your two multimaps (a copy constructor or assignment operator is missing or something like that). It's just a wild guess so don't put much stock on it.
I have been working for the last few weeks trying to track down a really difficult bug that crashes my application. First, the application was crashing on the assign of a std::string, then during the free of a local variable.
After careful inspection of the code, there was no reason for it to crash at these locations; however, it always crashed while trying to free an invalid pointer (i.e. a pointer that pointed to invalid memory). And I have no idea why this pointer was not pointing to the right location.
I suspect that the issue has to do with a memory corruption problem or pointer corruption problem of some sort. The problem is that I can't visually track it down....yet. I have no idea where to start looking in the code, and there are thousands of lines of code to go through so this does not seem like a realistic approach to the problem.
So in comes Valgrind...
A tool that I have depended upon many a time to find issues within the code that may lead to a crash of this type. However, this time it has come up empty handed! I do not see any errors in valgrind when the problem occurs and so hence the reason for me asking this question.
Are there any other applications that can complement valgrind and help find issues in the code that may cause a crash mentioned above?
Thanks!
I assume you're using valgrind's memcheck tool, which is what it is famous for. Since you are using valgrind already you might also try running your program through valgrind --tool=exp-sgcheck (formerly exp-ptrcheck), which is an experimental tool that is designed to catch certain types of errors that memcheck will miss, including access checks for stack and global arrays, and use of pointers that happen to point to a valid object but not the object that was intended. It does this by using a completely different mechanism, essentially tracking each pointer into memory rather than tracking the memory itself, and through use of heuristics.
Be aware that the tool is experimental, but you may find that it catches something significant. Currently it does not yet support OS X or non-Intel processors.
In my experience, coverity and purify have founds such kind of errors than valgrind didn't (in fact all found problems which weren't seen by the others).
But sometimes no tool give an hint and you have to dig more, add instrumentation, play with breakpoints on "modify memory at address", try to simply the testcase which fails and so on to find out the root cause. That's can be very painful.
My experience is that often this sort of problem is caused by a heap overflow. Electric Fence is a relatively simple allocation debugging tool I like to use. Its main use is as a dynamic analysis tool to check for heap overflows, a complement to "-fstack-protector-all" which checks for stack overflows.
More links to efence stuff.
Is it possible some stack corruption is occurring? If so, try enabling stack canaries with the -fstack-protector-all option, assuming you are using g++.
Other than that, have you cranked up warning flags to help identify suspicious code?
In my opinion, using a debugger with "reverse debugging" capabilities could help.
You would be able to step back in time and hopefully find out what was the real source of the problem.
Here are a couple of links:
http://www.gnu.org/software/gdb/news/reversible.html
http://undo-software.com/ (which apparently is free for non-commercial applications)
You didn't specify the platform, but I can recommend Gimpel PC-lint as an excellent static analysis tool (don't be fooled by the name!). They also offer FlexeLint for other platforms, but I have no personal experience of that product.
Have you tried using lint, flexlint or cppcheck. These may help identify a problem.
If you know what area of memory is being corrupted have you tried marking this memory as protected. This may mask your problem and not help at all but if it still crashes the point at where the memory is modified will help resolve your problem.
If valgrind can identify the bad pointer being passed to free(), you could try running the program under DDD, which can set a hardware watchpoing on the memory location and halt the program when it is getting a bad value. If the pointer is getting changed a lot you may have to write some code around malloc and free to keep track of which values are good and bad.
Looking for a quick and dirty way to identify the caller of a constructor (or any function for that matter) I am writing macros to help identify memory leaks by dumping the this pointers to OutputDebugString.
Knowing where ctor and dtor was called from would help identify the problem.
tnx
\0
If you're using visual studio you can attach the debugger and rather than having a break-point have a trace-point. You do this by right clicking the break-point and choosing When Hit.... Then select to print a message including the stack trace. This message will be sent to the output pane and you can analyze all calls at your leisure.
The best way I can think of is to run your program in a debugger and put a breakpoint in the constructor. Next, examine the call stack.
If you want to target one specific allocation in one specific class, you can keep an allocation count and see which allocation number doesn't get freed. Run the program again, and break on the right allocation number.
If you need to have the call stack dumped to a log, I know it is possible to generate a stack dump using for example win32 API. A more general approach would be to keep an explicit call stack as a global/thread specific state, for example in an std::vector<std::string>-object. (Use RAII to ensure that every push_back is accompanied by a pop_back)
It seems to be you are on windows (OutputDebugString). So you can use the StackWalk64 api to get the stacktrace.
See the "Printing the stack trace in C++ (MSVC)" question for more details.
There is also a lot of leak detection tool available (BoundsChecker, etc).
There is no quick and dirty way for this, C++ does not offer any portable way of looking into a stack-trace. If you want to search for memory-leaks, I'd recommend looking into valgrind and similar tools, they do a great job. As coding guideline, avoid memory-leaks in the first place by using RAII (always have an owner for a resource).
Using gcc? Why not generate a stack trace?
If you're using Linux then Valgrind does everything you want and more. I find it indispensable when developing in C++.
If you're using g++, you can build your project for coverage. When you run over some sample code, you can then view the coverage of your program using gcov.
This output includes the call tree, and you should be able to see calls to constructors, and the functions that are calling them.
The only downside I can think of is that you will only get output for code that is actually executed, and so you'll need to have good test cases. That being said, performing coverage analysis is well worth it anyway. Finally, I highly recommend that you use lcov to view the results!
Can you manipulate the ctor and dtor? I'm no C++ developer and you should easily see this, but perhaps in this case you could pass i.e. a reference on the caller to the constructor.
You running under Windows? Visual Leak Detector has helped me in the past find memory leaks.
Using RAII helps reduce memory leaks too.
If you are feeling adventurous then you can overload the new and delete functions. Paul Nettle does this in his MMGR.
The advise to use a debugger and the call stack is sound and probably the best solution possible. However if you are without a debugger it will not be much help.
Do you know the calling convention being used for your constructor? If so you can use some inline assembler (provided your compiler supports it) to examine the order of function calls. With std calling, the most common convention for Win32, popping the stack will reveal the pointer to the address to return to after the function has been called (i.e. some place in the calling function). This isn't ideal, but you can then go backwards from that point until you reach an address you know to be the start of a function. The only problem here is that you need to get the addresses for all of your functions to be able to do this... this can be done using a simple trick to get the value of eip into another register right at the top of the function, then moving this value into an array to be checked against later when debugging, something like (intel syntax):
call label
label:
pop eax
mov [address of next array entry], eax
Basically you don't, instead of you save all allocation/deallocation and discover who don´t free objects/areas.
See this answers
Heap corruption under Win32; how to locate?
Good lock.
Thanks everyone for the feedback. putting a break point in the ctor is not an option due to hundreds of calls for new objects in even a short lifecycle of the program.
Tracing macros in the ctor and dtor did the trick.
Visual Leak Detector and Stackwalk64 look very promising
also found AfxDumpStack(AFX_STACK_DUMP_TARGET_ODS); // OutputDebugString
but it is VERY noisy