I'm struggling to debug a multithreaded c++ program that I am working on. It seems to be crashing at the same spot in the code, but only some of the time due to a variable being deallocated by a different thread, causing the program to crash when the variable is accessed by a pointer after that.
Because of this, there is no way for me to step through the program for debugging. Is there any way to track all instances of a specific variable in gdb, and track what specific actions are performed to that memory location (so I can see who deallocated it, and where)? Or are there any other good ways to debug this?
I'm struggling to debug a multithreaded c++ program that I am working on
Debugging multithreaded programs with GDB (or any other debugger) is hard.
For the specific example you gave: one thread free()ing memory that another thread is still accessing, Valgind and AddressSanitizer will likely yield much better result than GDB.
Related
I have the following situation: My program runs fine, does all I want it to do, tests pass and Valgrind says it's all right. The only issue is the fact that if I pause execution at some point and try to inspect the state of some objects in the debug view of QtCreator (using GDB) some variables become <not-accessible> and, on resuming execution it reaches a segmentation fault.
To be a little more specific, the program is single threaded and this happens while following the pointers in a tree structure. The structure seems to be fine by the output of the tests.
Does anyone know about a possible cause? Maybe I have messed up with the stack in a way that luckily doesn't affect the tests or it may be just an IDE or debugger issue that should not care about? Thanks in advance for any answers.
Does anyone know about a possible cause?
Do you have multiple threads in your program?
When a program behaves differently depending on whether some GDB breakpoints are present or not, in 99.99% of the cases the program has a data race, and mere fact of stopping it at "inopportune" time exposes that fact.
On Linux, you could use Thread Sanitizer to check for data races.
I have been using gdb for most of a decade now and have never seen this particular problem. I upgraded to gdb 7.4 and the problem persists.
I am debugging a Cilk-multithreaded C++ application on RHEL5. Execution ceases at a seg fault. When I ask gdb to print the value of certain variables (which are boost::intrusive_ptr references to templated object instances) gdb will print the proper value, but will also resume execution on all threads for a very short time. I suspect it resumes execution because more of my debug print statements scroll to the terminal (it's not just clearing a buffer---I can keep printing it and it keeps resuming execution). This short spurt of continued execution causes the values of the variables I am tracking to change. This hinders debugging, to say the least.
I suspect that I have a memory leak and the stack is getting corrupted, but I've run valgrind on the code (with different initial conditions) and it shows no memory leaks in the major subsystem that I am debugging, except for a nominal Cilk-internal leak.
When I ask gdb to print the value of certain variables (which are boost::intrusive_ptr references to templated object instances) gdb will print the proper value, but will also resume execution on all threads for a very short time.
The only way that I know of for this to happen is if you have
A python pretty-printer for the type (boost::intrusive_ptr) and
That pretty-printer calls back into the inferior (being debugged) process.
You can disable all pretty-printers by e.g. disable pretty-printer. If that helps, you should probably figure out which exact pretty-printer is doing this, and contact its author.
my program executes exactly as desired when i run the debugger even with no breakpoints
when i run without debugging, i get a debug error
"This application has requested the Runtime to terminate it in an unusual way."
At one point, I call a function that sets a variable called currCode (an integer)
currCode = function();
//this throws debug error
If i add a cout of the variable currCode between this line and the next line, the program works fine with or without the debugger.
currCode = function();
cout << currCode; //this works!
Might try turning off optimization and see if you still get the problem.
There are many likely reasons for errors appearing in a program run directly from executable and one run by the debugger. Here are a few common ones:
Uninitialized Variables
DLL Hell
Timing
Heap or Stack Management
Again, the above are the most common.
Uninitialized Variables
Many debuggers will inadvertantly initialize your variables for you. A program run directly from an executable may not initialize the variable area, the way you expect. In the embedded systems world, this usually means not at all. So, get in the habit of initializing all variables, preferably when you declare them.
DLL Hell
Debuggers are nice and want to provide you with a nice experience, so they load a number of shared or dynamically linked libraries before your program is executed. Some of these libraries you will have to explicitly load.
Timing
Usually not common, but programs executing without a debugger run at different rates than those run a full speed with a debugger. This can make delay loops (spin loops) be have differently. Data buffers can have longer time to fill when using a debugger. When in doubt, use print statements in the release version to help narrow the location of the issue.
Heap or Stack Management
Debuggers usually provide code to protect your program from overrunning the stack, heap and other areas of memory. This comes with the feature to detect wild pointers and accessing data from invalid addresses. Also, debuggers want to protect what little memory the OS gives to them (they have to share memory with your program). A program running without a debugger can mess up stacks and heaps without any detections or generating faults.
In my c++ program I'm using a library which will "send?" a Sigtrap on a certain operations when
I'm debugging it (using gdb as a debugger). I can then choose whether I wish to Continue or Stop the program. If I choose to continue the program works as expected, but setting custom breakpoints after a Sigtrap has been caught causes the debugger/program to crash.
So here are my questions:
What causes such a Sigtrap? Is it a leftover line of code that can be removed, or is it caused by the debugger when he "finds something he doesn't like" ?
Is a sigtrap, generally speaking, a bad thing, and if so, why does the program run flawlessly when I compile a Release and not a Debug Version?
What does a Sigtrap indicate?
This is a more general approach to a question I posted yesterday Boost Filesystem: recursive_directory_iterator constructor causes SIGTRAPS and debug problems.
I think my question was far to specific, and I don't want you to solve my problem but help me (and hopefully others) to understand the background.
Thanks a lot.
With processors that support instruction breakpoints or data watchpoints, the debugger will ask the CPU to watch for instruction accesses to a specific address, or data reads/writes to a specific address, and then run full-speed.
When the processor detects the event, it will trap into the kernel, and the kernel will send SIGTRAP to the process being debugged. Normally, SIGTRAP would kill the process, but because it is being debugged, the debugger will be notified of the signal and handle it, mostly by letting you inspect the state of the process before continuing execution.
With processors that don't support breakpoints or watchpoints, the entire debugging environment is probably done through code interpretation and memory emulation, which is immensely slower. (I imagine clever tricks could be done by setting pagetable flags to forbid reading or writing, whichever needs to be trapped, and letting the kernel fix up the pagetables, signaling the debugger, and then restricting the page flags again. This could probably support near-arbitrary number of watchpoints and breakpoints, and run only marginally slower for cases when the watchpoint or breakpoint aren't frequently accessed.)
The question I placed into the comment field looks apropos here, only because Windows isn't actually sending a SIGTRAP, but rather signaling a breakpoint in its own native way. I assume when you're debugging programs, that debug versions of system libraries are used, and ensure that memory accesses appear to make sense. You might have a bug in your program that is papered-over at runtime, but may in fact be causing further problems elsewhere.
I haven't done development on Windows, but perhaps you could get further details by looking through your Windows Event Log?
While working in Eclipse with minGW/gcc compiler, I realized it's reacting very bad with vectors in my code, resulting to an unclear SIGTRAP signal and sometimes even showing abnormal debugger behavior (i.e. jumping somewhere up in the code and continuing execution of the code in reverse order!).
I have copied the files from my project into the VisualStudio and resolved the issues, then copied the changes back to eclipse and voila, worked like a charm. The reasons were like vector initialization differences with reserve() and resize() functions, or trying to access elements out of the bounds of the vector array.
Hope this will help someone else.
I received a SIGTRAP from my debugger and found out that the cause was due to a missing return value.
string getName() { printf("Name!");};
I've stumbled onto a very interesting issue where a function (has to deal with the Windows clipboard) in my app only works properly when a breakpoint is hit inside the function. This got me wondering, what exactly does the debugger do (VS2008, C++) when it hits a breakpoint?
Without directly answering your question (since I suspect the debugger's internal workings may not really be the problem), I'll offer two possible reasons this might occur that I've seen before:
First, your program does pause when it hits a breakpoint, and often that delay is enough time for something to happen (perhaps in another thread or another process) that has to happen before your function will work. One easy way to verify this is to add a pause for a few seconds beforehand and run the program normally. If that works, you'll have to look for a more reliable way of finding the problem.
Second, Visual Studio has historically (I'm not certain about 2008) over-allocated memory when running in debug mode. So, for example, if you have an array of int[10] allocated, it should, by rights, get 40 bytes of memory, but Visual Studio might give it 44 or more, presumably in case you have an out-of-bounds error. Of course, if you DO have an out-of-bounds error, this over-allocation might make it appear to be working anyway.
Typically, for software breakpoints, the debugger places an interrupt instruction at the location you set the breakpoint at. This transfers control of the program to the debugger's interrupt handler, and from there you're in a world where the debugger can decide what to do (present you with a command prompt, print the stack and continue, what have you.)
On a related note, "This works in the debugger but not when I run without a breakpoint" suggests to me that you have a race condition. So if your app is multithreaded, consider examining your locking discipline.
It might be a timing / thread synchronization issue. Do you do any multimedia or multithreading stuff in your program?
The reason your app only works properly when a breakpoint is hit might be that you have some watches with side effects still in your watch list from previous debugging sessions. When you hit the break point, the watch is executed and your program behaves differently.
http://en.wikipedia.org/wiki/Debugger
A debugger essentially allows you to step through your source code and examine how the code is working. If you set a breakpoint, and run in debug mode, your code will pause at that break point and allow you to step into the code. This has some distinct advantages. First, you can see what the status of your variables are in memory. Second, it allows you to make sure your code is doing what you expect it to do without having to do a whole ton of print statements. And, third, it let's you make sure the logic is working the way you expect it to work.
Edit: A debugger is one of the more valuable tools in my development toolbox, and I'd recommend that you learn and understand how to use the tool to improve your development process.
I'd recommend reading the Wikipedia article for more information.
The debugger just halts execution of your program when it hits a breakpoint. If your program is working okay when it hits the breakpoint, but doesn't work without the breakpoint, that would indicate to me that you have a race condition or another threading issue in your code. The breakpoint is stopping the execution of your code, perhaps allowing another process to complete normally?
It stops the program counter for your process (the one you are debugging), and shows the current value of your variables, and uses the value of your variables at the moment to calculate expressions.
You must take into account, that if you edit some variable value when you hit a breakpoint, you are altering your process state, so it may behave differently.
Debugging is possible because the compiler inserts debugging information (such as function names, variable names, etc) into your executable. Its possible not to include this information.
Debuggers sometimes change the way the program behaves in order to work properly.
I'm not sure about Visual Studio but in Eclipse for example. Java classes are not loaded the same when ran inside the IDE and when ran outside of it.
You may also be having a race condition and the debugger stops one of the threads so when you continue the program flow it's at the right conditions.
More info on the program might help.
On Windows there is another difference caused by the debugger. When your program is launched by the debugger, Windows will use a different memory manager (heap manager to be exact) for your program. Instead of the default heap manager your program will now get the debug heap manager, which differs in the following points:
it initializes allocated memory to a pattern (0xCDCDCDCD comes to mind but I could be wrong)
it fills freed memory with another pattern
it overallocates heap allocations (like a previous answer mentioned)
All in all it changes the memory use patterns of your program so if you have a memory thrashing bug somewhere its behavior might change.
Two useful tricks:
Use PageHeap to catch memory accesses beyond the end of allocated blocks
Build using the /RTCsu (older Visual C++ compilers: /GX) switch. This will initialize the memory for all your local variables to a nonzero bit pattern and will also throw a runtime error when an unitialized local variable is accessed.