I am debugging with GDB a crunching number C++ program. It takes 10 minutes till I reach the interesting function to be debugged. Then I inspect variables, understand parts of the program and recompile again, and run again GDB till I reach the point again.
This procedure is sometimes a bit time consuming. I wonder if somehow can be accelerated. Any ideas?
Thanks
You definitely can't have your compiler optimize the code to make it run faster before running GDB. Have you written good unit tests? Having a decent test suite might save you considerable time and prevent you from spending undue amount in the debugger.
there are gdb canned instruction(a sort of minilanguage where you automate the debugging process). and there are also python bindings that can help you automate gdb. debugging should be last resort, you should write tests instead or think more about what you write, this would speed up the deubgging process considerably(as you would probably not need to debug anymore, or very seldom).
Write tests which run the interesting function with various inputs. Then you can debug the function without having to worry about the rest of the code.
Have you tried UndoDB: http://undo-software.com/
It allows you to step back and forth - reversible debugging. While gdb has it's own reversible debugging these days, running in that mode has massive slowdown -- 20,000x or worse. UndoDB will run with about 1.7x slow-down, so you can quickly get to the interesting part, and then step back and forth to home in on your bug.
(Disclosure: I work for Undo Software)
Under GNU/Linux, you can also try:
checkpoint
...
restore n
if your program is not multithreaded (checkpoint uses internally fork(), so the same limitations apply).
It should same you the 10min you need to reach the beginning of your debugging!
Related
This is intended to be a general-purpose question to assist new programmers who have a problem with a program, but who do not know how to use a debugger to diagnose the cause of the problem.
This question covers three classes of more specific question:
When I run my program, it does not produce the output I expect for the input I gave it.
When I run my program, it crashes and gives me a stack trace. I have examined the stack trace, but I still do not know the cause of the problem because the stack trace does not provide me with enough information.
When I run my program, it crashes because of a segmentation fault (SEGV).
A debugger is a program that can examine the state of your program while your program is running. The technical means it uses for doing this are not necessary for understanding the basics of using a debugger. You can use a debugger to halt the execution of your program when it reaches a particular place in your code, and then examine the values of the variables in the program. You can use a debugger to run your program very slowly, one line of code at a time (called single stepping), while you examine the values of its variables.
Using a debugger is an expected basic skill
A debugger is a very powerful tool for helping diagnose problems with programs. And debuggers are available for all practical programming languages. Therefore, being able to use a debugger is considered a basic skill of any professional or enthusiast programmer. And using a debugger yourself is considered basic work you should do yourself before asking others for help. As this site is for professional and enthusiast programmers, and not a help desk or mentoring site, if you have a question about a problem with a specific program, but have not used a debugger, your question is very likely to be closed and downvoted. If you persist with questions like that, you will eventually be blocked from posting more.
How a debugger can help you
By using a debugger you can discover whether a variable has the wrong value, and where in your program its value changed to the wrong value.
Using single stepping you can also discover whether the control flow is as you expect. For example, whether an if branch executed when you expect it ought to be.
General notes on using a debugger
The specifics of using a debugger depend on the debugger and, to a lesser degree, the programming language you are using.
You can attach a debugger to a process already running your program. You might do it if your program is stuck.
In practice it is often easier to run your program under the control of a debugger from the very start.
You indicate where your program should stop executing by indicating the source code file and line number of the line at which execution should stop, or by indicating the name of the method/function at which the program should stop (if you want to stop as soon as execution enters the method). The technical means that the debugger uses to cause your program to stop is called a breakpoint and this process is called setting a breakpoint.
Most modern debuggers are part of an IDE and provide you with a convenient GUI for examining the source code and variables of your program, with a point-and-click interface for setting breakpoints, running your program, and single stepping it.
Using a debugger can be very difficult unless your program executable or bytecode files include debugging symbol information and cross-references to your source code. You might have to compile (or recompile) your program slightly differently to ensure that information is present. If the compiler performs extensive optimizations, those cross-references can become confusing. You might therefore have to recompile your program with optimizations turned off.
I want to add that a debugger isn't always the perfect solution, and shouldn't always be the go-to solution to debugging. Here are a few cases where a debugger might not work for you:
The part of your program which fails is really large (poor modularization, perhaps?) and you're not exactly sure where to start stepping through the code. Stepping through all of it might be too time-consuming.
Your program uses a lot of callbacks and other non-linear flow control methods, which makes the debugger confused when you step through it.
Your program is multi-threaded. Or even worse, your problem is caused by a race condition.
The code that has the bug in it runs many times before it bugs out. This can be particularly problematic in main loops, or worse yet, in physics engines, where the problem could be numerical. Even setting a breakpoint, in this case, would simply have you hitting it many times, with the bug not appearing.
Your program must run in real-time. This is a big issue for programs that connect to the network. If you set up a breakpoint in your network code, the other end isn't going to wait for you to step through, it's simply going to time out. Programs that rely on the system clock, e.g. games with frameskip, aren't much better off either.
Your program performs some form of destructive actions, like writing to files or sending e-mails, and you'd like to limit the number of times you need to run through it.
You can tell that your bug is caused by incorrect values arriving at function X, but you don't know where these values come from. Having to run through the program, again and again, setting breakpoints farther and farther back, can be a huge hassle. Especially if function X is called from many places throughout the program.
In all of these cases, either having your program stop abruptly could cause the end results to differ, or stepping through manually in search of the one line where the bug is caused is too much of a hassle. This can equally happen whether your bug is incorrect behavior, or a crash. For instance, if memory corruption causes a crash, by the time the crash happens, it's too far from where the memory corruption first occurred, and no useful information is left.
So, what are the alternatives?
Simplest is simply logging and assertions. Add logs to your program at various points, and compare what you get with what you're expecting. For instance, see if the function where you think there's a bug is even called in the first place. See if the variables at the start of a method are what you think they are. Unlike breakpoints, it's okay for there to be many log lines in which nothing special happens. You can simply search through the log afterward. Once you hit a log line that's different from what you're expecting, add more in the same area. Narrow it down farther and farther, until it's small enough to be able to log every line in the bugged area.
Assertions can be used to trap incorrect values as they occur, rather than once they have an effect visible to the end-user. The quicker you catch an incorrect value, the closer you are to the line that produced it.
Refactor and unit test. If your program is too big, it might be worthwhile to test it one class or one function at a time. Give it inputs, and look at the outputs, and see which are not as you're expecting. Being able to narrow down a bug from an entire program to a single function can make a huge difference in debugging time.
In case of memory leaks or memory stomping, use appropriate tools that are able to analyze and detect these at runtime. Being able to detect where the actual corruption occurs is the first step. After this, you can use logs to work your way back to where incorrect values were introduced.
Remember that debugging is a process going backward. You have the end result - a bug - and find the cause, which preceded it. It's about working your way backward and, unfortunately, debuggers only step forwards. This is where good logging and postmortem analysis can give you much better results.
My program is becoming non responsive when it is executing a certain function. How can I see what it's doing in real time to know why it's becoming non responsive?
It's not crashing, so I can't see the reason unfortunately. How can I find out what it's doing?
What can I do? What programs are out there that are good in this type of situation for both with and without the actual source code?
You need to run in the debugger, then break execution when the slowdown occurs. The place it stops, more than likely, will be where your problem is. If you start and stop multiple times, the place execution breaks most often will probably say for sure.
Or use a profiler. Either program will work with out without source, though it is much, MUCH, easier to use them with source and debug symbols (without requires quite a bit of assembler knowledge skill on the platform you are running.)
Have a look at Process Monitor from Sysinternals http://technet.microsoft.com/en-us/sysinternals/bb896645
A. with the source code
You can attach Visual Studio to a process and press break. Afterwards have a look at the call stack. If ti is not crashing its potentially no recursion. So you should see what it is doing.
If it is a multithreaded application switch between the threads and watch the call stack.
B. without the source code
I assume this is hard. The only thing that can be found is external analysis. Like wireshark to watch Network activities or Process Monitor from Sysinternals (procmon.exe) to see registry or file access.
Motivation: I cant get google cpu profiler to work on machine where code runs(with my last breath I curse libunwind :)), so I was wondering if the gdb supports high frequency random pausing of the program execution, storing the name of the function where break occured and counting how many times it paused in function x.
That is what I would call "run time sampling", there is probably more precise/smarter name.
I looked at the oprofile, but it is to complicated to a) figure out if it can do it b) to figure out how to do it
EDIT: apparently correct name is:
"statistical sampling method"
EDIT2: reason why Im offering a bounty for this is that I see some ppl on SO recommending doing manual break 10-20x and examining stack with bt...
Seems very wasteful when it comes to time, so I guestimate some smart ppl automated it. :)
EDIT3: gprof wont cut it... i tried running it recently on ARM system and output was trash... :( I guess its troubles with multithreading is the reason for that...
You can manually sample in GDB by pausing it at run time.
What you seem to think you want is gprof, but
if your goal is to make the program as fast as possible, then I would suggest
High frequency of sampling is not helpful.
Counting the number of samples where the program counter is in function X is not helpful except in artificially small programs.
If you follow that link, you will see the reasons why, and directions for how to do it successfully.
GDB would not do this well, although you could certainly hack something up that gave wildly inaccurate results.
I'd suggest Valgrind's "Callgrind" plugin. As a bonus it requires absolutely no recompilation or other special setup. All you need is valgrind installed in your system, and debug information in your program (or, full symbol information, at least; I'm not sure).
You then invoke your program like this:
valgrind --tool=callgrind <your program command line>
When it's done there will be a file name callgrind.out.<pid> in the current directory. You can read and visualise this file with a very nice GUI tool called kcachegrind (usually you have to install this separately).
The only problem is that, because callgrind slows the execution of your program slightly, the time spent in system calls can appear smaller (in percentage terms) than it really would be. By default, callgrind does not include system time in its counters, so the values it give you are a real comparison of the code in your program, if not the actual time 'under' that function. This can be confusing, at first, so if that happens you try adding --collect-systime=yes.
I'm not sure what the state of callgrind on ARM might be. ARMv7 is listed as a supported platform, but only says "fairly complete", whatever that means.
I'm currently working on a program (in C++, using Code::Blocks) that uses a lot of random numbers and takes a while to get going; most of the time, it works fine, but every now and then it performs an illegal operation and must shut down. Given the random numbers all over the place, and the fact that it currently takes ~3-5 minutes for the program to reach the stage at which the errors occur (this timeframe is normal/acceptable), reproducing the problems reliably and convenient is extremely difficult, and reporting on every other line of code to cout to manually track things is time-consuming, visually clutters reporting on things not related to bugs, and is not always helpful, since even if I know when the program stops, I sometimes don't know why.
Is there some way for me to see what the last operation in the program was before it crashed, and for me to see why this operation lead to a crash? Something within CodeBlocks would be best, but something third-party works too. It also needs to be something I can use every time I test the program, because I never know when a crash is going to occur.
That is what debuggers are for. Build the system with full debugging symbols, configure the system so that you get a full crash report (in linux a core file), and then launch the debugger with the core file (alternatively run the whole program inside the debugger, but that might take a while, running inside a debugger is usually much slower than running outside of it.
The debugger should be able to give you the state of the program when the illegal instruction happened and you will get some insight as of the state that the program was on. From there either you figure what is wrong, or maybe you can make a couple of smaller testcases that might trigger the error.
Debugging issues that cannot be reproduced systematically is a pain, good luck there!
Sounds like you want a debugger. Debugging C and C++ programs using GDB
It never happened to me. In Visual Studio, I have a part of code that is executed 300 times, I time it every iteration with the performance counter, and then average it.
If I'm running the code in the debugger I get an average of 1.01 ms if I run it without the debugger I get 1.8 ms.
I closed all other apps, I rebooted, I tried it many times: Always the same timing.
I'm trying to optimize my code, but before throwing me into changing the code, I want to be sure of my timings. To have something to compare with.
What can cause that strange behaviour?
Edit:
Some clarification:
I'm running the same compiled piece of code: the release build. The only difference is (F5 vs CTRL-F5)
So, the compiler optimization should not be invoved.
Since each calcuated times were verry small, I changed the way I benchmark: I'm now timing the 300 iterations and then divide by 300. I have the same result.
About caching: The code is doing some image cross correlation, with different images at each iterations. The steps of the processing are not modified by the data in the images. So, I think caching is not the problem.
I think I figured it out.
If I add a Sleep(3000) before running the tests, they give the same result.
I think it has something to do with the loading of misc. dlls. In the debugger, the dlls were loaded before any code was executed. Outside the debugger, the dlls were loaded on demand, and one or more were loaded after the timer was started.
Thanks all.
I don't think anyone has mentioned this yet, but the debug build may not only affect the way your code executes, but also the way the timer itself executes. This can lead to the timer being inaccurate / slower / definitely not reliable. I would recommend using a profiler as others have mentioned, and compare only similar configurations.
You are likely to get very erroneous results by doing it this way ... you should be using a profiler. You should read this article entitled The Perils of MicroBenchmarking:
http://blogs.msdn.com/shawnhar/archive/2009/07/14/the-perils-of-microbenchmarking.aspx
It's probably a compiler optimization that's actually making your code worse. This is extremely rare these days but if you're doing odd, odd stuff, this can happen.
Some debugger / IDEs like Visual Studio will automatically zero out memory for you in Debug mode; this may be a contributing factor.
Are you running the exact same code in the debugger and outside the debugger or running debug in the debugger and release outside? If so the code isn't the same. If you're running debug and release and seeing the difference you could turn off optimization in release and see what that does or run your code in a profiler in debug and release and see what changes.
The debug version initializes variables to 0 (usually).
While a release binary does not initialize variables (unless the code explicitly does). This may affect what the code is doing the ziae of a loop or a whole host of other possibilities.
Set the warning level to the highest level (level 4, default 3).
Set the flag that says treat warnings as errors.
Recompile and re-test.
Before you dive into an optimization session get some facts:
dose it makes a difference? dose this application runs twice as slow measured over a reasonable length of time?
how are the debug and release builds configured
what is the state of this project? Is it a complete software or are you profiling a single function ?
how are you running the debug and build releases , are you sure you are testing under the same conditions (e.g. process priority settings )
suppose you do optimize the code what do you have in mind ?
Having read your additional data a distant bell started to ring ...
When running a program in the debugger it will catch both C++ exceptions and structured exceptions (windows execution)
One event that will trigger a structured exception is a divide by zero, it is possible that the debugger quickly catches and dismiss this event (as a first chance exception handling) while the release code goes a bit longer before doing something about it.
so if your code might be generating such or similar exceptions it worth a while to look into it.