I am running some calculations in a program written in C++. For some input parameters I am running out of memory and my program is killed. I tried to spot the problematic part of my program using gdb (which usually works if I have other issues such as accessing memory at the wrong location), but my program is killed by the OS (Linux) when running out of memory, which makes it impossible to use backtrace on it.
Using valgrind and massif does not help either, after there is no massif-log if the program is killed due to being out of memory.
Therefore, are there other approaches I could use? One approach I can see is to simply print the current function name each time I call it (and then check the log to see which function I called last before I ran out of memory), but that would add a lot of code I don't want to write by hand. Are there "automated" methods doing that for me?
I have a program that I'm debugging in Visual Studio 2010. I have a reproducible error that occurs in the program and I am printing some diagnostic information. The error leaves the program in a bad state so I have to constantly restart the program. Each time I run the program the addresses for my structs are different. There are many of them and it would be much easier to debug if the addresses would stay the same each time I run the program.
The addresses look almost similar but are different. For example one struct has an address of 0x003F5540 one time, 0x003E5540 the next time, 0x00605540 and 0x004F5540 the next time.
The code executes exactly the same every time so I don't know why I see the slightly different addresses. I have turned off ASLR and DEP. What can I do to get the same addresses every time I run the program?
Thanks
Edit- It may not be possible to disable heap and stack randomization:
1st call to "new" always returns different addresses. How do I get it to return the same address?
There's no "may" about it, address randomization has been the core of every OS since 16 bit protected mode ones. Otherwise you couldn't run the same process twice. Or two processes that chose overlapping virtual base addresses.
Use symbol names instead of pointer values, that's what debug symbols are for!
my program executes exactly as desired when i run the debugger even with no breakpoints
when i run without debugging, i get a debug error
"This application has requested the Runtime to terminate it in an unusual way."
At one point, I call a function that sets a variable called currCode (an integer)
currCode = function();
//this throws debug error
If i add a cout of the variable currCode between this line and the next line, the program works fine with or without the debugger.
currCode = function();
cout << currCode; //this works!
Might try turning off optimization and see if you still get the problem.
There are many likely reasons for errors appearing in a program run directly from executable and one run by the debugger. Here are a few common ones:
Uninitialized Variables
DLL Hell
Timing
Heap or Stack Management
Again, the above are the most common.
Uninitialized Variables
Many debuggers will inadvertantly initialize your variables for you. A program run directly from an executable may not initialize the variable area, the way you expect. In the embedded systems world, this usually means not at all. So, get in the habit of initializing all variables, preferably when you declare them.
DLL Hell
Debuggers are nice and want to provide you with a nice experience, so they load a number of shared or dynamically linked libraries before your program is executed. Some of these libraries you will have to explicitly load.
Timing
Usually not common, but programs executing without a debugger run at different rates than those run a full speed with a debugger. This can make delay loops (spin loops) be have differently. Data buffers can have longer time to fill when using a debugger. When in doubt, use print statements in the release version to help narrow the location of the issue.
Heap or Stack Management
Debuggers usually provide code to protect your program from overrunning the stack, heap and other areas of memory. This comes with the feature to detect wild pointers and accessing data from invalid addresses. Also, debuggers want to protect what little memory the OS gives to them (they have to share memory with your program). A program running without a debugger can mess up stacks and heaps without any detections or generating faults.
I am running a C/C++ program in linux servers to serve videos. The program's(say named Plugin) core functionality is to convert videos and we fork a separate Plugin process for each video request. But I am having a weird problem for which sometimes server load average gets unexpectedly high. What I see from top command at this stage is that there are some processes which are running for long time and taking some huge CPU's.
When I debug this running program with gdb and backtrace stack,what I found is the corrupt stack: "Previous frame inner to this frame (corrupt stack?)". I have searched the net and found that this occurs if the program gets segmentation fault.
But what I know if the program gets segmentation fault, the program should crash and exit at that point. But surprisingly the program still running after segmentation fault.
What can be the causes of this? I know there must be some big problems in the program but I just can't understand from where to start fixing the problem...It would be great if any of you can show me some lights...
Thanks in advance
Attaching the debugger changes the behavior of the process so you won't get reliable investigation results most probably. Corrupted stack message from the debugger can mean that the particular debugger does not understand text info from the binary.
I would recommend running pstack several time subsequently on the problematic (this is known as "Monte Carlo performance profiling") and also attach strace or truss to the problematic and check what system calls is the process doing when consuming CPU.
Run your program under Valgrind and fix any invalid memory writes that it finds.
Certain optimisations, such as frame pointer omission, can make it harder for the debugger to understand the stack.
If you have the code, compile the program in debug and run Valgrind on it.
If you don't have the code, contact the author/provider of the program.
The corrupt stack message simply means the code is doing something weird with the memory. It does not mean the program has a segmentation fault. Also, the program can still run if it choose to handle the SIGSEGV signal.
If by forking you mean that you have some process which spawn and run other smaller processes, just monitor for such spikes and restart the process. This assumes that you have no access to the fix the program.
There could be some interesting manipulation of the stack taking place through assembly code manipulation, such as true tail-recursion optimization, self-modifying code, non-returning functions, etc. that may cause the debugger to be incapable of properly back-tracing the stack and causing it to trigger a corrupted stack error, but that doesn't necessarily mean that memory is corrupted ... but definitely something non-traditional is happening under the hood.
I've stumbled onto a very interesting issue where a function (has to deal with the Windows clipboard) in my app only works properly when a breakpoint is hit inside the function. This got me wondering, what exactly does the debugger do (VS2008, C++) when it hits a breakpoint?
Without directly answering your question (since I suspect the debugger's internal workings may not really be the problem), I'll offer two possible reasons this might occur that I've seen before:
First, your program does pause when it hits a breakpoint, and often that delay is enough time for something to happen (perhaps in another thread or another process) that has to happen before your function will work. One easy way to verify this is to add a pause for a few seconds beforehand and run the program normally. If that works, you'll have to look for a more reliable way of finding the problem.
Second, Visual Studio has historically (I'm not certain about 2008) over-allocated memory when running in debug mode. So, for example, if you have an array of int[10] allocated, it should, by rights, get 40 bytes of memory, but Visual Studio might give it 44 or more, presumably in case you have an out-of-bounds error. Of course, if you DO have an out-of-bounds error, this over-allocation might make it appear to be working anyway.
Typically, for software breakpoints, the debugger places an interrupt instruction at the location you set the breakpoint at. This transfers control of the program to the debugger's interrupt handler, and from there you're in a world where the debugger can decide what to do (present you with a command prompt, print the stack and continue, what have you.)
On a related note, "This works in the debugger but not when I run without a breakpoint" suggests to me that you have a race condition. So if your app is multithreaded, consider examining your locking discipline.
It might be a timing / thread synchronization issue. Do you do any multimedia or multithreading stuff in your program?
The reason your app only works properly when a breakpoint is hit might be that you have some watches with side effects still in your watch list from previous debugging sessions. When you hit the break point, the watch is executed and your program behaves differently.
http://en.wikipedia.org/wiki/Debugger
A debugger essentially allows you to step through your source code and examine how the code is working. If you set a breakpoint, and run in debug mode, your code will pause at that break point and allow you to step into the code. This has some distinct advantages. First, you can see what the status of your variables are in memory. Second, it allows you to make sure your code is doing what you expect it to do without having to do a whole ton of print statements. And, third, it let's you make sure the logic is working the way you expect it to work.
Edit: A debugger is one of the more valuable tools in my development toolbox, and I'd recommend that you learn and understand how to use the tool to improve your development process.
I'd recommend reading the Wikipedia article for more information.
The debugger just halts execution of your program when it hits a breakpoint. If your program is working okay when it hits the breakpoint, but doesn't work without the breakpoint, that would indicate to me that you have a race condition or another threading issue in your code. The breakpoint is stopping the execution of your code, perhaps allowing another process to complete normally?
It stops the program counter for your process (the one you are debugging), and shows the current value of your variables, and uses the value of your variables at the moment to calculate expressions.
You must take into account, that if you edit some variable value when you hit a breakpoint, you are altering your process state, so it may behave differently.
Debugging is possible because the compiler inserts debugging information (such as function names, variable names, etc) into your executable. Its possible not to include this information.
Debuggers sometimes change the way the program behaves in order to work properly.
I'm not sure about Visual Studio but in Eclipse for example. Java classes are not loaded the same when ran inside the IDE and when ran outside of it.
You may also be having a race condition and the debugger stops one of the threads so when you continue the program flow it's at the right conditions.
More info on the program might help.
On Windows there is another difference caused by the debugger. When your program is launched by the debugger, Windows will use a different memory manager (heap manager to be exact) for your program. Instead of the default heap manager your program will now get the debug heap manager, which differs in the following points:
it initializes allocated memory to a pattern (0xCDCDCDCD comes to mind but I could be wrong)
it fills freed memory with another pattern
it overallocates heap allocations (like a previous answer mentioned)
All in all it changes the memory use patterns of your program so if you have a memory thrashing bug somewhere its behavior might change.
Two useful tricks:
Use PageHeap to catch memory accesses beyond the end of allocated blocks
Build using the /RTCsu (older Visual C++ compilers: /GX) switch. This will initialize the memory for all your local variables to a nonzero bit pattern and will also throw a runtime error when an unitialized local variable is accessed.