How does the visual studio debugger detect function calls? - c++

When ive attached the visual studio debugger to a process, then add a new function breakpoint, and choose e.g. "system" as the function name. Visual studio will then detect whenever the system() function is called from the target process.
Is there any efficient way to detect function calls from a process using c++?

A debugger detects that a breakpoint has been hit in several ways.
The basic strategy is the software breakpoint. This means that the debugger replaces an instruction at the breakpoint target with a breakpoint trap instruction.
When the execution hits the breakpoint instruction, a CPU exception is caused which is then handled via the debugger. The debugger sees that execution has stopped at a certain address, and for a certain reason, and that confirms to it that the breakpoint has been hit.
Lastly, implementing breakpoint debugging is possible even without the help from a breakpoint instruction. A software breakpoint could be inserted into the target code as ordinary branch instruction that jumps into a routine inside debugger, rather than raising CPU exception which is handled through the debugger.
In addition to this, processors can (and do) support hardware breakpoints. Usually a very limited number of hardware breakpoints can be configured in the processor which tell it to stop at a certain address. This is less intrusive than a software breakpoint, and will work even if the code is in read-only memory.
Of course if you have a software breakpoint, the missing instruction which has been replaced has to be executed when the execution is restarted. The debugger must put the original instruction in place before resuming. But if that is all that is done, the breakpoint will effectively disappear. The breakpoint must be re-armed so that execution will stop if it is hit again. To achieve that, the debugger can put the processor into single step mode. It can replace the original instruction, then step the code in single step mode, then put in the software breakpoint again and resume.
If the processor doesn't have a single step mode, but only a break instruction, then the processor can simulate single stepping using a temporary software breakpoint (one that doesn't have to be re-armed once executed).

Related

How can you read/write asm registers from an .exe using C++?

I want to modify the value of a register that is in a certain program.
The only problem is, I don't know how to access it. If there is a way, how do I read/write into it? (preferred language C++)
If you want to modify a particular register one time while the program is running you can do so using a debugger, such as OllyDbg.
If you want to modify the code while the program isn't running, such that in the future when you run the program it behaves differently, you can view the Assembly using a disassembler such as IDA. But you'll also need something that can reassemble the program with your modifications, such as NAsm
You can also attach one program to another while both are running using the OpenProcess() function in windows. You can then read and write arbitrary values to the other process, including modifying it's code. This is a pretty tricky thing to set up and have work properly... It's how debuggers work, which are usually pretty complex pieces of software. Better to use an existing one than to try to write your own!
As far as I understand you correctly you want to write a program that:
Stops another running program at a certain address (breakpoint)
Waits until the breakpoint is reached
Reads the current value of a certain register
Writes another value into that register
Continues program execution after the breakpoint
Not using a breakpoint makes absolutely no sense because the assembly registers are used for completely different purposes in different parts of the program. So modifying an assembly register only makes sense when the other program is stopped at a specific position.
Under Windows, Win32 code (not .NET code), you may du this the following way:
Start the other .EXE file using CreateProcess with the DEBUG_PROCESS (and maybe - what would be more safe - the CREATE_SUSPENDED) flag set or...
...use DebugActiveProcess to debug an already running process.
Use ReadProcessMemoryand WriteProcessMemory to replace the assembler instruction (to be correct: the byte and not the instruction) at the breakpoint address by 0xCC (int3).
If you used CREATE_SUSPENDED use ResumeThread to actually start the other .EXE file
Call WaitForDebugEvent and ContinueDebugEvent until the breakpoint is reached. You may use GetThreadContext to get the location where the program is stopped.
Replace the int3 instruction by the original instruction using WriteProcessMemory
Use GetThreadContext and SetThreadContext to modify registers. Node: After an int3 instruction the EIP register must be decremented!
Use ContinueDebugEvent to let the program continue
Unfortunately your program has to wait for further debugging events of the observed EXE file and handle them (WaitForDebugEvent and ContinueDebugEvent) - even if you "just" want the program to run...
If you want to stop the program at this breakpoint multiple times it is becoming more complicated: You must set the "trace" flag in the EFlags register each time you continue after the breakpoint so a single-step is done; after that single step you have to replace the original assembler instruction by int3 again.
I myself did this years ago because I wrote some inspection program for that worked similar to "strace" in Linux...
If you are using C++.NET (or C#): There is a .NET assembly that allows you doing all this but using this assembly is not easier than the Win32 variant.
If you want to do this for educational uses only:
Unfortunately the debugging API of Windows is by far less easy to use then the debugging API (ptrace) of Linux. If you just want to do this for educational purposes (and the EXE file you want to inspect is written by you) I would do this excercice under Linux, not under Windows. However even under Linux this is not an easy job...

how GDB knows it has to break at specified break point?

A basic question & I am very new to C/C++ and GDB.
We use GDB to debug a process. We attach GDB to a process and then specify filename.c along with line number to put break point.
My question is "How would GDB or OS OR possibly anything else know that it has to break at specified line number (in filename.c) after we connect GDB to running process?"
What is coming into picture that, say, the current process is run in debug mode and a breakpoint is applied and the process execution has to break (wait for user input) at that point?
The same way that if your program stops or crashes at a particular point, the debugger can tell you where in the program that point is.
For both of these to work the program binary must contain additional debugging information that associates addresses in the program image with locations in the source code (source file and line number.)
To add a breakpoint at a particular line the debugger finds the program address closest to that line, modifies the copy of the executable in memory to insert a special "break" instruction at that location which will cause the program's execution to be interrupted, then "traces" the program's execution and waits for it to reach the breakpoint and stop.
For more details see http://eli.thegreenplace.net/2011/01/23/how-debuggers-work-part-1/ and http://www.howzatt.demon.co.uk/articles/SimplePTrace.html
I can't comment for the latest version of gdb - but many debuggers actually swap the assembly instruction at the desired breakpoint location (in memory) with an interrupt instruction. This "wakes up" the debugger which takes control at this point.
Using a substituted interrupt instruction means that the CPU can execute your program at full speed and "trip up" at the desired location.
Modern processors are very complex, however, and probably have far superior debugging features.
GDB is aware of your code : it knows all about it. When you set a breakpoint at a line, GDB gets the equivalant machine instruction address : all your code (as machine instructions) is loaded in memory, so the instructions of your code have an address.
So now GDB knows the adress of the instruction you want to break. When you run your programm, GDB will use ptrace, which allow GDB to "see" each instructions before their execution. Then GDB have just to look if the current instruction (which will be executed) is the same as your instruction (that you want to break).

Breakpoints not working when booting from Flash

In the past, I have been debugging executables loaded in the internal SRAM of my Cortex M3 (STM32F2) without problems. I have recently been loading my executable to Flash (because of size issues).
Ever since, debugging with GDB has not been working. As I understand, when the executable is in Flash, only hardware breakpoint can be used (as opposed to software breakpoints), and I have six hardware breakpoints. However, when setting just one hardware breakpoint GDB yields an error message:
(gdb) break main
Breakpoint 1 at 0x800019a: file src/main.c, line 88.
(gdb) c
Continuing.
Note: automatically using hardware breakpoints for read-only addresses.
(gdb) Warning:
Cannot insert hardware breakpoint 1.
Could not insert hardware breakpoints:
You may have requested too many hardware breakpoints/watchpoints.
What could be going wrong? Have my hardware breakpoints be taken in the background?
Note: I used OpenOCD to load the executable through JTAG.
So, there's basically two ways (plus one really bad way) that breakpoints can be implemented on any given debugger/platform combination:
Use some hardware capabilities ("hardware breakpoints") to cause the processor to trap when it hits a particular address. This is typically restricted to just a couple of breakpoints, if it's available at all.
For each breakpoint that's being set, replace the instruction at the breakpoint with a "trap" instruction of some variety (i.e, an instruction that will break into the debugger). When one of the breakpoints is hit, swap the original instruction back in and single-step once to make it run.
Single-step through the whole program. This one doesn't really count, because it's horrifically slow.
It sounds as though your debugger is only using method #2 ("software breakpoints"). The gotcha with this method is that it requires the program to be writable -- and flash memory isn't writable one instruction at a time, so this technique won't work.

How do breakpoints work in C++ code?

How do breakpoints work in C++ code? Are they special instructions inserted in between some assembler instructions when the code is compiled? Or is there something else in place? Also, how are stepping-through-the-code implemented? The same way as breakpoints...?
This is heavly depend on the CPU and debugger.
For example, one of the possible solution on x86 CPU:
Insert one-byte INT3 instruction on the required place
Wait until breakpoint exception hits
Compare exception address to the list of breakpoint to determine which one
Do breakpoint actions
Replace INT3 with original byte and switch the debugged process into trace mode (step-by-step execution of CPU instructions)
Continue debugged process
Immediately you catch trace exception - the instruction was executed
Put INT3 back
Watchpoints can be implemented in the similar way, but instead of INT3 you put the memory page where watched variable is into read only, or into no access mode, and wait for segmentation exception.
Stepping through assembly can also be done by using trace mode. Stepping through source lines can also be done by placing breakpoints onto next instructions, based on debug data.
Also some CPU has hardware breakpoint support, when you just load address into some register.
According to this blog entry on technochakra.com you are correct:
Software breakpoints work by inserting a special instruction in the program being debugged. This special instruction on the Intel platform is “int 3″. When executed it calls the debugger’s exception handler.
I'm not sure how stepping into or over the next instruction is implemented though. However, the article goes on to add:
For practical reasons, it is unwise to ask for a recompilation whenever a breakpoint is added or deleted. Debuggers change the loaded image of the executable in memory and insert the “int 3″ instruction at runtime.
However, this would only be used for the "run to current line option".
Single stepping is implemented at (assembler) code level not at C++ level. The debugger knows how to map the C++ code lines to code addresses.
There are different implementations. There are CPUs that support debugging with breakpoint registers. When the execution reaches the address in the breakpoint register, the CPU executes a breakpoint exception.
A different approach is to patch the code for the time of execution with a special instruction, at best a one-byte instruction. At x86 systems that usually int 3.
The first approach allows breakpoints in ROM, the second allows more breakpoints at the same time.
AFAIK all debuggers (for whatever compiled language) that allow an unlimited number of breakpoints use a variant of replacing the instruction to be breakpointed with a special value (as described above) and keeping a list of places where these values have been placed.
When the processor tries to execute one of these special values, an exception is raised, the debugger catches it and checks if the address of the exception is on its list of breakpoints.
If it is, the debugger is invoked and the user is given an opportunity to interact.
If it is NOT, then the exception is due to something that was in the program from the outset and the debugger lets the exception 'pass' to whatever error handler might be there.
Note also, that debugging self-modifying code can fail precisely because the debugger momentarily modifies the code itself. (Of course, nobody would ever write self-modifying, now would they? >;-)
For these reasons, it is important that the debugger be given the opportunity to remove all the breakpoints it sets before terminating the debugging session.

What exactly does a debugger do?

I've stumbled onto a very interesting issue where a function (has to deal with the Windows clipboard) in my app only works properly when a breakpoint is hit inside the function. This got me wondering, what exactly does the debugger do (VS2008, C++) when it hits a breakpoint?
Without directly answering your question (since I suspect the debugger's internal workings may not really be the problem), I'll offer two possible reasons this might occur that I've seen before:
First, your program does pause when it hits a breakpoint, and often that delay is enough time for something to happen (perhaps in another thread or another process) that has to happen before your function will work. One easy way to verify this is to add a pause for a few seconds beforehand and run the program normally. If that works, you'll have to look for a more reliable way of finding the problem.
Second, Visual Studio has historically (I'm not certain about 2008) over-allocated memory when running in debug mode. So, for example, if you have an array of int[10] allocated, it should, by rights, get 40 bytes of memory, but Visual Studio might give it 44 or more, presumably in case you have an out-of-bounds error. Of course, if you DO have an out-of-bounds error, this over-allocation might make it appear to be working anyway.
Typically, for software breakpoints, the debugger places an interrupt instruction at the location you set the breakpoint at. This transfers control of the program to the debugger's interrupt handler, and from there you're in a world where the debugger can decide what to do (present you with a command prompt, print the stack and continue, what have you.)
On a related note, "This works in the debugger but not when I run without a breakpoint" suggests to me that you have a race condition. So if your app is multithreaded, consider examining your locking discipline.
It might be a timing / thread synchronization issue. Do you do any multimedia or multithreading stuff in your program?
The reason your app only works properly when a breakpoint is hit might be that you have some watches with side effects still in your watch list from previous debugging sessions. When you hit the break point, the watch is executed and your program behaves differently.
http://en.wikipedia.org/wiki/Debugger
A debugger essentially allows you to step through your source code and examine how the code is working. If you set a breakpoint, and run in debug mode, your code will pause at that break point and allow you to step into the code. This has some distinct advantages. First, you can see what the status of your variables are in memory. Second, it allows you to make sure your code is doing what you expect it to do without having to do a whole ton of print statements. And, third, it let's you make sure the logic is working the way you expect it to work.
Edit: A debugger is one of the more valuable tools in my development toolbox, and I'd recommend that you learn and understand how to use the tool to improve your development process.
I'd recommend reading the Wikipedia article for more information.
The debugger just halts execution of your program when it hits a breakpoint. If your program is working okay when it hits the breakpoint, but doesn't work without the breakpoint, that would indicate to me that you have a race condition or another threading issue in your code. The breakpoint is stopping the execution of your code, perhaps allowing another process to complete normally?
It stops the program counter for your process (the one you are debugging), and shows the current value of your variables, and uses the value of your variables at the moment to calculate expressions.
You must take into account, that if you edit some variable value when you hit a breakpoint, you are altering your process state, so it may behave differently.
Debugging is possible because the compiler inserts debugging information (such as function names, variable names, etc) into your executable. Its possible not to include this information.
Debuggers sometimes change the way the program behaves in order to work properly.
I'm not sure about Visual Studio but in Eclipse for example. Java classes are not loaded the same when ran inside the IDE and when ran outside of it.
You may also be having a race condition and the debugger stops one of the threads so when you continue the program flow it's at the right conditions.
More info on the program might help.
On Windows there is another difference caused by the debugger. When your program is launched by the debugger, Windows will use a different memory manager (heap manager to be exact) for your program. Instead of the default heap manager your program will now get the debug heap manager, which differs in the following points:
it initializes allocated memory to a pattern (0xCDCDCDCD comes to mind but I could be wrong)
it fills freed memory with another pattern
it overallocates heap allocations (like a previous answer mentioned)
All in all it changes the memory use patterns of your program so if you have a memory thrashing bug somewhere its behavior might change.
Two useful tricks:
Use PageHeap to catch memory accesses beyond the end of allocated blocks
Build using the /RTCsu (older Visual C++ compilers: /GX) switch. This will initialize the memory for all your local variables to a nonzero bit pattern and will also throw a runtime error when an unitialized local variable is accessed.