winDBG .call equivalent in visual studio - c++

Question

For a homemade debugging tool built as a VS add-in, I need to:
break at some arbitrary point in my application
call into another method and break there (without adding code in that spot before runtime)
run some other command from my VS add-in at that second breakpoint
My first instinct on how to do this hit a wall at Hans' excellent answer here.
My second idea would be to set up the call to the other method from the breakpoint and have it execute when the application is allowed to continue (if you can see another way to do what I need, feel free to point it out, though !).
This would be trivial with WinDBG : just use .call and go. Unfortunately, I need to do this in Visual Studio.
Hence my question : is there some way to do this in VS ? I cannot find an equivalent to .call, nor a way to manipulate the registers and stack and emulate .call myself.

Answer 1

After some investigation, I believe the answer to this question is: there is no equivalent to .call in VS.
The only solution is to emulate the behavior of .call yourself by manipulating the stack pointer, instruction pointer, etc. This will obviously have limitations, e.g. mine will only work for the Microsoft x64 calling convention. Conversion to x86 and its myriad of calling conventions is left as an exercise for the reader ;)
Depending on your actual need, I have found two ways to do this. Remember, this is for calling into a function the next time the debuggee runs (so that you can break into it, since nested breakpoints are not supported). If you just need to call a function without breaking again, you are much better off just using the Immediate window to call it directly !
The easy way:
This will trick VS into thinking the current frame is in a DLL and method of your choosing. This is useful is the Expression Evaluator does not want to work in the DLL you are stopped in and needs to be in a different one.
WARNING: You cannot actually execute the method call you are faking without corrupting your stack (unless the method you are calling is very simple and you are very lucky).
Use the following to do this directly in the debugger via the Immediate window:
#rsp=#rsp-8
*((__int64*)$rsp)=#rip
#rip={,,<DLL to jump in.dll>}<method to call>
Now VS sees the DLL and method you specified as your current frame. Once you are done, use the following to return to the previous state:
#rip=*((__int64*)$rsp)
#rsp=#rsp+8
This can also be automated in a VS add-in by running these statements through EnvDTE.Debugger.GetExpression(), as demonstrated with the other method below.
The hard way:
This will work for actually calling the DLL and function you want and later returning from it cleanly. It is more complicated and more dangerous; any mistake will corrupt your stack.
It is also harder to get right for both debug and release mode, since the optimizer might have done complex things you were not expecting with the code of your callee and caller.
The idea is to emulate the Microsoft x64 calling convention (documented here) and break in the function called. We need to do the following things:
push parameters beyond the first 4 to the stack, in right to left order
create the shadow space on the stack(1)
push the return address, i.e. the current value of RIP
set RIP to the address of the function to call, just like above
save all the registers that the callee may change, and the caller might not expect to change. This basically means saving everything marked 'volatile'here.
set a breakpoint in the callee
run the debuggee
when the debuggee breaks again, perform whatever operations we want
step out
restore the saved registers
return RSP to the correct location (i.e. tear down the shadow space)
remove the breakpoint
(1) 32 bytes of scratch space for the callee to spill the first 4 arguments that are passed by registers (usually; the callee can actually use this however it likes).
Here is a simplified chunk of my VS addin to do this for a very basic case (non member function taking one parameter set to 0 and not touching too many registers). Anything beyond this is again left as an exercise for the reader ;)
EnvDTE90a.Debugger4 dbg = (EnvDTE90a.Debugger4)DTE.Debugger;
string method = "{,,dllname.dll}function";
string RAX = null, RCX = null, flags = null;
// get the address of the function to call and the address to break at (function address + a bit, to skip some prolog and help our breakpoint actually hit)
Expression expr = dbg.GetExpression3(method, dbg.CurrentThread.StackFrames.Item(1), false, false, false, 0);
string addr = expr.Value;
string addrToBreak = (UInt64.Parse(addr.Substring(2), NumberStyles.HexNumber) + 2).ToString();
if (!expr.IsValidValue)
return;
// set a breakpoint in the function to jump into
EnvDTE.Breakpoints bpsAdded = dbg.Breakpoints.Add("", "", 0, 0, "", dbgBreakpointConditionType.dbgBreakpointConditionTypeWhenTrue, "c++", "", 0, addrToBreak, 0, dbgHitCountType.dbgHitCountTypeNone);
if (bpsAdded.Count != 1)
return;
// set up the shadow space and parameter space
// NB: for 1 parameter : 4 words of shadow space, no further parameters... BUT, since the stack needs to be 16 BYTES aligned (i.e. 2 words) and the return address takes a single word, we need to offset by 5 !
dbg.GetExpression3("#rsp=#rsp-8*5", dbg.CurrentStackFrame, false, true, false, 0);
// set up the return address
dbg.GetExpression3("#rsp=#rsp-8*1", dbg.CurrentStackFrame, false, true, false, 0);
dbg.GetExpression3("*((__int64*)$rsp)=#rip", dbg.CurrentStackFrame, false, true, false, 0);
// save the registers
RAX = dbg.GetExpression3("#rax", dbg.CurrentStackFrame, false, true, false, 0).Value;
RCX = dbg.GetExpression3("#rcx", dbg.CurrentStackFrame, false, true, false, 0).Value;
// save the flags
flags = dbg.GetExpression3("#efl", dbg.CurrentStackFrame, false, true, false, 0).Value;
// set up the parameter for the call
dbg.GetExpression3("#rcx=0x0", dbg.CurrentStackFrame, false, true, false, 0);
// set the instruction pointer to our target function
dbg.GetExpression3("#rip=" + addr, dbg.CurrentStackFrame, false, true, false, 0);
dbg.Go(true);
// DO SOMETHING USEFUL HERE ! ;)
dbg.StepOut(true);
// restore all registers
dbg.GetExpression3("#rax=" + RAX, dbg.CurrentStackFrame, false, true, false, 0);
dbg.GetExpression3("#rcx=" + RCX, dbg.CurrentStackFrame, false, true, false, 0);
// restore flags
dbg.GetExpression3("#efl=" + flags, dbg.CurrentStackFrame, false, true, false, 0);
// tear down the shadow space
dbg.GetExpression3("#rsp=#rsp+8*5", dbg.CurrentStackFrame, false, true, false, 0);
}