Assembly code causes recursion - c++

Question

I've been writing a C application, and I came in need of x86 assembly language. I'm pretty new to assembly, and the following snippet causes recursion:
unsigned int originalBP;
unsigned fAddress;
void f(unsigned short aa) {
printf("Function %d\n", aa);
}
unsigned short xx = 77;
void redirect() {
asm {
pop originalBP
mov fAddress, offset f
push word ptr xx
push fAddress
push originalBP
}
}
If i call redirect , it will repeatedly output: "Function 1135"
First, here are a few information about the environment in which this code is executed:
This code is written to be executed under NTVDM
Tiny memory model is used ( all segment pointer registers point to the same segment )
Here's my expectation of what the code above should do ( this is most likely the culprit of the error ) :
Pop the stack and store value in originalBP; I believe the value is actually the address of the current function i.e. redirect
Push f's argument value ( value of xx ) to the stack
Push address of f to stack ( since there's only one segment, only offset is needed )
Push back the address of redirect
Of course, if this were the correct flow, recursion would be apparent ( except the part where 1135 is printed instead of 7). But interestingly, doing the same with a function with no arguments produces only one line of output i.e.:
unsigned int originalBP;
unsigned fAddress;
void f() {
printf("Function");
}
void redirect() {
asm {
pop originalBP
mov fAddress, offset f
push fAddress
push originalBP
}
}
This probably means that my understanding of the above code is completely wrong. What is the real issue in this code?
EDIT: I probably left some of the things unsaid:
This is a 16 bit application
Compiler used is Borland C++ 3.1, as Eclipse plugin
redirect is called from main as redirect()
EDIT (regarding Margaret Bloom's answer) Here's an example of instruction execution once redirect is called. Values in brackets represent stack pointer register and the value at that location before each instruction is executed:
call redirect
(FFF4-04E6) push bp
(FFF2-FFF6) mov bp, sp
(FFF2-FFF6) mov fAddress, offest f
(FFF2-FFF6) pop originalBP
(FFF4-04E6) pop originalRIP
(FFF6-0000) push xx (I've changed xx to 1187)
(FFF4-0755) push originalRIP
(FFF2-04E6) push fAddress
(FFF0-04AC) push originalBP
(FFEE-FFF6) pop bp
(FFF0-04AC) ret
(in f) (FFF2-04E6) push bp
(FFF0-FFF6) mov bp,sp
printf executes
(FFF0-FFF6) pop bp
(FFF2-04E6) ret
Next statements seems to be return 0; which is the end of main.
Execution continues trough bunch of lines, and somehow comes back to the line calling redirect.

Answer 1

For you second snippet, the one without arguments, the stack states are as follow:
Where | Stack (growing on the left)
----------------------+----------------------------
after redirect prolog redirect rip, redirect bp
pop originalBP redirect rip
push fAddress redirect rip, fAddress
push originalBP redirect rip, fAddress, redirect bp
after redirect epilog redirect rip, fAddress
after redirect return redirect rip (control moved to f)
after f prolog redirect rip, f bp
after f epilog redirect rip
after f return (control moved to redirect caller)
Where redirect rip means the return address (return IP) of the function redirect.
As you can see, upon entering of f the stack correctly points to redirect rip, the return address of redirect.
Upon exit, the control flows back to redirect caller.
For your first snippet, the stack goes as follow:
Where | Stack (growing on the left)
----------------------+----------------------------
after redirect prolog redirect rip, redirect bp
pop originalBP redirect rip
push word ptr xx redirect rip, xx
push fAddress redirect rip, xx, fAddress
push originalBP redirect rip, xx, fAddress, redirect bp
after redirect epilog redirect rip, xx, fAddress
after redirect return redirect rip, xx (control moved to f)
after f prolog redirect rip, xx, f bp
after f epilog redirect rip, xx
after f return (control moved to xx)
Upon entering of f we have redirect rip, xx on the stack when we should really have xx, redirect rip.
with the former configuration the parameter aa contains the return address of redirect and the return address of f is the value of xx.
Based on your answer to my comment the code looped by accident.
If you want to call f with arguments, be sure to push them before the return address:
pop originalBP
pop originalRIP
;Arguments go here
push xx
push originalRIP
push fAddress
push originalBP

Answer 2

You didn't post what compiler and compiling options you use to code that redirect.
With optimizations ON, you can't assume the full C function prologue/epilogue will be used, so you are operating with stack without any idea of it's layout (if there would be zero prologue/epilogue, then you did inject 2 values ahead of return address to caller, so redirect would simply return to caller (main?) which may essentially just exit -> no call to f = not your case).
As inside the asm block you already have the fn address, why don't you simply call it? The stack would be like: somebody calls redirect -> redirect calls some address -> address fn() -> returns to redirect -> returns to caller.
It looks to me like you are trying to modify it to: somebody calls redirect -> redirect calls some address -> address fn() -> returns to caller (skipping return to redirect). As the redirect epilogue is tiny bit of code, I don't see much benefit of that modification (also I don't see how it is "context switch" related).
Anyway, check your compiler options how to produce the assembly listing of final code to see how it does really compile, or even better, check it with debugger (step per instruction on assembly level).
EDIT (after providing the debug info):
when you get to return 0, there's additional alien xx injected in stack (sp being 0xFFF4) instead of sp being original FFF6 pointing to the 0.
The end of main probably does not handle this correctly (doing pop bp ret I guess), assuming the sp is correct upon return. (would it do the other C epilogue including mov sp,bp, it would probably survive your stack tampering).
Then again, if it would do other epilogue in all functions, it would do it in redirect() too, so you would have to modify bp as well to make the end of redirect() do ret into fAddress. Like dec bp, dec bp would probably suffice, as you have grown the stack by injecting 2B into params space.
Check the debug one more time when return 0 in main is hit, how it is implemented, if it can cope with modified sp or not (well, obviously it can't, as it loops by accident into redirect).
If that's the case, you should probably patch the main to restore sp before return 0;. I wonder whether simple mov sp,bp would do (bp should be FFF6 ahead of that).
Conclusion: tampering with stack frames across several calls is always tricky business. ;)

Answer 3

So, are you heading toward something like this? (because I sort of can't exactly put my finger on where your code from question will be used then, seems like basic stack exercise to give you idea how code execution can be affected, which will later evolve probably into something like this... maybe... and maybe not).
Fake context switch in 16b in some C-like pseudo code (ok, more like comments only :) ), has to be installed as some timed interrupt:
// should be some "far" type function to preserve "cs" as well
far void fakeThreadSwitch() {
asm {
cli ; or other means to disable thread switch (re-entry)
; store the current values of all registers
pusha
pushf
push ds
push es
; set `ds` to thread contexts data section
; figure out, which thread is currently running
; (have some "size_t currently_running = index;" in context section)
; if none, then pick some SLEEPING
; but have some [root_context] updated (.stack), so you can
; do final switch to it upon terminating the OS.
; verify the ss points to that thread stack ->
; if you by accident did interrupt OS kernel,
; then just return without touching anything (jump to "pop es")
; store ss:sp to [current_thread_context.stack]
; decide if you want to switch to some other context
; (or kill current) simulating "preemptive multitasking"
; if switch, set up all flags correctly (RUNNING/SLEEPING/index)
; load ss:sp from [next_thread_context.stack]
pop es
pop ds
popf
popa
sti ; or enable thread switch interrupt by other means
}
}
Then to start some new thread executing code at fAddress:
void startNewThread(void far *fAddress) {
// allocate some new context for the new thread
// (probably fixed array for max threads, searching for "FREE" one)
// ... (inits fields in some struct [new_thread_context])
// allocate some new stack memory for the new thread
// ... (sets [new_thread_context.stack_allocated])
// set up the stack for initial threadSwitch
uint16_t far * stackEnd = [new_thread_context.stack]
// reserve: es,ds + flags + all + cs:ip (to be executed) + OS exit trap (3x)
stackEnd -= (2 + 1 + 8 + 2 + 3);
// init the values in "stack"
stackEnd[0] = stackEnd[1] = [new_thread_context.ds]; // es, ds
stackEnd[2] = 0; // flags
stackEnd[3] = stackEnd[4] = stackEnd[5] = 0; // di, si, bp
stackEnd[6] = offset(stackEnd+11); // sp ahead of "pusha"
stackEnd[7] = stackEnd[8] = 0; // bx, dx
stackEnd[9] = stackEnd[10] = 0; // cx, ax
stackEnd[11] = segment(fAddress); // "return" to fAddress
stackEnd[12] = offset(fAddress);
// thread_exit_return is some trap function to handle
// far return inside fAddress code, which would probably require
// different design to make this truly usable (to fit C epilogue of f())
stackEnd[13] = segment(&thread_exit_return);
stackEnd[14] = offset(&thread_exit_return);
stackEnd[15] = thread_id;
[new_thread_context.stack] = stackEnd;
// all context data are ready for context switch, mark this thread "ready"
[new_thread_context.running] = SLEEPING;
// now in some future the context-switch may pick this thread from
// pool of sleeping threads, and will switch execution to it
// (through this artificially prepared stack image)
}
One of kernel handlers, this one designed as "landing" point for any f() finishing normally, which would just return (or calling this explicitly).
void thread_exit_return() {
// get the exited thread_id somehow
[thread_context.running] = FINISHED;
// deallocate [thread_context.stack_allocated]
// deallocate thread context (marking it as "FREE"?)
}
Would require some more thought and design how to run kernel itself (whether in just another thread, or in the original app context), and how to give it running time. And how to control kernel to execute new threads, or kill/exit old ones.
Anyway, the important part of this exercise is that push-all-in-thread-stack / pop-all-from-other-stack, to give you rough idea how preemptive multitasking works (although in 32b protected OS this involves much more trickery with CPU switching to protected layer (and back to user land) and using different stack for kernel, etc.. so only the principle is same).
Of course in 16b unprotected this is quite fragile construction, which can be damaged across different threads easily (and I very likely did oversight something important, so it would require very likely some heavy bug fixing to make it work).