Hooking usercall with inline asm - c++

Question

I disassembled an executable file with IDA pro. My goal is to hook the __usercall function. I know i need to wrap the __usercall with inine asm in my C++ code since i can't typedef thefunction. But i'm just not sure how this works.
I know the function takes an object as parameter and a *Vector3 as parameter in which the calculated value will be stored. Is there an easy way to tell which param will be what??
(sorry for the long code)
char __usercall sub_572EA0<al>(int a1<ecx>, int a2<edx>, int a3<eax>, int a4)
{
int v4; // edi#1
int v5; // esi#1
float v6; // eax#2
char v7; // al#3
int v8; // eax#5
char result; // al#11
int v10; // [sp+Ch] [bp-74h]#2
float v11; // [sp+10h] [bp-70h]#4
float v12; // [sp+14h] [bp-6Ch]#4
float v13; // [sp+18h] [bp-68h]#5
float v14; // [sp+1Ch] [bp-64h]#5
float v15; // [sp+20h] [bp-60h]#5
float v16; // [sp+24h] [bp-5Ch]#10
float v17; // [sp+28h] [bp-58h]#10
float v18; // [sp+2Ch] [bp-54h]#10
char v19; // [sp+30h] [bp-50h]#10
float v20; // [sp+3Ch] [bp-44h]#4
float v21; // [sp+40h] [bp-40h]#4
float v22; // [sp+44h] [bp-3Ch]#4
float v23; // [sp+54h] [bp-2Ch]#7
v4 = a3;
v5 = a1;
if ( a3 )
{
LODWORD(v6) = sub_55A920(*(_DWORD *)(a1 + 208));
if ( !sub_53ADD0(
v5,
v6,
v4,
(int)&v10) )
{
v7 = sub_4EC240(v4);
sub_4E3ED0(
1,
"Cannot find tag [%s]\n",
v7);
}
}
else
{
sub_572BE0();
*(float *)&v10 = *(float *)(v5 + 20) + v20;
v11 = *(float *)(v5 + 24) + v21;
v12 = *(float *)(v5 + 28) + v22;
}
v8 = dword_8FF12C;
v13 = flt_96A218;
v14 = flt_96A21C;
v15 = flt_96A220;
if ( dword_8FF12C == 2047 )
v8 = dword_8FF1D0;
sub_462250(
&v23,
&v13,
&v10,
&unk_82D6A0,
v8,
8400899);
if ( 1.0 == v23
|| (unsigned __int16)sub_492C50(&v23) == *(_DWORD *)(v5 + 208)
|| *(_UNKNOWN **)(v5 + 364) == &unk_FFFFFF
&& (v16 = v13
+ (*(float *)&v10 - v13)
* v23,
v17 = (v11 - v14) * v23 + v14,
v18 = v23 * (v12 - v15) + v15,
sub_4C35B0(
&v16,
v5 + 20,
v5 + 32,
&v19),
sub_432850(
*(_DWORD *)(v5 + 348),
&v19)) )
result = sub_550250(a4, &v13, &v10);
else
result = 0;
return result;
}
The ASM is probaly wrong, would something like this be close??
// Don't know what params goes where, ie: where the Vec3 goes and where the object goes
int __stdcall func_hook(param1, param2, param3, param4);
// Where to put the address? -->> 0x572EA0
// char __usercall sub_572EA0<al>(int a1<ecx>, int a2<edx>, int a3<eax>, int a4);
__declspec(naked) void func_hook()
{__asm{
push ebp
mov ebp, esp
mov ecx param1
mov edx param2
mov eax param3
push param4
call func_hook
leave
ret
}}
One thing missing in this piece of code is the address of the usercall (0x572EA0). Not sure where to put that...
This is how the program is calling the function. The call is at the bottom:
http://i43.tinypic.com/2mez9c8.jpg

Answer 1

that function you are hooking is Borland __fastcall, not __usercall (in fact there is actually no such convention, its just IDA's version of "unknown convention").
In terms of hooking this with inline asm, ECX, EDX and EAX are scratch registers, so we don't need to preserve them, and the call is well foermed so we don't need to worry about the stack:
static DWORD the_hook_address = 0x572EA0;
//init this somewhere with the correct (Base + RVA) address in case of module relocation (from ASLR etc.)
__declspec(naked) bool __stdcall the_hook(int a1, int a2, int a3, int a3)
{
__asm
{
MOV ECX,[ESP + 4]//a1
MOV EDX,[ESP + 8]//a2
MOV EAX,[ESP + 12]//a3
PUSH [ESP + 16]//a4
CALL the_hook_address
RETN 16 //4 * 4 args
}
}
I know the function takes an object as parameter and a *Vector3 as
parameter in which the calculated value will be stored. Is there an
easy way to tell which param will be what??
The 'easyness' depends on your experience in reverse engineering and with the program you are REing, in this case I'd say its a1, because you can see it move to a temporary, which is then accessed using pointer notation (IDA's way of representing unknown structs) to pull out 3 floats, which is generally what most apps use for vector components (and also most vectors have 3 components). It also helps greatly if you can actually debug the call in action, see what params are pointers, have a look at the functions call sites etc. For this reason I prefer using ollydbg for RE, supplementing it with IDA execution flow graphs for tricky jump sequences (think 20+ gotos in a function :<)