Microsoft Detours - unable to hook __thiscall function - c++

I am trying to hook an undocumented function which has the signature:
(void(__thiscall*)(int arg1, int arg2))0x6142E0;
I have looked at the detours sample "member" where it explains:
By default, C++ member functions use the __thiscall calling
convention. In order to Detour a member function, both the trampoline
and the detour must have exactly the same calling convention as the
target function. Unfortunately, the VC compiler does not support a
__thiscall, so the only way to create legal detour and trampoline functions is by making them class members of a "detour" class.
In addition, C++ does not support converting a pointer to a member
function to an arbitrary pointer. To get a raw pointer, the address
of the member function must be moved into a temporary member-function
pointer, then passed by taking it's address, then de-referencing it.
Fortunately, the compiler will optimize the code to remove the extra
pointer operations.
I have copied some code from the example and modified it but I cant seem to get this to work(original example code here):
class CDetour {
public:
void Mine_Target(int arg1, int arg2);
static void (CDetour::* Real_Target)(int arg1, int arg2);
};
void CDetour::Mine_Target(int arg1, int arg2) {
printf(" CDetour::Mine_Target! (this:%p)\n", this);
(this->*Real_Target)(arg1, arg2);
}
void (CDetour::* CDetour::Real_Target)(int arg1, int arg2) = (void(CDetour::*)(int arg1, int arg2)) (0x6142E0);
void hoo()
{
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)CDetour::Real_Target, (PVOID)(&(PVOID&)CDetour::Mine_Target));
DetourTransactionCommit();
}
I am not sure how to get this to work. The a bow code has two compiler errors:
void (CDetour::* CDetour::Real_Target)(int arg1, int arg2) = (void(CDetour::*)(int arg1, int arg2)) (0x6142E0);
//Error C2440 'type cast': cannot convert from 'int' to 'void (__thiscall CDetour::* )(int,int)'
and:
DetourAttach(&(PVOID&)CDetour::Real_Target, (PVOID)(&(PVOID&)CDetour::Mine_Target));
//Error C2440 'type cast': cannot convert from 'void (__thiscall CDetour::* )(int,int)' to 'PVOID &'
I hope someone can help me in the right direction because I am bout to give up on hooking __thiscall functions...
I am considering writing a global "__declspec(naken) void MyFunc(int, int)" function with inline assembly in order to preserve the "this pointer" as suggested here.

Detours is fairly old. Explicit compiler support for __thiscall is fairly new. Looks like there's support for it in Visual C++ 2005 and later. It seems the Detours documentation was never updated.

Try using a more powerful alternative http://www.nektra.com/products/deviare-api-hook-windows/deviare-in-process/ which is open source.

Related

How to pass method(as callback) as param to method?

I have static method in my Utils class
This is definition
/*static*/ void Utils::copy_files(void(*progress_callback)(int, int),
std::string const & path_from,
std::string const & path_to)
{
....
}
And here using
void TV_DepthCamAgent::progress_callback(int count, int copied_file)
{
printf("Progress :: %d :: %d\n", count, copied_file);
}
void TV_DepthCamAgent::foo()
{
...
shared::Utils::copy_files(progress_callback, path_from_copy, path_to_copy);
...
}
And this is an errors that I get
E0167 argument of type "void (TV_DepthCamAgent::)(int count, int copied_file)" is incompatible with parameter of type "void ()(int, int)"
Error C3867 'TV_DepthCamAgent::progress_callback': non-standard syntax; use '&' to create a pointer to member
What am I doing wrong?
Since you've tagged this C++ i'm assuming you want a C++ solution.
Since C++11 we can use std::function instead of the awkward C style pointer-to-function syntax.
So void(*progress_callback)(int, int) becomes std::function<void(int, int)> progress_callback
In regards to why you get that error it is because to pass a function pointer you must pass the function by reference
...
shared::Utils::copy_files(&progress_callback);
...
You must then pass the required arguments when you call it in copy_files.
You should use std::function and std::bind for this instead of the C style you seem to be writing in

How do I use std::bind when registering a function in Lua?

In the question How to register member function to lua without lua bind in c++ one answer suggested the following code:
class C {
public:
void blah(lua_State* L);
};
C inst;
lua_pushcclosure(L, std::bind(&C::blah, &inst, std::placeholder::_1), 0);
lua_setglobal(L, "blah");
(Quoted as it stood, including the small error in std::placeholders)
However, I cuold not get that to work. The error message I got back states that the function returned by std::bind can't be converted to a lua_CFunction.
I have also tried changing the return type of blah to int, but I get the same error message. If it's helpful to anyone, the full error message is:
Error C2664 'void lua_pushcclosure(lua_State *,lua_CFunction,int)': cannot convert argument 2 from 'std::_Binder<std::_Unforced,int (__thiscall C::* )(lua_State *),C *,const std::_Ph<1> &>' to 'lua_CFunction'
I even tried to change &C::blah to &inst.blah, but that unsurprisingly didn't work either.
Has anyone gotten it to work? Or is it just not meant to work?

Pointer static method that points to no static method

I've been trying to make a pointer function that points to a method doing something like this(visual C++):
struct test
{
int tst(int a)
{
return a * 4;
}
};
// ok, this the visual C++ compiler does not accept it... (mingw accept it)
int(*tstptr)(int) = (int(*)(int))&test::tst;
Then I've done something like this:
struct Dx
{
int SomeMethod()
{
return 4;
}
};
struct Dy
{
static int(*pSomeMethod)();
};
int(Dy::*pSomeMethod)() = (int( Dy::*)())&Dx::SomeMethod;
So far so good, this compiles without problems, but if I try call her:
Dy::pSomeMethod();
The compiler returns me:
Error 1 error LNK2001: external symbol "public: static int (__stdcall
* Dy::pSomeMethod) (void)" (? PSomeMethod#Dy##2P6GHXZA) unresolved
which I do not understand, because it is not suppose pSomeMethod he is not pointing at SomeMethod ?
The type of &test::tst is int (test::*) (int), which is a member function pointer.
You are trying to convert it to a regular pointer type, which is not possible because they are completely different.
That's why you will have this type cast error:
error C2440: 'type cast' :
cannot convert from 'int (__thiscall test::* )(int)' to 'int (__cdecl *)(int)'
int(Dy::*pSomeMethod)() = (int( Dy::*)())&Dx::SomeMethod;
Type Checked, so no complain from compiler.
As for:
Dy::pSomeMethod();
This a __cdecl.
But, SomeMethod is a __thiscall, which means it's really like this
int SomeMethod( Dx &this);
So, linker can't find a match.
You simply can't call non-static method without object of class Dx.
Your declaration of pSomeMethod defines a pointer to a function within class Dy that returns an int. You want to declare it as it appears in the linker error:
int (*Dy::pSomeMethod)();
which is a member of Dy that is a pointer to a function returning an int.
What you're trying to do won't work, since Dx::SomeMethod is a member function of Dx, which needs a this pointer. Calling thru pSomeMethod won't have one.
When done right, you can just assign the address of the function to the pointer without using a cast.

gluTessCallback error C2440

I am trying to use the function gluTessCallback but I get C2440 error. I have no idea why.
Here is the code:
#define callback void(CALLBACK*)()
template<typename T>
class Tessellation
{
private:
GLUtesselator *pTess;
void CALLBACK tessError(GLenum error)
{
sendErrorMessage((char *)gluErrorString(error), true);
}
public:
void Triangulation3D(T* & point, short numOfPoints)
{
pTess = gluNewTess();
gluTessCallback(pTess, GLU_TESS_ERROR, (callback)tessError);
}
};
The error is on gluTessCallback function:
error C2440: 'type cast' : cannot convert from 'overloaded-function' to 'void (__stdcall *)(void)'
Why do I get this compile error?
The error id C2440 on Visual Studio is a type conversion error.
The problem in your code is that you are trying to pass a class method Tessellation::tessError() as function pointer to gluTessCallback(), which expects a pointer to a global C-style function.
A class method is very different from a free/global function and you cannot pass it as a simple function pointer because it needs an object to go along with it every time, the this pointer.
A solution to your problem would be to declare tessError() as a static method, making it effectively the same as a free/global function scoped inside the class, like so:
template<typename T>
class Tessellation
{
private:
static void CALLBACK tessError(GLenum error)
{
sendErrorMessage((char *)gluErrorString(error), true);
}
...
And pass it to gluTessCallback():
gluTessCallback(pTess, GLU_TESS_ERROR, (callback)&Tessellation<T>::tessError);
The only downside to this approach is that a static tessError() can no longer access class variables. Which doesn't seem like a problem to you, since it is not doing so right now, it appears.

c++ pointers to overloaded functions

I'm trying to expose a overloaded function using boost::python.
the function prototypes are:
#define FMS_lvl2_DLL_API __declspec(dllexport)
void FMS_lvl2_DLL_API write(const char *key, const char* data);
void FMS_lvl2_DLL_API write(string& key, const char* data);
void FMS_lvl2_DLL_API write(int key, const char *data);
I'v seen this answer: How do I specify a pointer to an overloaded function?
doing this:
BOOST_PYTHON_MODULE(python_bridge)
{
class_<FMS_logic::logical_file, boost::noncopyable>("logical_file")
.def("write", static_cast<void (*)(const char *, const char *)>( &FMS_logic::logical_file::write))
;
}
results with the following error:
error C2440: 'static_cast' : cannot convert from 'overloaded-function' to 'void (__cdecl *)(const char *,const char *)'
None of the functions with this name in scope match the target type
trying the following:
void (*f)(const char *, const char *) = &FMS_logic::logical_file::write;
results:
error C2440: 'initializing' : cannot convert from 'overloaded-function' to 'void (__cdecl *)(const char *,const char *)'
None of the functions with this name in scope match the target type
what's wrong and how to fix it?
EDIT
I forgotten to mention a few things:
I'm using vs2010 pro on win-7
write is a member function of logical_file
FMS_logic is a namespace
Well the second attemp should work, if write is a pure function. From your code it seems you do have a memberfunction. Pointers to member-functions are ugly, you'd rather use a function object.
However: you would have to post the whole code, it is not clear whether write is a member-function or not.
Edit: if it is a member-function of FMS_logic::logical_file the syntax would be:
void (FMS_logic::logical_file::*f)(const char *, const char *) = &FMS_logic::logical_file::write;
This just applies for non-static member function, i.e. if a function is static or logical_file is just a namespace it is as you wrote it before.
Your code doesn't work because your function pointer type is wrong. You need to include all type qualifiers (your DLL qualifier is missing) and, as Klemens said, the class name. Putting this together, your code should read
.def("write", static_cast<void FMS_lvl2_DLL_API
(FMS_logic::logical_file::*)(const char *, const char *)>
(&FMS_logic::logical_file::write))
Thanks for the hint with the static_cast<>, I had the same problem as you, just without the dllexport, and after adding the static_cast it works :-)