C++ calling conventions -- converting between win32 and Linux/GCC? - c++

I'm knee deep in a project to connect Windows OpenVR applications running in Wine directly to Linux native SteamVR via a Winelib wrapper, the idea being to sidestep all the problems with trying to make what is effectively a very complicated device driver run inside Wine itself. I pretty much immediately hit a wall. The problem appears to be related to calling conventions, though I've had trouble getting meaningful information out of winedbg, so there's a chance I'm way way off.
The OpenVR API is C++ and consists primarily of classes filled with virtual methods. The application calls a getter (VR_GetGenericInterface) to acquire a derivative class object implementing those methods from the (closed source) runtime. It then calls those methods directly on the object given to it.
My current attempt goes like this: My wrapped VR_GetGenericInterface returns a custom wrapper class for the requested interface. This class's methods are defined in their own file separately compiled with -mabi=ms. It calls non-member methods in a separate file that is compiled without -mabi=ms, which finally make the corresponding call into the actual runtime.
This seems to work, until the application calls a method that returns a struct of some description. Then the application segfaults on the line the call happened, apparently just after the call returned, as my code isn't anywhere on the stack at that point and I verified with printfs that my wrapped class reaches the point of returning to the app.
This leads me to think there's yet another calling convention gotcha I haven't accounted for yet, related to structs and similar complex types. From googling it appears returning a struct is one of the more poorly-defined things in ABIs typically, but I found no concrete answers or descriptions of the differences.
What is most likely happening, and how can I dig deeper to see what exactly the application is expecting?

Related

Generating a tree of all possible call stacks

I am trying to tinker with some library code written in C++. A fairly complex application sits on top of the library. To tinker with the code, I often need to understand how a library function has been used throughout the codebase, and make sure that I am not breaking any downstream clients.
Suppose foo() is exported from my library's dll. In client code, bar() calls foo(), and baz() calls bar(). I need to make sure that bar and and baz both work after my changes. In my case, the call stack actually is quite deep, and not easy to manually trace because there is not one call stack, there are numerous ways my library function can land at the top of a call stack.
Using either Visual Studio, or g++, or clang, is there a way to generate a tree such that my library function is at the root, and the branches are all the various ways my function can land at the top of the call stack? I mean does such a feature already exist in one of the popular toolchains? If not, do you know any other way of generating such a tree?
I don't think any of the compilers have options to generate this information.
In the general case, there are many confounding factors that would make this very difficult:
If there's recursion in the code, then the tree you want is actually a graph/network with cycles.
Virtual methods, function pointers, and member function pointers probably make this the equivalent of the halting problem. If you have two concrete classes A and B that share a common base class that offers virtual method foo(), then you'd have to do exhaustive analysis to determine whether a particular call of foo() through a pointer or reference to the base class should be counted as a call to A::foo() or B::foo() or both. Ditto for the various flavors function pointers.
If you rely on system or other third-party libraries that can call back into your code, you'd better have source for them. For example, a Windows GUI program typically has window procedures that are called from system code, possibly in response to a call from your code into the system. Since you don't wouldn't have the windows sources, you'd have to assume that any and all of your callbacks could be called at any time, and thus your "tree" would have many roots.
The modern way to deal with this is not to analyze all the ways your library can be called, but to document all the ways it should be called. Build a test suite that calls the library in all the reasonable ways you want to support. Then you can tinker and then run your test suite to see if you've broken the library's contract. If, in integration testing, you find a client of the library that's broken by your changes, it indicates the test suite is incomplete or the client is calling the library in an unsupported way.

Hook functions class in offset (from .exe in a .dll)

I have a .exe application and I need to create some customizations to this executable, so I need to hook a dll in it for the changes to be loaded. Until then, everyone knows.
The scenario is this:
Hook(0xOffset, &myClass::myFunc);
There is a class in .exe that I need to rewrite completely and I've done that in my dll, but I'm having trouble with the hook of the functions of class, they aren't static. I've read many topics and I could not implement it with any method presented. In some cases, the compiler will not accept, in others cases has accepted but the .exe could not find the actual address of the function.
A friend gave me a solution, but it is a little confusing to understand how I can call the function there and from what I saw would be very big in my source code and many loops, so to speak.
Could help me?
Member functions are indeed far more complex. You have to deal with normal inheritance, multiple inheritance, and virtual inheritance; with direct calls and virtual calls. Possibly the worst is dealing with member function pointers, which are entirely unlike normal function pointers.
As a result, many solutions deal only with the easy cases. It's perfectly normal that a solution capable of dealing with all edge cases takes a lot of code.

C# / C++ Asynchronous reverse pinvoke?

I need to call C# code from a native C/C++ .dll asynchronously.
While searching how to do I found that I could create a C# delegate and get a function pointer from it, which I would use inside my native code.
The problem is that my native code needs to run asynchronously, i.e in a separate thread created from the native code, which means that the native code called from the C# will return to C# before the delegate is called.
I read in another SO question that someone had trouble with this, despite what the MSDN says, because his delegate was garbage collected before it gets called, due to the asynchronous nature of his task.
My question is : is it really posible to call a C# delegate using a function pointer from a native code running in a thread that was created inside the native code ? Thank you.
No, this is a universal bug and not specific to asynchronous code. It is just a bit more likely to byte in your case since you never have the machinery behind [DllImport] to keep you out of trouble. I'll explain why this goes wrong, maybe that helps.
A delegate declaration for a callback method is always required, that's how the CLR knows now to make the call to the method. You often declare it explicitly with the delegate keyword, you might need to apply the [UnmanagedFunctionPointer] attribute if the unmanaged code is 32-bit and assumes the function was written in C or C++. The declaration is important, that's how the CLR knows how the arguments you pass from your native code need to be converted to their managed equivalent. That conversion can be intricate if your native code passes strings, arrays or structures to the callback.
The scenario is heavily optimized in the CLR, important because managed code inevitably runs on an unmanaged operating system. There are a lot of these transitions, you can't see them because most of them happen inside .NET Framework code. This optimization involves a thunk, a sliver of auto-generated machine code that takes care of making the call to foreign method or function. Thunks are created on-the-fly, whenever you make the interop call that uses the delegate. In your case when C# code passes the delegate to your C++ code. Your C++ code gets a pointer to the thunk, a function pointer, you store it and make the callback later.
"You store it" is where the problem starts. The CLR is unaware that you stored the pointer to the thunk, the garbage collector cannot see it. Thunks require memory, usually just a few handful of bytes for the machine code. They don't live forever, the CLR automatically releases the memory when the thunk is no longer needed.
"Is no longer needed" is the rub, your C++ code cannot magically tell the CLR that it no longer is going to make a callback. So the simple and obvious rule it uses is that the thunk is destroyed when the delegate object is garbage collected.
Programmers forever get in trouble with that rule. They don't realize that the life-time of the delegate object is important. Especially tricky in C#, it has a lot of syntax sugar that makes it very easy to create delegate objects. You don't even have to use the new keyword or name the delegate type, just using the target method name is enough. The lifetime of such a delegate object is only the pinvoke call. After the call completes and your C++ code has stored the pointer, the delegate object isn't referenced anywhere anymore so is eligible for garbage collection.
Exactly when that happens, and the thunk is destroyed, is unpredictable. The GC runs only when needed. Could be a nanosecond after you made the call, that's unlikely of course, could be seconds. Most tricky, could be never. Happens in a typical unit test that doesn't otherwise calls GC.Collect() explicitly. Unit tests rarely put enough pressure on the GC heap to induce a collection. It is a bit more likely when you make the callback from another thread, implicit is that other code is running on other threads that make it more likely that a GC is triggered. You'll discover the problem quicker. Nevertheless, the thunk is going to get destroyed in a real program sooner or later. Kaboom when you make the callback in your C++ code after that.
So, rock-hard rule, you must store a reference to the delegate to avoid the premature collection problem. Very simple to do, just store it in a variable in your C# program that is declared static. Usually good enough, you might want to set it explicitly back to null when the C# code tells your C++ code to stop making callbacks, unlikely in your case. Very occasionally, you'd want to use GCHandle.Alloc()instead of a static variable.

Getting to know the caller is from which DLLs

Currently, I have a C++ exe project, which dynamic load N DLLs.
Those DLLs will perform calling to the functions which is re-inside exe project.
Now, within my exe project, I wish to know the callers are coming from which DLLs.
Is it possible to do so using any available Windows API?
It depends on what your actual goal is. You cannot do it if you're expecting the DLLs to be possibly malicious (that is, if you're expecting them to try to trick you). But if it's just for debugging or logging or something relaitvely harmless like that, you can look at the stack and get the address that the ret instruction will use to return to the caller, enumerate through the loaded DLLs and test which of them that address is inside of.
To get the "return address", you can use the _ReturnAddress intrinsic in Visual C++, and then you can use the GetModuleHandleEx function, passing in GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS to get a handle to the DLL that the address is inside of.
But I must repeat: you cannot base security decisions off the results of this test. It is very easy for malicious code to fake and "trick" your program into thinking it's a "trusted" or "safe" DLL. As I said, if it's just for debugging or logging or something, then go right ahead.
Also, this will obviously only tell you the DLL the immediate caller is inside of. You can't do it if you're 5 levels deep or something....
If you have given the same callback to multiple DLL's, then it is up to them to provide you with information about who's who. Most API callback have a parameter you can pass to the callback. If this is so for your callbacks, you can use this to identify the DLLs.
It probably isn't possible considering that the call stack will come back down to your exe anyway.
EDIT: By the look of your post, is this a hypothetical situation?
Is this helpful? Check the parameter 'GetModuleBaseRoutine'
If you're architecting the exe, and you're not assuming the DLL's are hostile (see Dean's answer), you might be able to achieve the effect by providing each DLL with a different set of pointers for the callback functions, which each in-turn forward to the actual callback functions. You could then associate the calls with the calling DLL, based on which pass-through callback was actually called.
Of course, this assumes you're providing the callback addresses to the DLL's, but presumably this would be the normal design for an application where a DLL called back into the calling exe. It won't work if the DLL is mucking around in your process memory for internal functions, of course, but then you're probably into the hostile situation.

Problems regarding Boost::Python and Boost::Threads

Me and a friend are developing an application which uses Boost::Python. I have defined an interface in C++ (well a pure virtual class), exposed through Boost::Python to the users, who have to inherit from it and create a class, which the application takes and uses for some callback mechanism.
Everything that far goes pretty well. Now, the function callback may take some time (the user may have programmed some heavy stuff)... but we need to repaint the window, so it doesn't look "stuck".We wanted to use Boost::Thread for this. Only one callback will be running at a time (no other threads will call python at the same time), so we thought it wouldn't be such a great deal... since we don't use threads inside python, nor in the C++ code wrapped for python.
What we do is calling PyEval_InitThreads() just after Py_Initialize(), then, before calling the function callback inside it's own boost thread, we use the macro PY_BEGIN_ALLOW_THREADS and, and the macro PY_END_ALLOW_THREADS when the thread has ended.
I think I don't need to say the execution never reaches the second macro. It shows several errors everytime it runs... but t's always while calling the actual callback. I have googled a lot, even read some of the PEP docs regarding threads, but they all talk about threading inside the python module (which I don't sice it's just a pure virtual class exposed) or threading inside python, not about the main application calling Python from several threads.
Please help, this has been frustrating me for several hours.
Ps. help!
Python can be called from multiple threads serially, I don't think that's a problem. It sounds to me like your errors are just coming from bad C++ code, as you said the errors happened after PY_BEGIN_ALLOW_THREADS and before PY_END_ALLOW_THREADS.
If you know that's not true, can you post a little more of your actual code and show exactly where its erroring and exactly what errors its giving?