Using C++ app in .NET - c++

Question

I am new to using PInvoke with Classic C++ and I asked this question yesterday: Using Windows API functions in .NET
I have now created a very simple C++ program as follows:
#include <iostream>
#include <stdio.h>
#include <string>
extern "C" __declspec(dllexport) int hello()
{
//printf ("Hello World!\n");
return 1;
}
I then compiled the DLL using the following command: g++ -c mydll.cpp
Then created the shared library using the following command: g++ -shared -o mydll.dll mydll.o, then copied mydll.dll to C:\Windows\syswow64.
I then created a new VB.NET project and created the following code:
Imports System.Runtime.InteropServices
Public Class TestPlatformInvoke
<DllImport("mydll.dll", CallingConvention:=CallingConvention.Cdecl)> _
Public Shared Function hello() As Integer
End Function
Public Sub New()
Try
Dim test As Integer = hello() 'Line 6 ex As Exception
'I don't swallow exceptions
MsgBox("test")
Catch ex As Exception
End Try
End Sub
End Class
The application exits after calling hello(). It does not throw an exception.
I am trying to get to grips with how this works. I don't have any commercial experience with c++; only academic experience at university.
Here is an image of Dependancy Walker for mydll.dll.

Answer 1

Declare is VB6 legacy. You should use p/invoke and the DllImport attribute.
<DllImport("mydll.dll", CallingConvention:=CallingConvention.Cdecl)> _
Public Shared Function hello() As Integer
End Function
There are lots of ways that this could fail. Perhaps the DLL is not loading because there's a 32/64 bit mismatch. Also, your calling conventions do not match. Your DLL will be using cdecl but your VB code uses stdcall. The pinvoke above fixes that.
Most seriously your function does not appear to have been exported from the DLL. That's going to make it fail for sure. Use Dependency Walker to determined whether or not your function has been exported. I'm not so familiar with g++ so you'll have to work out how to export your function using the GNU toolchain. But watch out for name decoration and name mangling. You may need to take care with how you export your function so that it is exported with the desired name.

Answer 2

If you use Visual Studio to debug programs and include the DLL in your project (Visual Studio 2012 Express Edition allows including projects of different types in the solution), you may set the "Allow native code debugging" option to automatically switch the debugger from the VB.NET to C++, when you P/Invoke function from the DLL.
Here's a working Proof-of-concept. Simply unzip, open in Visual Studio, build and run. You may compare this program with your project and find the differences, which makes your code failing.
I guess, that you haven't actually exported the function from the DLL. Simply including it in the DLL's code may not be enough: check, which functions are exported from the DLL using the Dependency Walker (other link) program. For example, I had to add folowing declaration:
__declspec(dllexport) int __stdcall Test() { ... }
And additionally create the .def file for the names for exported functions not to be decorated.
Try to modify your DLL source code and change the function declaration to:
extern "C" __declspec(dllexport) int hello()
{
return 1;
}