How can I debug a C++ DLL function, called from VBA, using Visual Studio - c++

Question

I have written a DLL function in C++, which I am calling from VBA (Excel).
How can I setup the Visual Studio properties to allow me to debug the function? I have tried specifying Excel, but that doesn’t seem to work.

Answer 1

You have two choices: "direct debug", or "attach".
I strongly prefer the "direct debug" approach for a long list of reasons omitted from here.
There are steps required on both the DLL and Excel/VBA sides, your posting is unclear if all of those steps are addressed.
There are variations on the following:
1) In VS, depending on the version, enter Project Settings, or Project Properties, or equivalent, in the "Debug (not release) Target", go to the Debug or Debugging settings. :
a) There will be an field called "Executable for debugging session", or "command", or something like that depending on VS ver. Here, enter the full path of your Excel exe
b) Optionally, if the same "test spread sheet" is used frequently, enter the full path of your xls (or whatever) in the field called "Command argument", or "program argument" or as in your VS ver.
You may need to surround this with double quotes (e.g. if there are spaces in your path/file names).
c) You can also set the output of your project to a Dir that is "addin helpful", such as a Dir called AddIn (c.f. having the DLL end up in Debug (or Release) Dirs)
... it is assumed that your DLL has all the bits required to export the functions, with the project being of type DLL, plus any DLLEXPORT and compiler directives, etc etc.
... the specifics of the DLLEXPORT settings (and related compiler switches) and Calling Convention will determine many things ... it is assumed you have done all that correctly and consistently (and especially consistently with what the Excel-side is expecting).
... your DLL may or may not have a DLL_Main, if it does, more discussion is required.
2) Before anything else, be sure to have created the Excel-side "interface" for your DLL, ie. the "Add-In". This can be either via .xla, or via .xll. I strongly suggest the .xla route as your first approach.
See the Excel help files etc for creating the .xla
Then, in your XLA's VBA Module(s), declare the functions/subs etc from your DLL. For example, if you have a DLL called Add2_DLL.dll, which contains an exported function "add2_pho_xl", then:
Public Declare Function Add2_Pho_XX Lib "E:\EclipseWorkSpace\Add2_DLL\Debug\Add2_DLL.dll" _
Alias "add2_pho_xl" (A As Double, B As Double) As Double
I have used the Alias approach here, for reasons required below.
In many instances, this declaration can be used directly as User Defined Function (UDF) in your sheets, etc. However, for a vast number of cases, you will need to create a VBA "spinner" function/sub that creates the "ultimate" UDF, and relies on this direct entry function (see below). This is also a very long story, but necessary where more complex matters are required (arrays, variants, etc etc).
NOTICE:
a) the DLL's full path is required unless special steps have been taken. If your Addin is for general distribution ... a much longer discussion is required.
b) the Alias must be the EXACT entry name of the function in your DLL. If you view near the end of the DLL (or .Def) files, and unless you set your DLL modules as Private, those will show the entry names expected on the DLL side.
In this example, the entry name is NOT "decorated" due to the choices in the Calling Convention and compiler switches, but it could look something like
"_add2_pho_xl_#08" etc depending your choices.
... in any case, you must use whatever the DLL has.
3) Once both the .xla and dll exist (it is best if they are in the same Dir), Excel must be "told" about the Add-In. The easiest approach is Excel/Tools/Addins, but there are various strategies for "registering" DLL functions.
4) CRUCIALLY, the argument list properties/declarations MUST BE CONGRUENT with BOTH those in your DLL and the Calling Convention. Three (of many possible) examples of "issues" are,
(i) A Boolean on the VBA-side is two-bytes, if the Bool/Logical on your DLL side is 1-byte, then the Debug will fail, since the two sides "cannot connect" properly.
(ii) Strings ... this can be a very long story. It depends if sent ByVal or ByRef, it depends if the DLL side has "hidden length" Args, etc. etc. Remember, VBA uses COM/OLE VBStrings, which have their own can of worms.
(iii) Variants/Objects: these require a tome onto themselves
5) If ALL (and likely more) of the above have gone well, then in VS set your break points, if required, and "Go" or "Start" the debug (depending on VS ver, etc.). This should launch Excel, and if you also set the target xls, it will launch too. Go to the cell(s) where you addin function (e.g. =add2_pho_XX(A1, B1) ) resides, and execute the cell (sometimes using the "fx" menu item is useful, but there are other ways also).
Some hints:
a) if the func execution crashes/hangs etc Excel and does not even arrive back to the VS side, then likely there is a Arg list conflict (e.g. you are passing a Double to an Int or a million other possibilities), or Calling Convention conflict, etc.
b) In general, you may (while in the VS debug session) simultaneously perform a VBA debug session. That is, after starting the VS bebug, entre the VBA IDE, and set break points in VBA UDF's, if a "spinner" UDF's have been created. For example, a VBA UDF that relies also on the DLL's function.
Private Function Add2_Pho( FirstNum as Double, SecondNum as Double, Optional lMakeRed as Variant) As Variant
'
'
Add2_Pho = Add2_Pho_XX( FirstNum, SecondNum ) ' this is the actual DLL func accessed via Delcare above
'
If( IsMissing(lMakeRed) ) Then
Else
If( cBool(lMakeRed) ) Then
If( Add2_Pho < 0 ) Then
'
' ' make the result Red, or something
'
End If
End If
End If
'
'
End Function
... here, setting a break point at the first line can be helpful to see if the UDF is even entered on the VBA side. If it is, click Continue in VBA, and see if it makes it to the VS side, if not, check Args, Calling Convention, etc again, etc etc
c) If the cell's content are #Value or some other unexpected result, then at least the UDF is "recognised" but not functioning correctly, either due to sheet->VBA issues, or VBA-> DLL issues, or after the return DLL-> VBA
d) Save often! and Use the VBA IDE's Debug/ Compile VBA Project before running anything just make sure VBA internal consistency.
e) Also, if you are using VBA/XLA's, then get a copy of CleanProject (VBA can mess up its internals sometimes, and this Tool will be a life saver)

Answer 2

Please make sure that Debug mode is the active mode.
How to debug your DLL with Excel/VBA