Debugging 64bit DLL inspector showing incorrect values - c++

I've built a 64 bit version of the OpenH264 DLL in debug mode (with no compiler optimisations) and am calling it in C# (via Unity). I'm using Visual Studio 2017 to attach to the running Unity process and debug my dll.
I can place breakpoints and step through the code, however all values for the code seem to be random (or sometimes null).
For example this is where the debugger reports NULL as the value of a pointer but is stopped on the line after a failed NULL check.
Why is this happening and what can I do to get correct values when inspecting variables?

Maybe due to "At this time, Visual Studio Tools for Unity only supports managed DLLs. It does not support debugging of native code DLLs, such as those written in C++."

1 Please try to switch the Common Language Runtime Support under Configuration Properties :
2 Another potential reason is that the variable is not in the current context:
While debugging, Visual Studio will keep track of every variable in the current context

Related

Cannot see variables when debugging a x64 bit DLL in Visual Studio

I am trying to step through a C/C++ program (MapServer) in Visual Studio, but I cannot see the local variables.
I created a new project by adding the main .exe I built from source. I can set break points and step through the program without problem, but I cannot see what is stored in any of the variables. I am running all this on a Virtual Machine - the host OS is Windows 10, and the VM is Windows 7.
The program is:
built as a debug release
has no optimizations
the symbols load fine
I am fairly certain it is due to it being a x64 bit release, as I can use the exact same approach for a x32 build and see the variables.
Example of the x32 debugger (the first 5 variables are correct - the others are unset):
Example of x64 debug session (note the program works fine):
I thought it may be due to VS2008, so I also tried in VS2015, but had similar (failing) results:
Trying to access variables in the Immediate Window produces:
// working VS2008 x32 build
map->name
0x00ffcb40 "WFS_server"
// VS2008 x64 build
map->name
0x0000000000000000 <Bad Ptr>
// VS2015 x64 build
map->name
0x0000000000000000 <NULL>
Am I missing some debugger setting in Visual Studio to set the debug project to x64? Or is there some casting issue in the source code that produces this?
Any pointers appreciated..
The above was caused by stdrup being deprecated on Windows, and the heap becoming corrupted.
The MapServer issue on GitHub can be seen at https://github.com/mapserver/mapserver/pull/5277
This question also describes a similar issue: Heap corruption with strdup

Source code is different from original version?

I'm setting break points in Visual Studio before running the Local Windows Debugger, and they all say this when the debugger is running, with a small warning label on each break point. I can't see why this would be happening; people have talked about using different versions of Visual Studio and getting this error, but I made this project earlier today, pasting the base code from my professor into a new project created on my copy of Visual Studio 2012.
I found this when trying to fix a strange error in my program that wouldn't go away even though I manually fixed things (it was a string error that claimed I was using an index outside the bounds of the string; setting this value to 0 explicitly did not fix it) so I presume that this is the actual culprit.
Make sure you're properly building the code (Build Solution or Rebuild Solution).
In the output panel you should now be able to locate where the binaries are located.
Make sure you're debugging the said binaries by looking in the Debugging page of your project properties, the Command property should most likely be set at $(TargetPath).
Other things to look for. The project should be "Set as startup project", the program database option should be activated (by default). Don't modify the source after you started debugging.

64 bit exe crashing outside visual studio but working inside visual studio

I compiled a program using freeglut, optix, cuda and other libraries (some of them dinamically loaded). It compiles and runs without problems in Visual Studio but it crashes if I execute it outside Visual Studio. Both release and debug versions work within VS, they both crash without any information on Windows 8 if I try to execute them directly.
I already included all the necessary DLLs, that didn't work.
What could be the problem?
Most such observations are usually coming from undefined behavior -- using uninitialized variable, dangling pointers/refs, overrunning buffer.
You may try to use Application Verifier, with some luck it might rearrange the used memory enough for you to trigger the problem under debug to help corner it.
Also, when it crashes you should get a prompt to launch VS and inspect the problem -- did it not indicate a hint? What was the immediate cause of the crash and what you had on the call stack there?
You can try comparing the environments between visual studio and the default environment.
Dependency walker should identify any missing DLLs.
Get WinDBG, then File > Open Executable and run the program under WinDBG. When it crashes, you will get some more information. My answer here describes an issue in .net, but the concept applies to native C++ as well.
Visual studio runs executables under "debug" mode, meaning a debugger is present.
What does this mean? If you check out the msvcrt implementation, if the runtime detects a debugger is present (IsDebuggerPresent), then heaps preform differently.
What does this mean? It means buffer sizes are "nudged" upwards, it means memory allocations are wiped clean by default (no need to memset), etc.
This can cause a variety of bugs to manifest, or some more subtle bugs to be hidden.

C++ 64bit, variable not found

I have a problem with my C++ application. It was developed on a 32bit pc, on Microsoft Visual Studio 2008, and now I am trying to run it on a 64bit pc.
On my 32bit pc it works fine; on the 64bit pc, Visual Studio does not give any compilation problem, but then on execution gives wrong results.
And I have undestood why.
In the code, I define a variable, of tipe "dag", that is a structure for a direct acyclic graph. By debugging the software, I noticed that, although I declared it, later the software is not able to insert data in it, and the debugger says:
CXX0017: Error: symbol "dags" not found
Here's my code:
Dag<int64_t>* dags = new Dag<int64_t>();
dags = getDagsFromRequest2(request, dags);
The very strange thing is that, if I follow the flow inside getDagsFromRequest2() function, I can clearly see that dags variable is full of data: on "quickwatch", it shows 2342 nodes inside it. But when I come back from getDagsFromRequest2() function to this part of the code, debugger says "CXX0017: Error: symbol "dags" not found". How is it possible?
You can also see this screenshot from my Visual Studio debug set.
What could be the problem?
Thanks a lot
There are a few possibilities to consider:
Running in Release builds. Switch to a Debug build.
Using a Debug build that has optimizations enabled and/or debug information disabled. Disable the optimizations and enable the debug information (look in another project for the relevant settings).
A corrupt build of some sort. Clean and rebuild the entire solution.
Memory corruption which is preventing the debugger from displaying the variable. Ensure that no memory issues exist with a tool like Valgrind.
A VS bug. This report for VS2010 seems to suggest a known bug with similar characteristics for example. Ensure all patches and hotfixes for VS2008 are installed.
The variable dags is defined as your code compiles. The error you see is simply related to the debugger. I am guessing it is caused by running the application in Release mode which sometimes causes confusing and wrong watches values. Try changing the mode to debug(there is a drop down from which you can choose the build mode).
EDIT: as you say you are running in Debug mode, my next guess is that this behavior could be caused by stack corruption. Try using valgrind to detect if that is the case. It may take a while to start with it,but it is worth it and will detect if you have some memory corruption.

Tool for debugging Borland & Visual Studio applications

sometimes I have to debug an application that was written with Borland C++ Builder. This application loads dlls compiled with Visual C++. Is there a debugger that can debug both parts of the application? Currently I have to decide - either I can easily set break points and see the source in Visual Studio or I have to start Borland C++, but I can't work with the source from the Visual-Studio compiled dll.
thank you for your help,
Tobias
You could try OllyDbg - version 1.x does not seem to support the latest Win version but there is also the 2.0 although it's still in alpha state(haven't tried myself that one yet).
EDIT - clarification:
Source debugging OllyDbg reads debugging information in Borland and
Microsoft formats. This information includes source code and names of
functions, labels, global and static variables. Support for dynamical
(stack) variables and structures is very limited.
The above is take from here.
UPDATE:
I'm not familiar with the Borland C++ Builder but at this link you can find some articles explaining how to deal with some interoperability issues between Borland and MS that might be of help.
if both parts built using ulink linker and have debug info you could try cdb32 debugger (from the ulink linker author)
cdb32 is still in its alpha stage though and I personally never tried such "mixed" debugging
Have you tried loading the DLL code in VS, loading the app code in BCB, and having both debuggers attached to the same running process at the same time? Not sure if Windows will allow that, but it might be worth a try.
I suspect there is no perfect answer to your question, you are going to have to compromise in some way, as I'm sure you are already doing.
I have a similar problem to yours at work. The applications that I work on are written in Python instead of Borland C++, but like your situation, these apps rely on a rather large Visual Studio compiled DLL for some functions.
My method of debugging these applications involves a combination of two debugging strategies: the use of an interactive debugger and the so called "printf" debugging technique.
What I basically do is pick one of the two areas as my main debug focus, and that determines my debugging approach:
If for a given situation I decide that I need to debug the DLL with greater detail, then I work with the VS debugger. I set the executable to run in the DLL project as my python script and that enables full debugging of the DLL code. If I need debugging support from the Python side, then I add print statements. If I need a breakpoint on the Python side to inspect some values, I just print all those values and immediately after call a C++ function that does nothing, but that has a breakpoint set in VS.
When I need to concentrate more on the Python side more I use a Python interactive debugger, but I have VS with the DLL project loaded on the side so that I can quickly add any necessary printfs on the DLL and recompile, so essentially the reverse of the above.
I know it's not the answer you expect, but it is a decent solution in my opinion.
It looks that it is possible to convert the debugging information generated by C++ Builder to a format understood by WinDbg (link to discussion). If so you could use it to debug both parts of your application (I haven't tried this though).
you can convert the .map files to Microsoft's debug file format
http://code.google.com/p/map2dbg/
now you can use Windebug; there is also a tool mentioned to convert to pdb format, so you could try the vc++ debugger