How to debug a DLL project using QtCreator? - c++

I have written a non-Qt C DLL compiled with mingw using the QtCreator IDE.
When I inject the DLL into a process, the DLL causes that process to crash at a certain line of code. I found this line to be the culprit through the use of OutputDebugString. I know how to fix the line and the DLL works when that line is modified. However, for the purpose of learning how to use a debugger, I have left the line broken and unmodified.
How would I use the QtCreator debugger to find that same exact line is causing the problem? When QtCreator is set to compile in debug mode, pressing F5 results in the following dialog because there is no EXE:

The solution is to run the debugger attached to the executable before injecting the DLL. The DLL must be built in debug configuration and its project needs to be opened. Attach the debugger using instructions here and inject the DLL. If executable crashes you should see the stack trace and your DLL code when you click on selected stack frame. But be careful: if you corrupted the memory of the process the stack trace might be incomplete or incorrect.

Related

How to debug an external process under C++ Builder?

I have a DLL that is written in C++ Builder. The DLL is built with "debug info" on.
I cannot run and debug this DLL from the IDE because it is loaded dynamically as a plugin in another process (main exe). My DLL needs to be copied into the main exe's folder first.
So, I attached the IDE to that process but I don't know what to do from here.
I know that if I click a button in the main process to load and use my DLL an AV is raised and a custom message is shown with some minor into about the error. The debugger won't step in when the AV is raised. Probably the error is caught at a higher level.
It would be nice if I could also attach the source-code of my DLL and put a breakpoint in it.
How do I get more info about that AV (its origins)?
(Basically any hint from those that did this type of debugging would be very helpful).
I have a DLL that is written in C++ Builder. The DLL is built with "debug info" on.
I cannot run and debug this DLL from the IDE because it is loaded as a plugin in another process.
Yes, you can.
Open the DLL project in the IDE, go into the project's Run parameters and set the desired EXE as the project's Host. This way, when you "run" the DLL project for debugging, the IDE will execute the Host instead and attach the debugger to that process. When the Host process loads your DLL into memory, you can then step through and debug the DLL's code as needed.
If the Host process is already running before you start your debugging, you can simply Attach the debugger to the Host process manually before it loads your DLL, and then the debugger will still be able to step through the DLL's code once the Host loads the DLL into memory.

How to debug program crashing before main()

I'm using QtCreator with the visual studio 2015 kit on windows 8.1 to build a program I developed and tested on Linux, on linux it works correctly, but on windows it's just crashing immediately, and I have no idea what to look for.
The only external libraries, aside from QT I'm using are opengl and glew, so I don't think it's those.
Is there anything that's known to work in GNU C++ but crash immediately in MSVC?
Usually this kind of crashes have absolutely nothing to do with your program. It's an external library linking issue. I had this issue recently with the OpenSplice DDS library. I linked to a library that caused a segmentation fault before anything started. I resolved the issue by linking the pre-compiled libraries 1-by-1, and check each if that fixes the program.
What I recommend you to do is: Remove the libraries and resources you're linking to gradually, until your program starts and prints "Hello world" from the first line of main().
Another way to go is, make a new empty program, and link the same resources you're using in your program. This is easier, as it doesn't involve modifying your program.
This is what I would do.
Start by rebuilding the entire solution or project from a clean state. Just in case this is just some weird dependency issue that resulted in something not getting recompiled. Never hurts.
As Neil said in the comments for the question, the crash is possibly coming from a global variable who's constructor runs before main or WinMain. Are you sure you don't have something declared as "static" or at global scope that might have a constructor?
Now do the following:
Open Visual Studio.
From the menu, select File->Open->Project/Solution...
When the file open dialog pops up, select the EXE produced by Qt
Creator. (That's right - you are opening the EXE as a project). This directory is typically one folder level above the Qt project (..\build-yourapp-Desktop_Qt_5_7_0_MSVC2015_32bit-Debug\debug)
Now press the green arrow to start debugging (menu->Debug->Start
Debugging). If all goes well, your program will fail early and
Now chances are high that the program is not going to run at all under Visual Studio because Qt Creator doesn't copy all the Qt*.dll binaries to your build directory. You'll get a bunch of dialogs popping up saying that "The program can't start because Qt5-XYZ.dll can't be found". This is easily fixed by updating your PATH environment in any of the following way to include your Qt5.x.0\5.x\msvc2015\bin folder to your PATH.
You add it from the command linke and then re-launch devenv.exe from the command line.
You can add it globally from Control Panel->System->Advanced. Then restart Visual Studio from the Windows desktop.
With the EXE debug project open from within Visual Studio, just right click on the project name (not parent solution) and a dialog will popup that allows you to edit startup settings. One of which is the Environment.
And that should do it. From there you can start the debugger on your EXE, set breakpoints as needed, and analyze the call stack on crash.
It's really easy: build all the libraries you use, including Qt, with debug information (those can be release builds as long as the PDB files are generated). Then run your application under a debugger (e.g. F5 under Qt Creator), and it will stop at the point of the crash.
The code that runs before main and is known to cause trouble will be the global object initialization: you're likely running into the static initialization order fiasco.
Another cause for the problem could be stackoverflow. In Linux, the stack size by default is usually 8 MB whereas in Windows it's just 1 MB.
Try to link with /STACK:8388608 switch. If it works, you might consider allocating more data on the heap and stay with the default stack size of 1 MB.

How to set C++ breakpoint in eclipse when source is compiled with ccache?

Recently, our development team is starting to use ccache to do faster compile (the compile is done from sandbox /usr/x).
Now, when I compile from my sandbox (/usr/y), and try to set a breakpoint in the code in Eclipse (GDB (DSF) process launcher), it fails to find the file.
Further investigation shows that Eclipse gdb uses the complete path of the file to set a breakpoint (e.g. b /usr/y/untouchedFile.cpp:1234), but the actual path (in the gdb debugger) is actually /usr/x/untouchedFile.cpp.
The only thing that works is to set a breakpoint on the console by typing it, and do a source file mapping when the breakpoint is hit.
I would like to set the breakpoint by clicking on the code line (which used to work before ccache).
I was wondering if there is a way to get around this.
Thanks!

How to debug a DYLIB in XCode?

I am an experienced Visual Studio developer who has recently taken on an OSX 10.6 project (a C++ server project with no UI).
I have been successfully debugging the application using the XCode debugger (setting breakpoints, etc.) for months, including debugging the source code for various static libraries that are linked into the final executable program.
However, tonight I was required to debug (with breakpoints) a DYLIB that is also built from our source code, but that is linked dynamically at runtime with the application (the name of the DYLIB is read from an .ini file by the main application).
Unfortunately, the usual method I use of debugging the application (right-clicking the custom executable and selecting "Debug with Breakpoints"), though it does successfully run the debugger and allow me to debug the application (along with its statically linked libraries), exhibits the following undesired behavior when I attempt to hit a breakpoint in the source code for the DYLIB:
-> The XCode debugger reports that the breakpoint was hit in the sense that I see the function and line number in the status bar at the bottom of the XCode windows (along with an indication that this is a gdb message), and the application halts execution. However, there is no stack trace, no variables, nothing - just a completely empty debugger window. The buttons to to "step over", "step into", etc, are disabled. No editor window appears in the debugger (and hence no visual indication that the debugger has stopped on the line indicated). Opening the file by hand does not reveal the debugger hitting the line.
Unfortunately, this is useless for me as far as my attempts to debug the DYLIB.
I have hunted far and wide tonight researching and attempting to find a way for the XCode debugger to successfully hit breakpoints in a meaningful way in the source code for this dynamically linked DYLIB. I have of course done a number of clean/rebuilds. I have made certain that "load symbols lazily" is unchecked and then cleaned/rebuilt. I have restarted, and I have also deleted the "build" directory and rebuilt. I also deleted the user-specific files in the .xcodeproj package. (Note also that I am of course building and running all code, including the DYLIB code, in Development mode with all optimizations off, and generating debug symbols for all.) However, my attempts have been unsuccessful. Nor can I find so much as a single mention of this problem on internet forums.
Any help in instructing me how to use XCode to successfully debug a DYLIB that is linked to my application would be appreciated.
Thanks,
Dan.
Update -
This problem is resolved. It was my lack of experience with OSX that caused me to fail to see this. Despite the fact that my DYLIB project was part of the same XCode project as the executable that calls it, and despite the fact that the DYLIB was built in the same directory as the executable, at runtime the debugged application was not accessing the DYLIB from this location. Instead, it was accessing it from a (different) install location. I have not as of this moment tracked down where the install location is "cooked" into the application, but by copying the final executable/DYLIB into the expected install location and creating a new custom executable that points to the executable in this location, debugging of both the DYLIB and the executable works.
Thanks,
Dan.

Debugging InProc COM Dll

I have a project in VC++ 6.0 where there is an exe and a InProc COM Dll. I want to be able to place a breakpoint somewhere in the InProc COM DLL, but VC++ won't allow me to set a breakpoint.
I have the source code for this DLL, however I cannot figure out how I can place a breakpoint in the code and the debug it?
Can someone help me.
It has been some time since I worked with COM but IIRIC, inside your COM project configure your executable as the launching application. It should work (sorry, I don't have VC++ 6.0 installed here anymore :().
If it doesn't work, you can try to attach the debugger to the running application.
In either cases make sure you have full debug information in your COM server.
Hope this helps.
Attach to the process
Open Project->Settings (Alt+F7)
Open Debug tab, category Additional DLLs
Add you in-proc server DLL
Save .opt file on closing the debugger
This way next time you attach to process or manually open the .opt file, your in-proc server DLL gets loaded, its PDB gets parsed, last open source files get loaded, breakpoints get loaded.
The reason why "additional dlls" setting is needed here is because you in-proc server doesn't get loaded until an instance of his is CoCreated. So the debugger doesn't load its PDB file and the source files are treated as unknown text files, so the breakpoints in them get inactive (white).
Two things you can look into
Uncheck Require source files to exactly match the original version in the Debugging options dialog
If that fails, compile the DLL again (preferably with optimizations disabled /Od) and use the new DLL with its PDB file.
Not sure if this will work in VC6 but you could try _asm int 3 where you want the code to break, this should cause a breakpoint in your code and allow you to debug it.