My app is crash. So I am using windbg to check the trace log. Here is my trace log in windbg:
FAILED_INSTRUCTION_ADDRESS:
cbmwk5!unloaded+161c0
727261c0 ?? ???
ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) amd64fre
LAST_CONTROL_TRANSFER: from 423c1d76 to 72725817
IP_MODULE_UNLOADED:
cbmwk5!unloaded+15817
72725817 ?? ???
PRIMARY_PROBLEM_CLASS: BAD_INSTRUCTION_PTR_NULL
DEFAULT_BUCKET_ID: BAD_INSTRUCTION_PTR_NULL
STACK_TEXT:
023eebfc 72725817 cbmwk5!unloaded+0x15817
023eec00 423c1d76 unknown!unknown+0x0
023eec30 7272590a cbmwk5!unloaded+0x1590a
023eec58 72723a39 cbmwk5!unloaded+0x13a39
023ef120 76b65762 shell32!SHGetFolderPathW+0x180
023ef128 72720813 cbmwk5!unloaded+0x10813
023ef144 7271110e cbmwk5!unloaded+0x110e
023ef164 72715916 cbmwk5!unloaded+0x5916
023ef59c 7271636b cbmwk5!unloaded+0x636b
Could you please help me how to check the size of cbmwk5.dll according to STACK_TEXT?
What is the meaning of "+0x15817" int the statement:
023eebfc 72725817 cbmwk5!unloaded+0x15817
I tried to reload by using command:
.reload /unl cbmwk5.dll
and then type: !analyze -v
but the error missing cbmwk5.dll occurs:
SYMSRV: c:\localsymbols\cbmwk5.dll\506DCE083b000\cbmwk5.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/cbmwk5.dll/506DCE083b000/cbmwk5.dll not found
DBGHELP: C:\Program Files (x86)\Windows Kits\8.1\Debuggers\cbmwk5.dll - file not found
DBGENG: cbmwk5.dll - Image mapping disallowed by non-local path.
DBGHELP: No header for cbmwk5.dll. Searching for dbg file
DBGHELP: .\cbmwk5.dbg - file not found
DBGHELP: .\dll\cbmwk5.dbg - path not found
DBGHELP: .\symbols\dll\cbmwk5.dbg - path not found
DBGHELP: cbmwk5.pdb - file not found
*** WARNING: Unable to verify timestamp for cbmwk5.dll
*** ERROR: Module load completed but symbols could not be loaded for cbmwk5.dll
DBGHELP: cbmwk5 - no symbols loaded
DBGHELP: C:\Program Files (x86)\Windows Kits\8.1\Debuggers\cbmwk5.dll - file not found
SYMSRV: C:\Program Files (x86)\Windows Kits\8.1\Debuggers\x64\sym\cbmwk5.dll\506DCE083b000\cbmwk5.dll not found
Thanks a lot.
What is the meaning of "+0x15817" int the statement
+0x15817 means that the debugger has no clue whatsoever which function was called. It just doesn't know anything about the DLL, only where it was once loaded. So it can only annotate the address with the DLL name and a very large offset. Otherwise visible from the SYMSRV trace messages, the debugger made an attempt to download the PDB file for the DLL but the symbol server doesn't know anything about the DLL. Which is certainly not unusual, it is a 3rd party DLL, not Microsoft's. Even Google has never heard of it.
FAILED_INSTRUCTION_ADDRESS:
cbmwk5!unloaded+161c0
The unloaded annotation is your strongest clue to the problem. The code crashed because the DLL was unloaded from memory. Yet the program tried to call it anyway. With no code left to execute (note the ??), the processor is forced to give up and generates an access violation. That was the end of the program, it cannot continue operating.
023ef120 76b65762 shell32!SHGetFolderPathW+0x180
The stack trace gives a (weak) clue to the underlying problem. Beware that this is speculation. But the presence of shell function like SHGetFolderPathW() is a strong hint that this is a misbehaving shell extension. They can do a lot of damage since they tend to be injected into your program when you use one of the common shell dialogs, like OpenFileDialog. In other words, it doesn't have anything to do with your program, it is somebody else's crappy code that made the program bomb.
You fix this kind of problem by disabling shell extensions one by one until the problem disappears. SysInternals' AutoRuns utility is the weapon of choice. It has to be done by the machine owner, there's little that you can do but give advice.
Related
On Linux and Mac, when using dlopen() to load a shared library that links to another library, if linking fails because of a missing symbol, you can get the name of the missing symbol with dlerror(). It says something like
dlopen failed: cannot locate symbol "foo"
On Windows, when using LoadLibrary() to load a DLL with a missing symbol, you can only get an error code from GetLastError() which for this type of issue will always be 127. How can I figure out which symbol is missing, or a more verbose error message from LoadLibrary() that explains why the function failed?
I figured out a way using the MSYS2 terminal. Other methods might work with GUI software.
A major caveat is that this can't be done in pure C/C++ and released for end users. It's for developers only, but it's better than nothing.
Install Debugging Tools for Windows by downloading the Windows SDK and unchecking everything except Debugging Tools.
I could be wrong, but it seems that installing this software installs a hook into the Windows kernel to allow LoadLibrary() to write verbose information to stderr.
Open the MSYS2 Mingw64 terminal as an administrator and run
'/c/Program Files (x86)/Windows Kits/10/Debuggers/x64/gflags.exe' -i main.exe +sls
This prints the following to the terminal to confirm that the registry has been changed.
Current Registry Settings for main.exe executable are: 00000002
sls - Show Loader Snaps
Use -sls instead of +sls if you need to undo, since I believe that the change takes place for all programs called main.exe in Windows globally, not just for your file.
Then running main.exe should print debug information to stderr, but since I'm debugging an -mwindows application, it's not working for me.
But for some reason, running the binary with MSYS2's gdb allows this debug information to be printed to stderr.
Install mingw-w64-x86_64-gdb with MSYS2 and run gdb ./main.exe and type run or r.
Search for a section similar to the following.
warning: 1ec8:43a0 # 764081125 - LdrpNameToOrdinal - WARNING: Procedure "foo" could not be located in DLL at base 0x000000006FC40000.
warning: 1ec8:43a0 # 764081125 - LdrpReportError - ERROR: Locating export "foo" for DLL "C:\whatever\plugin.dll" failed with status: 0xc0000139.
warning: 1ec8:43a0 # 764081125 - LdrpGenericExceptionFilter - ERROR: Function LdrpSnapModule raised exception 0xc0000139
Exception record: .exr 00000000050BE5F0
Context record: .cxr 00000000050BE100
warning: 1ec8:43a0 # 764081125 - LdrpProcessWork - ERROR: Unable to load DLL: "C:\whatever\plugin.dll", Parent Module: "(null)", Status: 0xc0000139
warning: 1ec8:43a0 # 764081171 - LdrpLoadDllInternal - RETURN: Status: 0xc0000139
warning: 1ec8:43a0 # 764081171 - LdrLoadDll - RETURN: Status: 0xc0000139
Great! It says Procedure "foo" could not be located in DLL so we have our missing symbol, just like in POSIX/UNIX's dlopen().
While the answer from Remy Lebeau is technically correct, determining the missing symbol from GetLastError() is still possible on a Windows platform. To understand what exactly is missing, understanding the terminology is critical.
Symbol:
When a DLL is compiled, it's functions are referenced by symbols.
These symbols directly relate to the functions name (the symbols are
represented by visible and readable strings), its return type, and
it's parameters. The symbols can actually be read directly through a
text editor although difficult to find in large DLLs.DLL Symbols - C++ Forum
To have a missing symbol implies that a function within cannot be found. If this error occurs prior to using GetProcAddress(), then it's possible that any number of functions cannot be loaded due to missing prerequisites. This means it is possible that a library that you are attempting to load also requires a library that the first cannot load. These levels of dependency may go on for an unknown number of layers, but the only answer that GetLastError() can determine is that there was a missing symbol. One such method is by using Dependency Walker to determine the missing library the first library requires. Once all required libraries are available and can be found by that library (which can be its own can of worms), that library can be loaded via LoadLibrary().
Another error at startup of my basic app which only tests my DLL (the core deliverable of my project) by calling its init method. I have analysed what happens using Process Monitor, and filtered according to process name, looking only for events related with my the main executable of my basic app.
What should I be looking for there ? I guess anything that says SUCCESS in the result column does not point to the problem - but what about the rest ? It seems to be looking for my DLL all over the world although it first tried in the current directory and found it there already ...
An excerpt here, saved from ProcMon in CSV format : http://pastebin.com/YHSeQUk0
As I said it tries to open my DLL in many places (following my PATH environment variable ?). When looking for it in the correct folder (which is the same as where the exe is located and running) it goes through a series of actions which mostly end up in SUCCESS (if not otherwise specified):
IRP_MJ_CREATE
IRP_MJ_CLEANUP
IRP_MJ_CLOSE
IRP_MJ_CREATE
FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION (result is : FILE LOCKED
WITH ONLY READERS)
FASTIO_ACQUIRE_FOR_CC_FLUSH
FASTIO_ACQUIRE_FOR_SECTION_SYNCHRONIZATION (result is SUCCESS)
FASTIO_ACQUIRE_FOR_CC_FLUSH
It does all of this a second time later. To me this looks ok. No other DLL-related event show up.
There is this other event at the top :
RegOpenKey in HKLM\System\CurrentControlSet\Control\Srp\GP\DLL which first results in REPARSE and then in NAME NOT FOUND. But many things happen afterwards, this doesn't look like a blocking point - and anyhow no clue what this is about.
dumpbin /dependents ConsoleApplication1.exe yields the following :
File Type: EXECUTABLE IMAGE
Image has the following dependencies:
uss_map_interface.dll
VCRUNTIME140.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
KERNEL32.dll
The first one is my dll. If I remove it the message at startup turns into "uss_map_interface.dll is missing from your computer" - so I cannot believe this woukld be related to my DLL ... all the other I found in Windows/System32 or SysWOW64.
I am compiling both my DLL and the test basic app with MVS Express 2015 (v140) with x64 configs. The problem also appears if I change to x86 configs. The funny thing also is that if I compile my DLL with MinGW in a Qt project - it works fine. If that can provide a hint, here follow the dependencies of the Qz/MinGW-compiled version :
File Type: DLL
Image has the following dependencies:
libgcc_s_dw2-1.dll
KERNEL32.dll
msvcrt.dll
libstdc++-6.dll
Anyhow, thanks in advance !
Trying to debug a crash in one of our DLL's. It is loaded into Server Manager and crashes when trying to configure Active Directory Certificate Services (the DLL is a registered provider). I know the crash is an access violation and I have the pdb file, just don't know how to go about debugging this. I've read pages such as this and this (didn't help). I tried to glean the info using windbg (using lm to get the loaded address, which appears to be 8000000:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe" -z myKSP.dll
Then
0:000> lm
start end module name
00000001`80000000 00000001`8005e000 ...
Then, since the Event Viewer tells me:
Exception code: 0xc0000005
Fault offset: 0x000000000002a601
I tried to view that:
0:000> ln 80000000+2a601
Browse module
Set bu breakpoint
Nothing is shown.
I have VS2015, so, I tried to attach to the serververmanager.exe process. Next, I tried loading symbols via Tools->Options->Debugging->Symbols and specifying the path, but, when I set a breakpoint, I always receive "no symbols have been loaded". In the previous symbol windows, I set the cache folder, which downloaded a bunch of stuff, but that did not seem to load anything.
Clearly, I'm not using the tools correctly. How do I debug a DLL, compiled in Release mode, PDB is available, that is loaded by the ServerManager.exe or whatever sub-process it might spawn)?
Start windbg, press Ctrl-D to open your dump file, then type the following. That should give you either a significant stack after one of the kp500, or at least will tell you whether the pdb file doesn't match the binary.
.symfix
.sympath+ <FOLDER_WITH_YOUR_PDB>
.reload
!sym noisy
.reload /v /f myKSP.dll
!sym quiet
kp500
.ecxr
kp500
Trying to debug minidump in windbg/VS. Cannot get symbols for msvcr90 of version 9.0.30729.8387. Was trying to do .symfix. If I set sympath for my application symbols, I can get part of call stack which of my code, but not of msvcr. If I use .symfix, partially I can get several calls from c runtime, but, obviously, not from my code.
!sym noisy without .symfix:
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/msvcr90.dll/51EA1BBDa3000/msvcr90.dll not found
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\msvcr90.dll - file not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/msvcr90.dll/51EA1BBDa3000/msvcr90.dll not found
DBGENG: C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\msvcr90.dll - Couldn't map image from disk.
Unable to load image C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\msvcr90.dll, Win32 error 0n2
DBGENG: msvcr90.dll - Partial symbol image load missing image info
DBGHELP: Module is not fully loaded into memory.
DBGHELP: Searching for symbols using debugger-provided data.
*** WARNING: Unable to verify timestamp for msvcr90.dll
The last thing I can see with this is call from msvcr90!_freefls. In all ways I can get only part of call stack and I need full. Is there a way to obtain right symbols? Similiar problem with symbols for 9.0.30729.6871.
Client OS: Windows Server 2012 (build 9200) 64-bit, Windows 7 (build 7600) 64-bit, Windows Vista (build 6000) 64-bit, Windows Server 2008 R2 (build 7600) 64-bit.
You can have multiple symbol paths so the debugger can get symbols from multiple places.
.Symfix
Is a good first command to run.
Then use
.sympathy+ c:\<My_symbols>
The other problem above is
Couldn't map image from disk.
that comes from
DBGENG: C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\msvcr90.dll - Couldn't map image from disk.
Unable to load image C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\msvcr90.dll, Win32 error 0n2
DBGENG: msvcr90.dll - Partial symbol image load missing image info
DBGHELP: Module is not fully loaded into memory.
I would make another copy of that file and point your symbol path to that to see if the debugger can load it from there.
Solved with reproducing problem on client OS with attached debugger.
After the program I am debugging crashes, I am left with heap dump *.mdmp file & appcompat.txt in my Temp directory. I understand that appcompat.txt is an error report. Is there a description of its format?
My appcompat.txt lists a number of DLLs. Am I correct assuming that the reason for a crash could have only come from one of the listed DLLs? Can I limit my debugging effort to the DLLs listed in appcompat.txt?
Thanks in advance!
The minidump file is far more informative for diagnosing crashes:
Install Debugging Tools for Windows, if you don't already have it.
Set up the symbol path variable _NT_SYMBOL_PATH to point to the Microsoft symbol server
Run Windbg and do File -> Open Crash Dump and locate your .dmp or .mdmp file
Type !analyze -v.
This will try to isolate the location of the crash. Note that just because a crash occurs in a particular dll it doesn't mean that is where the bug resides - it could be because an invalid parameter has been passed in from your application code. The analysis should hopefully show you a meaningful stack and an error code which should help in working out the actual cause of the crash.