Trying to debug minidump in windbg/VS. Cannot get symbols for msvcr90 of version 9.0.30729.8387. Was trying to do .symfix. If I set sympath for my application symbols, I can get part of call stack which of my code, but not of msvcr. If I use .symfix, partially I can get several calls from c runtime, but, obviously, not from my code.
!sym noisy without .symfix:
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/msvcr90.dll/51EA1BBDa3000/msvcr90.dll not found
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\msvcr90.dll - file not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: \\eserver\symstore\ms\msvcr90.dll\51EA1BBDa3000\msvcr90.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/msvcr90.dll/51EA1BBDa3000/msvcr90.dll not found
DBGENG: C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\msvcr90.dll - Couldn't map image from disk.
Unable to load image C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\msvcr90.dll, Win32 error 0n2
DBGENG: msvcr90.dll - Partial symbol image load missing image info
DBGHELP: Module is not fully loaded into memory.
DBGHELP: Searching for symbols using debugger-provided data.
*** WARNING: Unable to verify timestamp for msvcr90.dll
The last thing I can see with this is call from msvcr90!_freefls. In all ways I can get only part of call stack and I need full. Is there a way to obtain right symbols? Similiar problem with symbols for 9.0.30729.6871.
Client OS: Windows Server 2012 (build 9200) 64-bit, Windows 7 (build 7600) 64-bit, Windows Vista (build 6000) 64-bit, Windows Server 2008 R2 (build 7600) 64-bit.
You can have multiple symbol paths so the debugger can get symbols from multiple places.
.Symfix
Is a good first command to run.
Then use
.sympathy+ c:\<My_symbols>
The other problem above is
Couldn't map image from disk.
that comes from
DBGENG: C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\msvcr90.dll - Couldn't map image from disk.
Unable to load image C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\msvcr90.dll, Win32 error 0n2
DBGENG: msvcr90.dll - Partial symbol image load missing image info
DBGHELP: Module is not fully loaded into memory.
I would make another copy of that file and point your symbol path to that to see if the debugger can load it from there.
Solved with reproducing problem on client OS with attached debugger.
Related
On Linux and Mac, when using dlopen() to load a shared library that links to another library, if linking fails because of a missing symbol, you can get the name of the missing symbol with dlerror(). It says something like
dlopen failed: cannot locate symbol "foo"
On Windows, when using LoadLibrary() to load a DLL with a missing symbol, you can only get an error code from GetLastError() which for this type of issue will always be 127. How can I figure out which symbol is missing, or a more verbose error message from LoadLibrary() that explains why the function failed?
I figured out a way using the MSYS2 terminal. Other methods might work with GUI software.
A major caveat is that this can't be done in pure C/C++ and released for end users. It's for developers only, but it's better than nothing.
Install Debugging Tools for Windows by downloading the Windows SDK and unchecking everything except Debugging Tools.
I could be wrong, but it seems that installing this software installs a hook into the Windows kernel to allow LoadLibrary() to write verbose information to stderr.
Open the MSYS2 Mingw64 terminal as an administrator and run
'/c/Program Files (x86)/Windows Kits/10/Debuggers/x64/gflags.exe' -i main.exe +sls
This prints the following to the terminal to confirm that the registry has been changed.
Current Registry Settings for main.exe executable are: 00000002
sls - Show Loader Snaps
Use -sls instead of +sls if you need to undo, since I believe that the change takes place for all programs called main.exe in Windows globally, not just for your file.
Then running main.exe should print debug information to stderr, but since I'm debugging an -mwindows application, it's not working for me.
But for some reason, running the binary with MSYS2's gdb allows this debug information to be printed to stderr.
Install mingw-w64-x86_64-gdb with MSYS2 and run gdb ./main.exe and type run or r.
Search for a section similar to the following.
warning: 1ec8:43a0 # 764081125 - LdrpNameToOrdinal - WARNING: Procedure "foo" could not be located in DLL at base 0x000000006FC40000.
warning: 1ec8:43a0 # 764081125 - LdrpReportError - ERROR: Locating export "foo" for DLL "C:\whatever\plugin.dll" failed with status: 0xc0000139.
warning: 1ec8:43a0 # 764081125 - LdrpGenericExceptionFilter - ERROR: Function LdrpSnapModule raised exception 0xc0000139
Exception record: .exr 00000000050BE5F0
Context record: .cxr 00000000050BE100
warning: 1ec8:43a0 # 764081125 - LdrpProcessWork - ERROR: Unable to load DLL: "C:\whatever\plugin.dll", Parent Module: "(null)", Status: 0xc0000139
warning: 1ec8:43a0 # 764081171 - LdrpLoadDllInternal - RETURN: Status: 0xc0000139
warning: 1ec8:43a0 # 764081171 - LdrLoadDll - RETURN: Status: 0xc0000139
Great! It says Procedure "foo" could not be located in DLL so we have our missing symbol, just like in POSIX/UNIX's dlopen().
While the answer from Remy Lebeau is technically correct, determining the missing symbol from GetLastError() is still possible on a Windows platform. To understand what exactly is missing, understanding the terminology is critical.
Symbol:
When a DLL is compiled, it's functions are referenced by symbols.
These symbols directly relate to the functions name (the symbols are
represented by visible and readable strings), its return type, and
it's parameters. The symbols can actually be read directly through a
text editor although difficult to find in large DLLs.DLL Symbols - C++ Forum
To have a missing symbol implies that a function within cannot be found. If this error occurs prior to using GetProcAddress(), then it's possible that any number of functions cannot be loaded due to missing prerequisites. This means it is possible that a library that you are attempting to load also requires a library that the first cannot load. These levels of dependency may go on for an unknown number of layers, but the only answer that GetLastError() can determine is that there was a missing symbol. One such method is by using Dependency Walker to determine the missing library the first library requires. Once all required libraries are available and can be found by that library (which can be its own can of worms), that library can be loaded via LoadLibrary().
I'm studying kernel mode driver following to this Youtube video and preparing for debugging a driver in a VirtualBox VM, with WinDbg and Virtual KD.
I set up the symbol file by clicking
File / Symbol file path
add symbol path
SRV*c:\symbols* http://msdl.microsoft.com/download/symbols
put a check mark to "reload" item
click ok
After that, Windbg's screen is as follows:
************* Path validation summary **************
Response Time (ms) Location
Deferred SRV*c:\symbols* http://msdl.microsoft.com/download/symbols
kd> .reload
Connected to Windows 10 17134 x64 target at (Sun Oct 7 13:16:30.147 2018 (UTC + 9:00)), ptr64 TRUE
Loading Kernel Symbols
...............................................................
................................................................
..........................
Loading User Symbols
Loading unloaded module list
......Unable to enumerate user-mode unloaded modules, Win32 error 0n30
I can not understand symbol files functioning properly.
Are Symbol files currently not available?
I use lml command in such case.
If symbol files are loaded, you can find module name like this.
2: kd> lml
start end module name
ffff9e54`ba960000 ffff9e54`ba9d7000 win32k (pdb symbols) c:\symbols\win32k.pdb\901A464ABCFD2696F50FFB02C607B4661\win32k.pdb
fffff803`6921a000 fffff803`69aef000 nt (pdb symbols) c:\symbols\ntkrnlmp.pdb\9378084E8DBD4AB1A155099BCE693E341\ntkrnlmp.pdb
Trying to debug a crash in one of our DLL's. It is loaded into Server Manager and crashes when trying to configure Active Directory Certificate Services (the DLL is a registered provider). I know the crash is an access violation and I have the pdb file, just don't know how to go about debugging this. I've read pages such as this and this (didn't help). I tried to glean the info using windbg (using lm to get the loaded address, which appears to be 8000000:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\windbg.exe" -z myKSP.dll
Then
0:000> lm
start end module name
00000001`80000000 00000001`8005e000 ...
Then, since the Event Viewer tells me:
Exception code: 0xc0000005
Fault offset: 0x000000000002a601
I tried to view that:
0:000> ln 80000000+2a601
Browse module
Set bu breakpoint
Nothing is shown.
I have VS2015, so, I tried to attach to the serververmanager.exe process. Next, I tried loading symbols via Tools->Options->Debugging->Symbols and specifying the path, but, when I set a breakpoint, I always receive "no symbols have been loaded". In the previous symbol windows, I set the cache folder, which downloaded a bunch of stuff, but that did not seem to load anything.
Clearly, I'm not using the tools correctly. How do I debug a DLL, compiled in Release mode, PDB is available, that is loaded by the ServerManager.exe or whatever sub-process it might spawn)?
Start windbg, press Ctrl-D to open your dump file, then type the following. That should give you either a significant stack after one of the kp500, or at least will tell you whether the pdb file doesn't match the binary.
.symfix
.sympath+ <FOLDER_WITH_YOUR_PDB>
.reload
!sym noisy
.reload /v /f myKSP.dll
!sym quiet
kp500
.ecxr
kp500
My app is crash. So I am using windbg to check the trace log. Here is my trace log in windbg:
FAILED_INSTRUCTION_ADDRESS:
cbmwk5!unloaded+161c0
727261c0 ?? ???
ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) amd64fre
LAST_CONTROL_TRANSFER: from 423c1d76 to 72725817
IP_MODULE_UNLOADED:
cbmwk5!unloaded+15817
72725817 ?? ???
PRIMARY_PROBLEM_CLASS: BAD_INSTRUCTION_PTR_NULL
DEFAULT_BUCKET_ID: BAD_INSTRUCTION_PTR_NULL
STACK_TEXT:
023eebfc 72725817 cbmwk5!unloaded+0x15817
023eec00 423c1d76 unknown!unknown+0x0
023eec30 7272590a cbmwk5!unloaded+0x1590a
023eec58 72723a39 cbmwk5!unloaded+0x13a39
023ef120 76b65762 shell32!SHGetFolderPathW+0x180
023ef128 72720813 cbmwk5!unloaded+0x10813
023ef144 7271110e cbmwk5!unloaded+0x110e
023ef164 72715916 cbmwk5!unloaded+0x5916
023ef59c 7271636b cbmwk5!unloaded+0x636b
Could you please help me how to check the size of cbmwk5.dll according to STACK_TEXT?
What is the meaning of "+0x15817" int the statement:
023eebfc 72725817 cbmwk5!unloaded+0x15817
I tried to reload by using command:
.reload /unl cbmwk5.dll
and then type: !analyze -v
but the error missing cbmwk5.dll occurs:
SYMSRV: c:\localsymbols\cbmwk5.dll\506DCE083b000\cbmwk5.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/cbmwk5.dll/506DCE083b000/cbmwk5.dll not found
DBGHELP: C:\Program Files (x86)\Windows Kits\8.1\Debuggers\cbmwk5.dll - file not found
DBGENG: cbmwk5.dll - Image mapping disallowed by non-local path.
DBGHELP: No header for cbmwk5.dll. Searching for dbg file
DBGHELP: .\cbmwk5.dbg - file not found
DBGHELP: .\dll\cbmwk5.dbg - path not found
DBGHELP: .\symbols\dll\cbmwk5.dbg - path not found
DBGHELP: cbmwk5.pdb - file not found
*** WARNING: Unable to verify timestamp for cbmwk5.dll
*** ERROR: Module load completed but symbols could not be loaded for cbmwk5.dll
DBGHELP: cbmwk5 - no symbols loaded
DBGHELP: C:\Program Files (x86)\Windows Kits\8.1\Debuggers\cbmwk5.dll - file not found
SYMSRV: C:\Program Files (x86)\Windows Kits\8.1\Debuggers\x64\sym\cbmwk5.dll\506DCE083b000\cbmwk5.dll not found
Thanks a lot.
What is the meaning of "+0x15817" int the statement
+0x15817 means that the debugger has no clue whatsoever which function was called. It just doesn't know anything about the DLL, only where it was once loaded. So it can only annotate the address with the DLL name and a very large offset. Otherwise visible from the SYMSRV trace messages, the debugger made an attempt to download the PDB file for the DLL but the symbol server doesn't know anything about the DLL. Which is certainly not unusual, it is a 3rd party DLL, not Microsoft's. Even Google has never heard of it.
FAILED_INSTRUCTION_ADDRESS:
cbmwk5!unloaded+161c0
The unloaded annotation is your strongest clue to the problem. The code crashed because the DLL was unloaded from memory. Yet the program tried to call it anyway. With no code left to execute (note the ??), the processor is forced to give up and generates an access violation. That was the end of the program, it cannot continue operating.
023ef120 76b65762 shell32!SHGetFolderPathW+0x180
The stack trace gives a (weak) clue to the underlying problem. Beware that this is speculation. But the presence of shell function like SHGetFolderPathW() is a strong hint that this is a misbehaving shell extension. They can do a lot of damage since they tend to be injected into your program when you use one of the common shell dialogs, like OpenFileDialog. In other words, it doesn't have anything to do with your program, it is somebody else's crappy code that made the program bomb.
You fix this kind of problem by disabling shell extensions one by one until the problem disappears. SysInternals' AutoRuns utility is the weapon of choice. It has to be done by the machine owner, there's little that you can do but give advice.
I wrote a C++ project called 'Foo' using Microsoft Visual Studio 2005 Verison 8.0.50727.762 (SP.050727-7600) on Windows XP Professional Version 2002 Service Pack 3. I built the project into Foo.exe. Then, I copied the file Foo.exe to a Windows Server 2003 Enterprise Edition Service Pack 2. It runs fine.
Now, I try to debug it using WinDbg:6.12.0002.663 X86 Version 5.2.
No symbol path, source path or image path is set
When I open C:\foo\Foo.exe from the 'File > Open Executable' menu, I see this error:
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -
Foo.pdb is not present in the same folder where Foo.exe is present. Why do I not see an error for Foo.exe?
On running the command ld * in WinDbg, I see these errors:
0:000> ld *
*** ERROR: Module load completed but symbols could not be loaded for Foo.exe
Symbols loaded for Foo
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\msvcrt.dll -
Symbols loaded for msvcrt
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\kernel32.dll -
Symbols loaded for kernel32
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.6195_x-ww_44262B86\MSVCR80.dll -
Symbols loaded for MSVCR80
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.6195_x-ww_44262B86\MSVCP80.dll -
Symbols loaded for MSVCP80
Symbols already loaded for ntdll
Now, let me show two attempts to fix it. One worked, the other didn't.
Downloading Windows Server 2003 with Service Pack 2 x86 retail symbols didn't work
I went to http://msdn.microsoft.com/en-us/windows/hardware/gg463028 and downloaded:
Windows Server 2003 with Service Pack 2 x86 retail symbols, all languages (File size: 154 MB - Most customers want this package.)
I ran it and installed the symbols in C:\Symbols. In WinDbg, I set the symbol path as:
C:\Symbols;C:\MySymbols
C:\MySymbols contained the symbol for Foo.exe which had a file-name: Foo.pdb.
On opening C:\foo\Foo.exe from the 'File > Open Executable' menu, I got the same error mentioned in the previous section.
However, the ld * command showed fewer errors this time.
0:000> ld *
Symbols loaded for Foo
Symbols loaded for msvcrt
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\kernel32.dll -
Symbols loaded for kernel32
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.6195_x-ww_44262B86\MSVCR80.dll -
Symbols loaded for MSVCR80
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.6195_x-ww_44262B86\MSVCP80.dll -
Symbols loaded for MSVCP80
Symbols already loaded for ntdll
Using Microsoft Symbol Server fixed it
This time I set the symbol path as:
SRV*C:\MicrosoftSymbols*http://msdl.microsoft.com/download/symbols;C:\MySymbols
Now, on opening C:\foo\Foo.exe from the 'File > Open Executable' menu, I got no errors. I found that C:\microsoftsymbols\ntdll.pdb\F7024C7F15FE4BEA992FF38BE58AC11C2\ntdll.pdb was downloaded automatically.
This time ld * command also ran fine without any errors.
0:000> ld *
Symbols loaded for Foo
Symbols loaded for msvcrt
Symbols loaded for kernel32
Symbols loaded for MSVCR80
Symbols loaded for MSVCP80
Symbols already loaded for ntdll
The following additional symbols were downloaded automatically while the command was running.
C:\microsoftsymbols\kernel32.pdb\BE496DC9472F4438B080C70594D8F9CC2\kernel32.pdb
C:\microsoftsymbols\msvcp80.i386.pdb\E8A9423E890A4ADDB38F8F756268437C1\msvcp80.i386.pdb
C:\microsoftsymbols\msvcr80.i386.pdb\54C9E2F351544D1CB39517DC4B299EA81\msvcr80.i386.pdb
C:\microsoftsymbols\msvcrt.pdb\A7F38CEE7E684B94B7AA9FFFCAB446851\msvcrt.pdb
Questions
Why did I not see an error regarding symbol file for Foo.exe when no symbol path was specified?
Why downloading Windows Server 2003 with Service Pack 2 x86 retail symbols and specifying the path to it as the symbol path didn't work?
More information
In case it helps you to answer my question, here is my system details as obtained from winmsd.exe > System Summary:
OS Name: Microsoft(R) Windows(R) Server 2003, Enterprise Edition
Version: 5.2.3790 Service Pack 2 Build 3790
Other OS Description : Not Available
OS Manufacturer: Microsoft Corporation
System Name: EULER
System Manufacturer: VMware, Inc.
System Model: VMware Virtual Platform
System Type: X86-based PC
Processor: x86 Family 16 Model 4 Stepping 2 AuthenticAMD ~2417 Mhz
Processor: x86 Family 16 Model 4 Stepping 2 AuthenticAMD ~2416 Mhz
BIOS Version/Date: Phoenix Technologies LTD 6.00, 7/22/2008
SMBIOS Version: 2.4
Windows Directory: C:\WINDOWS
System Directory: C:\WINDOWS\system32
Boot Device: \Device\HarddiskVolume1
Locale: United States
Hardware Abstraction Layer: Version = "5.2.3790.3959 (srv03_sp2_rtm.070216-1710)"
User Name: Not Available
Time Zone: India Standard Time
Total Physical Memory: 3,839.45 MB
Available Physical Memory: 1.56 GB
Total Virtual Memory: 5.60 GB
Available Virtual Memory: 3.33 GB
Page File Space: 2.00 GB
Page File: C:\pagefile.sys
Why did I not see an error regarding symbol file for Foo.exe when no symbol path was specified?
Answer to 1:
You did when windbg tried to load the symbols for Foo
0:000> ld *
*** ERROR: Module load completed but symbols could not be loaded for Foo.exe
Symbols loaded for Foo
The reason that it didn't show up when you first loaded was that windbg didn't try to load the symbols at first for foo. Most likely because Foo had yet to be called. The program was just starting up and your program logic hadn't actually been called yet.
Why downloading Windows Server 2003 with Service Pack 2 x86 retail symbols and specifying the path to it as the symbol path didn't work?
Answer to 2:
I can't say with 100% certainty, but most likely security patches had been applied so that the pdbs that were included with the retail symbols didn't include the changes. When you went to MS site you can get all the symbols for all released versions of the DLLs including patched versions.
Why did I not see an error regarding symbol file for Foo.exe when no
symbol path was specified?
While the previous answer is correct, note that image files contain a reference the build location of their PDBs by default. Thus, it's possible for WinDBG to magically find the PDB for an image that you built on your local machine. This is explained in more detail here:
http://analyze-v.com/?p=245