WinDbg does not show full stack trace for some minidump-files - c++

Question

I'm trying to set up a crashdump system to better be able to debug errors that I can't replicate easily on my own system
Here's my test program (Compiled with Release x64 Configuration with Visual Studio 2017):
#include "pch.h"
#include <iostream>
#include <Windows.h>
#include <DbgHelp.h>
#pragma comment(lib,"Dbghelp.lib")
BOOL MiniDumpWriteDump(
HANDLE hProcess,
DWORD ProcessId,
HANDLE hFile,
MINIDUMP_TYPE DumpType,
PMINIDUMP_EXCEPTION_INFORMATION ExceptionParam,
PMINIDUMP_USER_STREAM_INFORMATION UserStreamParam,
PMINIDUMP_CALLBACK_INFORMATION CallbackParam
);
LONG CreateMiniDump( struct _EXCEPTION_POINTERS *pExceptionInfo )//EXCEPTION_POINTERS* pep )
{
// Open the file
HANDLE hFile = CreateFile( "MiniDump.dmp", GENERIC_READ | GENERIC_WRITE,
0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );
if( ( hFile != NULL ) && ( hFile != INVALID_HANDLE_VALUE ) )
{
// Create the minidump
MINIDUMP_EXCEPTION_INFORMATION mdei;
mdei.ThreadId = GetCurrentThreadId();
mdei.ExceptionPointers = pExceptionInfo;
mdei.ClientPointers = FALSE;
MINIDUMP_TYPE mdt = MiniDumpNormal;
BOOL rv = MiniDumpWriteDump( GetCurrentProcess(), GetCurrentProcessId(),
hFile, mdt, (pExceptionInfo != 0) ? &mdei : 0, 0, 0 );
if( !rv )
std::cout<<"MiniDumpWriteDump failed. Error: "<<GetLastError()<<std::endl;
else
std::cout<<"Minidump created."<<std::endl;
// Close the file
CloseHandle( hFile );
}
else
{
std::cout<<"CreateFile failed. Error: "<<GetLastError()<<std::endl;
}
return EXCEPTION_EXECUTE_HANDLER;
}
static void ftest3()
{
throw std::runtime_error("Error");
}
static void ftest2()
{
ftest3();
}
static void ftest1()
{
ftest2();
}
int main()
{
SetUnhandledExceptionFilter(&CreateMiniDump);
ftest1();
}
(Source: http://www.debuginfo.com/articles/effminidumps2.html)
If I run it on my primary system (on which I compiled the program), and then load the .dmp-file in Visual Studio or WinDbg, the full stack trace is displayed correctly and I can see where the error has occurred:
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 000000fd`3bbbf6f0 00007ff8`a72142cd KERNELBASE!RaiseException+0x68
01 000000fd`3bbbf7d0 00007ff7`7b4f130f VCRUNTIME140!_CxxThrowException+0xad [f:\dd\vctools\crt\vcruntime\src\eh\throw.cpp # 133]
02 000000fd`3bbbf840 00007ff7`7b4f1319 test_minidump!ftest3+0x1f [c:\users\florian\documents\projects\test_minidump\test_minidump\test_minidump.cpp # 63]
03 000000fd`3bbbf890 00007ff7`7b4f1329 test_minidump!ftest2+0x9 [c:\users\florian\documents\projects\test_minidump\test_minidump\test_minidump.cpp # 68]
04 000000fd`3bbbf8c0 00007ff7`7b4f1346 test_minidump!ftest1+0x9 [c:\users\florian\documents\projects\test_minidump\test_minidump\test_minidump.cpp # 73]
05 000000fd`3bbbf8f0 00007ff7`7b4f1838 test_minidump!main+0x16 [c:\users\florian\documents\projects\test_minidump\test_minidump\test_minidump.cpp # 79]
06 (Inline Function) --------`-------- test_minidump!invoke_main+0x22 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl # 78]
07 000000fd`3bbbf920 00007ff8`b45d3034 test_minidump!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl # 288]
08 000000fd`3bbbf960 00007ff8`b71a1461 kernel32!BaseThreadInitThunk+0x14
09 000000fd`3bbbf990 00000000`00000000 ntdll!RtlUserThreadStart+0x21
However, if I run the program on a different system, and then try to analyze the dump on my own system, all I get is this:
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 00000021`ccbef870 00007ffe`8cfd4722 KERNELBASE!RaiseException+0x68
01 00000021`ccbef950 00000021`ccbef9f0 VCRUNTIME140!_CxxThrowException+0xc2 [f:\dd\vctools\crt\vcruntime\src\eh\throw.cpp # 136]
02 00000021`ccbef958 00007ffe`00000000 0x00000021`ccbef9f0
03 00000021`ccbef960 000001ee`2e25ff30 0x00007ffe`00000000
04 00000021`ccbef968 00000000`00000000 0x000001ee`2e25ff30
Which isn't exactly helpful. I've loaded the dump-file in WinDbg and then used the following commands:
.symfix
.sympath+ C:\Users\Florian\Documents\Projects\test_minidump\x64\Release
.srcpath C:\Users\Florian\Documents\Projects\test_minidump\test_minidump
!analyze -v
.ecxr
k
Here's the full output from WinDbg:
Microsoft (R) Windows Debugger Version 10.0.17134.12 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\Florian\Desktop\tmp\MiniDump.dmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: srv*
Executable search path is:
Windows 10 Version 17134 MP (4 procs) Free x64
Product: WinNt, suite: SingleUserTS
17134.1.amd64fre.rs4_release.180410-1804
Machine Name:
Debug session time: Sun Sep 23 11:56:45.000 2018 (UTC + 2:00)
System Uptime: not available
Process Uptime: not available
..............
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(169c.1b78): C++ EH exception - code e06d7363 (first/second chance not available)
ntdll!NtGetContextThread+0x14:
00007ffe`9abfbc44 c3 ret
0:000> .symfix
0:000> .sympath+ C:\Users\Florian\Documents\Projects\test_minidump\x64\Release
Symbol search path is: srv*;C:\Users\Florian\Documents\Projects\test_minidump\x64\Release
Expanded Symbol search path is: cache*;SRV*https://msdl.microsoft.com/download/symbols;c:\users\florian\documents\projects\test_minidump\x64\release
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
OK C:\Users\Florian\Documents\Projects\test_minidump\x64\Release
0:000> .srcpath C:\Users\Florian\Documents\Projects\test_minidump\test_minidump
Source search path is: C:\Users\Florian\Documents\Projects\test_minidump\test_minidump
************* Path validation summary **************
Response Time (ms) Location
OK C:\Users\Florian\Documents\Projects\test_minidump\test_minidump
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
TIMELINE_ANALYSIS: 1
Timeline: !analyze.Start
Name: <blank>
Time: 2018-09-24T10:08:32.944Z
Diff: 87107944 mSec
Timeline: Dump.Current
Name: <blank>
Time: 2018-09-23T09:56:45.0Z
Diff: 0 mSec
DUMP_CLASS: 2
DUMP_QUALIFIER: 400
CONTEXT: (.ecxr)
rax=0000000000000000 rbx=00007ffe973da570 rcx=0000000000000000
rdx=0000000000000030 rsi=00007ff798203cd0 rdi=000001ee2e25c750
rip=00007ffe97bba388 rsp=00000021ccbef870 rbp=00000021ccbef9b0
r8=0000000000000000 r9=0000000000000000 r10=000001ee2e0f0000
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=00000021ccbef9f0 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
KERNELBASE!RaiseException+0x68:
00007ffe`97bba388 488b8c24c0000000 mov rcx,qword ptr [rsp+0C0h] ss:00000021`ccbef930=000052660787015b
Resetting default scope
FAULTING_IP:
KERNELBASE!RaiseException+68
00007ffe`97bba388 488b8c24c0000000 mov rcx,qword ptr [rsp+0C0h]
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffe97bba388 (KERNELBASE!RaiseException+0x0000000000000068)
ExceptionCode: e06d7363 (C++ EH exception)
ExceptionFlags: 00000001
NumberParameters: 4
Parameter[0]: 0000000019930520
Parameter[1]: 00000021ccbef9f0
Parameter[2]: 00007ff798203cd0
Parameter[3]: 00007ff798200000
BUGCHECK_STR: CPP_EXCEPTION
DEFAULT_BUCKET_ID: CPP_EXCEPTION
PROCESS_NAME: test_minidump.exe
ERROR_CODE: (NTSTATUS) 0xe06d7363 - <Unable to get error code text>
EXCEPTION_CODE: (NTSTATUS) 0xe06d7363 - <Unable to get error code text>
EXCEPTION_CODE_STR: e06d7363
EXCEPTION_PARAMETER1: 0000000019930520
EXCEPTION_PARAMETER2: 00000021ccbef9f0
EXCEPTION_PARAMETER3: 00007ff798203cd0
EXCEPTION_PARAMETER4: 7ff798200000
WATSON_BKT_PROCSTAMP: 5ba8b478
WATSON_BKT_MODULE: KERNELBASE.dll
WATSON_BKT_MODSTAMP: b0bb231d
WATSON_BKT_MODOFFSET: 3a388
WATSON_BKT_MODVER: 6.2.17134.165
MODULE_VER_PRODUCT: Microsoft® Windows® Operating System
BUILD_VERSION_STRING: 17134.1.amd64fre.rs4_release.180410-1804
MODLIST_WITH_TSCHKSUM_HASH: 636d437315fdc5447e163e64094b1f6a26123a83
MODLIST_SHA1_HASH: 74dec0c40d229cd1e109927f1f5e794fa53165ac
DUMP_FLAGS: 0
DUMP_TYPE: 2
ANALYSIS_SESSION_HOST: DESKTOP-VFOALG8
ANALYSIS_SESSION_TIME: 09-24-2018 12:08:32.0944
ANALYSIS_VERSION: 10.0.17134.12 amd64fre
THREAD_ATTRIBUTES:
PROBLEM_CLASSES:
ID: [0n317]
Type: [#APPLICATION_FAULT_STRING]
Class: Primary
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
Name: Omit
Data: Add
String: [CPP_EXCEPTION]
PID: [Unspecified]
TID: [Unspecified]
Frame: [0]
PRIMARY_PROBLEM_CLASS: CPP_EXCEPTION
LAST_CONTROL_TRANSFER: from 00007ffe8cfd4722 to 00007ffe97bba388
STACK_TEXT:
00000021`ccbef870 00007ffe`8cfd4722 : 00000021`ccbef9f0 00007ffe`00000000 000001ee`2e25ff30 00000000`00000000 : KERNELBASE!RaiseException+0x68
00000021`ccbef950 00000021`ccbef9f0 : 00007ffe`00000000 000001ee`2e25ff30 00000000`00000000 00000001`e06d7363 : VCRUNTIME140!_CxxThrowException+0xc2
00000021`ccbef958 00007ffe`00000000 : 000001ee`2e25ff30 00000000`00000000 00000001`e06d7363 00000000`00000000 : 0x00000021`ccbef9f0
00000021`ccbef960 000001ee`2e25ff30 : 00000000`00000000 00000001`e06d7363 00000000`00000000 00000000`00000000 : 0x00007ffe`00000000
00000021`ccbef968 00000000`00000000 : 00000001`e06d7363 00000000`00000000 00000000`00000000 00000000`00000004 : 0x000001ee`2e25ff30
THREAD_SHA1_HASH_MOD_FUNC: 74c55fd19222dcb5ad415427110811b213f4c078
THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 7592380e0f749a34b2d654a7c3b829a033ee79fa
THREAD_SHA1_HASH_MOD: ad69e1f2190edbb9a7df71b4b6403437528645f2
FOLLOWUP_IP:
KERNELBASE!RaiseException+68
00007ffe`97bba388 488b8c24c0000000 mov rcx,qword ptr [rsp+0C0h]
FAULT_INSTR_CODE: 248c8b48
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: KERNELBASE!RaiseException+68
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: KERNELBASE
IMAGE_NAME: KERNELBASE.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 0
STACK_COMMAND: ~0s ; .ecxr ; kb
BUCKET_ID: CPP_EXCEPTION_KERNELBASE!RaiseException+68
FAILURE_EXCEPTION_CODE: e06d7363
FAILURE_IMAGE_NAME: KERNELBASE.dll
BUCKET_ID_IMAGE_STR: KERNELBASE.dll
FAILURE_MODULE_NAME: KERNELBASE
BUCKET_ID_MODULE_STR: KERNELBASE
FAILURE_FUNCTION_NAME: RaiseException
BUCKET_ID_FUNCTION_STR: RaiseException
BUCKET_ID_OFFSET: 68
BUCKET_ID_MODTIMEDATESTAMP: 0
BUCKET_ID_MODCHECKSUM: 27e598
BUCKET_ID_MODVER_STR: 6.2.17134.165
BUCKET_ID_PREFIX_STR: CPP_EXCEPTION_
FAILURE_PROBLEM_CLASS: CPP_EXCEPTION
FAILURE_SYMBOL_NAME: KERNELBASE.dll!RaiseException
FAILURE_BUCKET_ID: CPP_EXCEPTION_e06d7363_KERNELBASE.dll!RaiseException
TARGET_TIME: 2018-09-23T09:56:45.000Z
OSBUILD: 17134
OSSERVICEPACK: 1
SERVICEPACK_NUMBER: 0
OS_REVISION: 0
SUITE_MASK: 256
PRODUCT_TYPE: 1
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
OSEDITION: Windows 10 WinNt SingleUserTS
OS_LOCALE:
USER_LCID: 0
OSBUILD_TIMESTAMP: 2020-08-28 06:38:41
BUILDDATESTAMP_STR: 180410-1804
BUILDLAB_STR: rs4_release
BUILDOSVER_STR: 10.0.17134.1.amd64fre.rs4_release.180410-1804
ANALYSIS_SESSION_ELAPSED_TIME: a63
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:cpp_exception_e06d7363_kernelbase.dll!raiseexception
FAILURE_ID_HASH: {1253aecc-520d-655b-58e3-5eb61e209188}
Followup: MachineOwner
---------
0:000> .ecxr
rax=0000000000000000 rbx=00007ffe973da570 rcx=0000000000000000
rdx=0000000000000030 rsi=00007ff798203cd0 rdi=000001ee2e25c750
rip=00007ffe97bba388 rsp=00000021ccbef870 rbp=00000021ccbef9b0
r8=0000000000000000 r9=0000000000000000 r10=000001ee2e0f0000
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=00000021ccbef9f0 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
KERNELBASE!RaiseException+0x68:
00007ffe`97bba388 488b8c24c0000000 mov rcx,qword ptr [rsp+0C0h] ss:00000021`ccbef930=000052660787015b
0:000> k
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 00000021`ccbef870 00007ffe`8cfd4722 KERNELBASE!RaiseException+0x68
01 00000021`ccbef950 00000021`ccbef9f0 VCRUNTIME140!_CxxThrowException+0xc2 [f:\dd\vctools\crt\vcruntime\src\eh\throw.cpp # 136]
02 00000021`ccbef958 00007ffe`00000000 0x00000021`ccbef9f0
03 00000021`ccbef960 000001ee`2e25ff30 0x00007ffe`00000000
04 00000021`ccbef968 00000000`00000000 0x000001ee`2e25ff30
There are no warnings or errors from what I can tell, so why am I not getting the full stack trace information?

Answer 1

VCRUNTIME140!_CxxThrowException+0xad [f:\dd\vctools\crt\vcruntime\src\eh\throw.cpp # 133]
versus
VCRUNTIME140!_CxxThrowException+0xc2 [f:\dd\vctools\crt\vcruntime\src\eh\throw.cpp # 136]
is a bit worrying. It suggests that the minor version of the MSVC runtime doesn't match between the two machines (because otherwise, why would they invoke RaiseException from different source lines). If your WinDbg session doesn't have symbols for the remote machine's VCRUNTIME140 -- or, worse yet, if it's using the wrong PDB for it -- backtracing is going to become problematic at that point. WinDbg should automatically download and load symbols for whichever version is referenced from the minidump, but I suspect this isn't happening properly.
Use lm v to show the module listing for the minidump, and pay close attention to the checksum information. I suspect you'll see a different checksum for vcruntime140.dll between the two machines. If you do, you'll need to figure out why you're not getting the right symbols for the dll.

Answer 2

Thanks to the tips by users Sneftel and Sean Cline, I found the solution. Running the command !itoldyouso vcruntime140 gives me the following output:
sig MISMATCH: vcruntime140.amd64.pdb and VCRUNTIME140.dll
The problem was that my system which I use for debugging runs on an AMD CPU, while the system that generated the crashdump runs on an intel CPU. For some reason the vcruntime140.dll is system-specific and differs between the two. (Also, on AMD systems the pdb is called "vcruntime140.amd64.pdb", on intel it's "vcruntime140.i386.pdb".)
I've copied the vcruntime140.dll, vcruntime140.i386.pdb and msvcp140.dll from my intel system and added them to the symbol path for WinDbg. Then I ran the same commands as before, and now it outputs the full stacktrace.
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP RetAddr Call Site
00 00000021`ccbef870 00007ffe`8cfd4722 KERNELBASE!RaiseException+0x68
01 00000021`ccbef950 00007ff7`9820130f VCRUNTIME140!_CxxThrowException+0xc2 [f:\dd\vctools\crt\vcruntime\src\eh\throw.cpp # 136]
02 00000021`ccbef9d0 00007ff7`98201319 test_minidump!ftest3+0x1f [c:\users\florian\documents\projects\test_minidump\test_minidump\test_minidump.cpp # 63]
03 00000021`ccbefa20 00007ff7`98201329 test_minidump!ftest2+0x9 [c:\users\florian\documents\projects\test_minidump\test_minidump\test_minidump.cpp # 68]
04 00000021`ccbefa50 00007ff7`98201346 test_minidump!ftest1+0x9 [c:\users\florian\documents\projects\test_minidump\test_minidump\test_minidump.cpp # 73]
05 00000021`ccbefa80 00007ff7`98201838 test_minidump!main+0x16 [c:\users\florian\documents\projects\test_minidump\test_minidump\test_minidump.cpp # 79]
06 (Inline Function) --------`-------- test_minidump!invoke_main+0x22 [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl # 78]
07 00000021`ccbefab0 00007ffe`98983034 test_minidump!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl # 288]
08 00000021`ccbefaf0 00007ffe`9abd1461 kernel32!BaseThreadInitThunk+0x14
09 00000021`ccbefb20 00000000`00000000 ntdll!RtlUserThreadStart+0x21