Crash dump analysis for multi threaded c++ application crash [closed] - c++

Question

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 1 year ago.
Improve this question
This is a question on generic scenario. I have a multi threaded c++ application which crashed and I have the crash dump. There might be hundreds of thread running and any of them might have caused the crash.
What would be the good approach to start analyzing the crash dump.
Among many threads (already logged information under dump file) how to find any specific thread causing the crash. Should I look for any specific criteria because I cannot go ahead and analyze all the threads and their stacks.
Any other useful information/clue you want to suggest.
Thank you very much in advance

Answer 1

We call the following code a minimum reproducible example and you should provide it as part of your question in the future.
It creates 100 threads, synchronizes them so that they all start running at the same time (needs C++ 20). One of the threads will generate an exception randomly, so that we don't know which one it was.
#include <random>
#include <thread>
#include <vector>
using namespace std::chrono_literals;
std::random_device rd;
std::mt19937 twister(rd());
std::uniform_int_distribution<int> dist(0, 100);
std::counting_semaphore<100> synchronizer(0);
void randomCrash()
{
synchronizer.acquire();
if (dist(twister) < 2)
{
throw std::exception();
}
std::this_thread::sleep_for(1000ms); // Ensure the thread is still there when we analyze the dump
}
int main()
{
std::vector<std::thread> threads;
for (int i = 0; i < 100; i++)
{
std::thread t(&randomCrash);
threads.push_back(std::move(t)); // Threads can't be copied, so move it
}
std::cout << "Created 100 threads.\r\n";
synchronizer.release(100);
std::cout << "100 threads running now.\r\n";
for (std::thread& th : threads)
{
if (th.joinable())
{
th.join();
}
}
std::cout << "Done. Ooops ... no exception happened? Well, that's randomness.\r\n";
}
If we now open the crash dump, we can see that it has already switched to thread 92 which caused the exception by looking at the prompt:
0:092>
But let's pretend that didn't work by using the command ~0s, so we're back on the main thread.
0:092> ~0s
ntdll!NtWaitForSingleObject+0x14:
00007ffc`cbcacc94 c3 ret
0:000>
Using the ~ command, you can identify the thread(s) which caused an exception:
0:000> ~
. 0 Id: 3edc.cc8 Suspend: 1 Teb: 0000004a`fb029000 Unfrozen
[...]
91 Id: 3edc.38d0 Suspend: 1 Teb: 0000004a`fb0df000 Unfrozen
# 92 Id: 3edc.2418 Suspend: 1 Teb: 0000004a`fb0e1000 Unfrozen
93 Id: 3edc.4788 Suspend: 1 Teb: 0000004a`fb0e3000 Unfrozen
[...]
103 Id: 3edc.43e4 Suspend: 1 Teb: 0000004a`fb0f7000 Unfrozen
The current thread has a dot (.) and the threads with an exception have a hash (#). Note that the dot may hide the hash if the current thread is the one which threw the exception. So you can easily switch to the thread
0:000> ~92s
ucrtbase!abort+0x4e:
00007ffc`c960286e cd29 int 29h
and look at the call stack
0:092> k
# Child-SP RetAddr Call Site
00 0000004a`80efe500 00007ffc`c9601f9f ucrtbase!abort+0x4e
01 0000004a`80efe530 00007ffc`b6e01aab ucrtbase!terminate+0x1f
02 0000004a`80efe560 00007ffc`b6e02317 VCRUNTIME140_1!FindHandler<__FrameHandler4>+0x45b [D:\...\frame.cpp # 693]
03 0000004a`80efe730 00007ffc`b6e040d9 VCRUNTIME140_1!__InternalCxxFrameHandler<__FrameHandler4>+0x267 [D:\...\frame.cpp # 357]
04 0000004a`80efe7d0 00007ffc`cbcb1f6f VCRUNTIME140_1!__CxxFrameHandler4+0xa9 [D:\...\risctrnsctrl.cpp # 306]
05 0000004a`80efe840 00007ffc`cbc61454 ntdll!RtlpExecuteHandlerForException+0xf
06 0000004a`80efe870 00007ffc`cbcb0a9e ntdll!RtlDispatchException+0x244
07 0000004a`80efef80 00007ffc`c96bd759 ntdll!KiUserExceptionDispatch+0x2e
08 0000004a`80eff6b0 00007ffc`a9f36480 KERNELBASE!RaiseException+0x69
09 0000004a`80eff790 00007ff7`49ec13fd VCRUNTIME140!_CxxThrowException+0x90 [D:\...\throw.cpp # 75]
0a 0000004a`80eff7f0 00007ff7`49ec1ecb WhichThreadCrashes!randomCrash+0x1bd [C:\...\WhichThreadCrashes.cpp # 19]
0b (Inline Function) --------`-------- WhichThreadCrashes!std::invoke+0x2 [C:\...\type_traits # 1585]
0c 0000004a`80eff850 00007ffc`c95b1bb2 WhichThreadCrashes!std::thread::_Invoke<std::tuple<void (__cdecl*)(void)>,0>+0xb [C:\...\thread # 55]
0d 0000004a`80eff880 00007ffc`cb7c7034 ucrtbase!thread_start<unsigned int (__cdecl*)(void *),1>+0x42
0e 0000004a`80eff8b0 00007ffc`cbc62651 kernel32!BaseThreadInitThunk+0x14
0f 0000004a`80eff8e0 00000000`00000000 ntdll!RtlUserThreadStart+0x21
So we can see it crashes in randomCrash().
Once you know how it works, you can also switch to the thread with the exception directly by using ~#s:
0:092> ~0s
ntdll!NtWaitForSingleObject+0x14:
00007ffc`cbcacc94 c3 ret
0:000> ~#s
ucrtbase!abort+0x4e:
00007ffc`c960286e cd29 int 29h
0:092>
Also, !analyze -v should give you
0:000> !analyze -v
[...]
STACK_COMMAND: ~92s ; .ecxr ; kb