My project is quite large and multithreaded. There should be a bug which crashes the whole program.
For release version, it stuck sometimes, but does not appear very often.
For debug code, it is more likely to appear. And the stack trace of gdb is the following.
0 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:81
1 0x00007dff8270c700 in ?? ()
2 0x00007ffff6dde38d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
This information is not enough for me to locate the buggy code.
So my question is: how to get more information from the crash? any advanced use of gdb or other advanced tools?
============= Update ==============
One more information to add, after printing out all the thread ids, I figure out the thread that crashed. The only difference of the thread is that it is detached from the std thread object. If anyone has any experience with this, please tell me.
============= Update2 ================
This problem is not solved yet, and turn out to be a sever one.
If I run in the terminal, it'll crash the whole terminal and all other programs currently running under my username.
The system is then down and not accessible by ssh for a while. There are some other users getting broken pipe and it seems my program has made sshd not responsive.
After a while I'm able to login again, and find that the binary file of the program is broken (truncated) and need to recompile.
For me it looks like a memory or stack overwrite or access of dead pointers or objects.
To catch these kind of errors I like to use tools like efence or valgrind. With actual compilers you also can use thread sanitizer or the memory sanitizer. Both works with clang and g++.
If you can not catch the problem with that, you also should install the debug library version of the standard libs. Sometimes a wrong value crashes inside the g++lib or some other libs which results in hard to debug situations. With the debug infos installed you can catch this much easier.
Related
I'd like to collect just the stacktrace for crashes which would normally result in very large coredumps. It seems like one option is to attach to the process when it's in a crashed but not yet cleaned up state. I tried gstack which uses gdb but gdb didn't like the fact that the process had already crashed.
Does anyone know if libunwind could do this?
This question seemed relevant:
How to get a "backtrace" (like gdb) using only ptrace (linux, x86/x86_64)
and contained a reference to this example:
http://git.savannah.gnu.org/cgit/libunwind.git/plain/tests/test-ptrace.c?h=v1.0-stable
Thanks a bunch!
I am working on a big code base. It is heavily multithreaded.
After running the linux based application for a few hours, in the end, right before reporting, the application silences. It doesn't die, it doesn't crash, it just waits there. Joins, mutexes, condition variables ... any of these can be the culprit.
If it had crashed, I would at least have a chance to find the source using debugger. But this way, I have no clue how to use what tool to find the bug. I can't even post a code sample for you. The only thing that can possibly help is to tap MANY places with cout to get a visual where the application is.
Have you been in such a situation? What do you recommend?
If you're running under Linux then just use gdb to run the program. When the application 'silences', interrupt it with CTRL+C, then type backtrace to see the call stack. With this you will find out the function where your application was blocked.
Incase of linux, gdb will be great help. Another tool that can be of great help is strace (This can also be used where there are problems with program for with source is not readily available because strace does not need recompilation to trace them.)
strace shall intercept/record system calls that are called by a process and also the signals that are received by a process. It will be able to show the order of events and all the return/resumption paths of calls. This can take you almost closer to the area of problem.
iotop, LTTng and Ftrace are few of other tools that be helpful to you in this scenario.
I'm writing a software renderer in g++ under mingw32 in Windows 7, using NetBeans 7 as my IDE.
I've been needing to profile it of late, and this need has reached critical mass now that I'm past laying down the structure. I looked around, and to me this answer shows the most promise in being simultaneously cross-platform and keeping things simple.
The gist of that approach is that possibly the most basic (and in many ways, the most accurate) way to profile/optimise is to simply sample the stack directly every now and then by halting execution... Unfortunately, NetBeans won't pause. So I'm trying to find out how to do this sampling with gdb directly.
I don't know a great deal about gdb. What I can tell from the man pages though, is that you set breakpoints before running your executable. That doesn't help me.
Does anyone know of a simple approach to getting gdb (or other gnu tools) to either:
Sample the stack when I say so (preferable)
Take a whole bunch of samples at random intervals over a given period
...give my stated configuration?
Have you tried simply running your executable in gdb, and then just hitting ^C (Ctrl+C) when you want to interrupt it? That should drop you to gdb's prompt, where you can simply run the where command to see where you are, and then carry on execution with continue.
If you find yourself in a irrelevant thread (e.g. a looping UI thread), use thread, info threads and thread n to go to the correct one, then execute where.
Well. that´s the question. Just that.
I got an app made with SDL and OpenGL. SDL opens an extra window (which is the console) additional to the graphical one. When i execute i´m getting a 3 output error, the console tells me. And it gets closed (the graphical one).
But i know this happens when a SIGSEGV signal is received (don´t know how to capture it) and it appears in my IDE (Code::blocks) while debugging. But this time nothing appears, and everything works all right. But when executing it crashes..
What the...
What kind of error can i expect?. Sometimes it gets closed, sometimes it doesn´t. How to know what kind of problem i got?.
SIGSEGV is a segmentation fault, you're trying to access memory that isn't accessible to your process.
Assuming you're on a UNIXy system, you should be able to get the program to core dump and then look at the core dump in a debugger; alternatively, use a memory debugger like Valgrind to pinpoint the memory management issue that's causing this problem.
Sometimes my c++ program crashes in debug mode, and what I got is a message box saying that an assertion failed in some of the internal memory management routines (accessing unallocated memory etc.). But I don't know where that was called from, because I didn't get any stack trace. How do I get a stack trace or at least see where it fails in my code (instead of library/ built-in routines)?
If you have a crash, you can get information about where the crash happened whether you have a debug or a release build. And you can see the call stack even if you are on a computer that does not have the source code.
To do this you need to use the PDB file that was built with your EXE. Put the PDB file inside the same directory as the EXE that crashed. Note: Even if you have the same source code, building twice and using the first EXE and the second PDB won't work. You need to use the exact PDB that was built with your EXE.
Then attach a debugger to the process that crashed. Example: windbg or VS.
Then simply checkout your call stack, while also having your threads window open. You will have to select the thread that crashed and check on the callstack for that thread. Each thread has a different call stack.
If you already have your VS debugger attached, it will automatically go to the source code that is causing the crash for you.
If the crash is happening inside a library you are using that you don't have the PDB for. There is nothing you can do.
If you run the debug version on a machine with VS, it should offer to bring it up and let you see the stack trace.
The problem is that the real problem is not on the call stack any more. If you free a pointer twice, that can result in this problem somewhere else unrelated to the program (the next time anything accesses the heap datastructures)
I wrote this blog on some tips for getting the problem to show up in the call stack so you can figure out what is going on.
http://www.atalasoft.com/cs/blogs/loufranco/archive/2007/02/06/6-_2200_Pointers_2200_-on-Debugging-Unmanaged-Code.aspx
The best tip is to use the gflags utility to make pointer issues cause immediate problems.
You can trigger a mini-dump by setting a handler for uncaught exceptions. Here's an article that explains all about minidumps
Google actually implemented their own open source crash handler called BreakPad, which also mozilla use I think (that's if you want something more serious - a rich and robust crash handler).
If I remember correctly that message box should have a button which says 'retry'. This should then break the program (in the debugger) at the point where the assertion happened.
CrashFinder can help you locate the place of the exception given the DLL and the address of the exception reported.
You can take this code and integrate it into your application to have a stack trage automatically generated when there is an uncaught exception. This is generally performed using __try{} __except{} or with a call to SetUnhandledExceptionFilter which allows you to specify a callback to all unhandled exceptions.
You can also have a post-mortem debugger installed on the client system. This is a decent, general way to get information when you do not have dump creation built into your application (maybe for an older version for which you must still get information).
Dr. Watson on Windows can be installed by running: drwtsn32 -i Running drwtsn32 (without any options) will bring up the configuration dialog. This will allow the creation of crash dump files, which you can later analyze with WinDbg or something similar.
You can use Poppy for this. You just sprinkle some macros across your code and it will gather the stack trace, together with the actual parameter values, local variables, loop counters, etc. It is very lightweight so it can be left in the release build to gather this information from crashes on end-user machines