I want to know how to use libumem on solaris. If I follow http://www.unix.com/man-page/OpenSolaris/3malloc/umem_debug/ and start the process with all the options, how will I get the output?
Can I get a text file of the results?
I have used wdb on HP-UX for the same. This generates a text file after the program exits, that I can analyze later. Can I do that same for libumem?
Note: This is a remote debugging, I will not have access to the system until afterwards.
You can create a core file of the process before it exits and examine the code with mdb later. One way to generate that core file at the right moment could be a dtrace script that will trigger the gcore just when exit is called.
I think libumem will generate a core when things goes wrong , you can analyze this core using mdb , certain commands like ::umem_status , umem_verify will help you in finding the corruption
Related
I have an ARM Cortex-M4 embedded system running FreeRTOS. I have implemented a crash log dump mechanism that writes a file to a storage device in the event of a fatal error, such as divide-by-zero, null-pointer, address errors, invalid instruction, or assertions. In this file, I write, among other things, the contents of the stack at the time of the exception.
This system captures crashes that happen in the field, so the idea is to analyze the crashes that are returned to me and determine the source of the problem as best I can. I can easily connect the log to the .elf that was generated when that version of the code was compiled. I just need a way to parse it.
I assume there are tools that can do this already (I can't be the first to do this), but I'm having trouble finding something on The Series of Tubes(tm) that fits the bill.
Is there a good starting point to create a tool that can parse the .elf from compilation and follow a stack dump to create such a report?
For the benefit of anyone else with this problem, here's what I am doing:
Google has a tool called "breakpad" that can parse .elf and crashlog files in the "minidump" format, which was originally created by Microsoft and adapted by Google for Chrome.
I am writing a tool to convert my stack traces to minidump format in the hopes of using the breakpad tools to analyze my crash logs.
Here's a link to breakpad: https://github.com/google/breakpad/blob/master/docs/getting_started_with_breakpad.md
Before I ask my question I describe briefly how I get backtrace from my clients.
I write a C++ application on linux (opensuse).
This application is launched by a script (the launcher), and if the application crashes, a core dump is generated (because ulimit -c unlimited).
Then the launcher generates a backtrace from the core file whith gdb, and starts again the application, which gives the possibility to the user to send a crash report containing the backtrace.
Now my problem and my question :
the problem : the core dump can be quite big (up to 5 or 10 GB). The copy of the core file takes a certain time (up to 2 minutes). This is a problem for my clients : it's too long between the crash and the application auto-restart.
the question : I generate the backtrace with gdb from 1) my program 2) the core file.
When the application crashes, a custom script is called by Piping core dumps to a program : could I, in this program, direclty attach gdb to the "dying" program and generate the backtrace earning the time to copy the core file on the HDD ?
Thanks in advance.
Just a remark:
I did all I can to reduce to the minimum the size of the core dump (no debug symbol, only dump what is needed for a backtrace (see Controlling which mappings are written to the core dump))
There is an excellent answer in how to obtain a stacktrace without generating a core dump in copy/paste form here.
It will generate a stacktrace to stderr, but you could easily do something different like post the stacktrace data using HTTP etc.
You do not have to add whole gdb just to do backtrace of crashing program. Just intercept signal like SIGBUS and when signalled you can use backtrace() or simply call gstack with pid of your program.
Is there any api's which load a crash dump, symbols from the symbol store and then allow programmatic access to the information?
I want to create a tool to auto process crash dumps and produce a report based on them.
Use a command file for Windbg which sets up your configs and then runs !analyze. Set windbg to output to a file (as described here: windbg: Command output to text file)
Then you can enjoy trying to sort, categorize, etc. the output from that. !analyzewill do a decent job of analyzing most crashes. If it can't, I doubt you'd be doing a better job with your own code, unless you have a lot of experience in analyzing crash logs by program (I haven't ever tried, although I have a decent idea of what to look for, I wouldn't necessarily want to write code to actually do it - I have written code to write logs when the system has crashed, so I know what you usually need).
The Windows dbghelp API loads symbols and line number information to support making stack back traces. Do not know API for reading crash dump files.
Debug diagnostic tool has a com interface that can load a dump, process it using symbols and give you the information back.
http://www.microsoft.com/en-au/download/details.aspx?id=26798
Tutorial: http://codenasarre.wordpress.com/2011/06/14/how-to-control-a-debugger-engine/
I'm dealing with some hundreds of thousands of lines of code, and I'm stumped where this process is accessing a particular file. I've given up searching the code, I just cannot find out.
So, here I am -- asking a question I'm almost certain there is no simple solution for.
I've tried FileMon, ProcMon from SysInternals, and while I can see the file got accessed, it doesn't show the call stack or any useful piece of information.
I wish I could break into the debugger when that happens; I thought maybe I could write some extension for FileMon that would signal to me when an access happens, and then I could throw a Debug.Break into my process.
Any insight or ideas appreciated.
Set a breakpoint on CreateFile(). Write one in main() so you can easily trace into it an find the API entrypoint. Switch to disassembly view before single-stepping.
Is the file created by the program or is it pre-existing? What would happen if you renamed the file on disk, maybe that could help you get a stack trace? If it's generated by the program does the file name adhere to a specific pattern, maybe you could look for the format string that populates this pattern, e.g. "c:\%d-%d-%d.txt", and then look for the lines that use this string.
If my C++ app crashes on Windows I want to send useful debugging information to our server.
On Linux I would use the GNU backtrace() function - is there an equivalent for Windows?
Is there a way to extract useful debugging information after a program has crashed? Or only from within the process?
(Advice along the lines of "test you app so it doesn't crash" is not helpful! - all non-trivial programs will have bugs)
The function Stackwalk64 can be used to snap a stack trace on Windows.
If you intend to use this function, you should be sure to compile your code with FPO disabled - without symbols, StackWalk64 won't be able to properly walk FPO'd frames.
You can get some code running in process at the time of the crash via a top-level __try/__except block by calling SetUnhandledExceptionFilter. This is a bit unreliable since it requires you to have code running inside a crashed process.
Alternatively, you can just the built-in Windows Error Reporting to collect crash data. This is more reliable, since it doesn't require you to add code running inside the compromised, crashed process. The only cost is to get a code-signing certificate, since you must submit a signed binary to the service. https://sysdev.microsoft.com/en-US/Hardware/signup/ has more details.
You can use the Windows API call MiniDumpWriteDump if you wish to roll your own code. Both Windows XP and Vist automate this process and you can sign up at https://winqual.microsoft.com to gain access to the error reports.
Also check out http://kb.mozillazine.org/Breakpad and http://www.codeproject.com/KB/debug/crash_report.aspx for other solutions.
This website provides quite a detailed overview of stack retrieval on Win32 after a C++ exception:
http://www.eptacom.net/pubblicazioni/pub_eng/except.html
Of course, this will only work from within the process, so if the process gets terminated or crashes to the point where it terminates before that code is run, it won't work.
Generate a minidump file. You can then load it up in windbg or Visual Studio and inspect the entire stack where the crash occurred.
Here's a good place to start reading.
Its quite simple to dump the current stackframe addresses into a log file. All you have to do is get such a function called on program faults (i.e. a interrupt handler in Windows) or asserts. This can be done at released versions as well. The log file then can be matched with a map file resulting in a call stack with function names.
I published a article about this some years ago.
See http://www.ddj.com/architect/185300443
Let me describe how I handle crashes in my C++/WTL application.
First, in the main function, I call _set_se_translator, and pass in a function that will throw a C++ exception instead of using structured windows exceptions. This function gets an error code, for which you can get a Windows error message via FormatMessage, and a PEXCEPTION_POINTERS argument, which you can use to write a minidump (code here). You can also check the exception code for certain "meltdown" errors that you should just bail from, like EXCEPTION_NONCONTINUABLE_EXCEPTION or EXCEPTION_STACK_OVERFLOW :) (If it's recoverable, I prompt the user to email me this minidump file.)
The minidump file itself can be opened in Visual Studio like a normal project, and providing you've created a .pdb file for your executable, you can run the project and it'll jump to the exact location of the crash, together with the call stack and registers, which can be examined from the debugger.
If you want to grab a callstack (plus other good info) for a runtime crash, on a release build even on site, then you need to set up Dr Watson (run DrWtsn32.exe). If you check the 'generate crash dumps' option, when an app crashes, it'll write a mini dump file to the path specified (called user.dmp).
You can take this, combine it with the symbols you created when you built your server (set this in your compiler/linker to generate pdb files - keep these safe at home, you use them to match the dump so they can work out the source where the crash occurred)
Get yourself windbg, open it and use the menu option to 'load crash dump'. Once it's loaded everything you can type '~#kp' to get a callstack for every thread (or click the button at the top for the current thread).
There's good articles to know how to do this all over the web, This one is my favourite, and you'll want to read this to get an understanding of how to helpyourself manage the symbols really easily.
You will have to set up a dump generation framework in your application, here is how you may do it.
You may then upload the dump file to the server for further analysis using dump analyzers like windbg.
You may want to use adplus to capture the crash callstack.
You can download and install Debugging tools for Windows.
Usage of adplus is mentioned here:
Adplus usage
This creates the complete crash or hang dump. Once you have the dump, Windbg comes to the rescue. Map the correct pdbs and symbols and you are all set to analyze the dump. To start with use the command "!analyze -v"