I have a crash handler installed at:
/proc/sys/kernel/core_pattern
Which pipes the incoming coredump to a file and then extracts the stacktrace via gdb.
The problem is that sometimes these coredumps can be very very large (often over 30GB). In these cases, users wait and wait while the coredump is written to disk before the application hangs.
Any suggestions on how to handle these very large coredumps in a non-blocking fashion? I don't care about the coredump per se, the stacktrace is what is valuable.
I don't have access to source, so answers such as https://serverfault.com/questions/627747/non-blocking-core-dump-on-linux aren't that practical.
Thanks!
I don't have access to source, so answers such as https://serverfault.com/questions/627747/non-blocking-core-dump-on-linux aren't that practical.
If the application is dynamically linked, you can LD_PRELOAD google coredumper into it, so the answer is somewhat practical.
Another possible alternative is http://github.com/tbricks/tbstack.
Certainly dumping 30GB of RAM just to throw most of it away is wasteful.
Related
When an executable is running on Linux, it generates processes, threads, I/O ... etc, and uses libraries from languages like C/C++, sometimes there might be timers in question, is it possible to monitor this? how can I get a deep dive into these software and processes and what is going on in the background?
I know this stuff is abstracted from me because I shouldn't be worrying about it as a regular user, but I'm curious to what would I see.
What I need to see are:
System calls for this process/thread.
Open/closed sockets.
Memory management and utilization, what block is being accessed.
Memory instructions.
If a process is depending on the results of another one.
If a process/thread terminates, why, and was it successful?
I/O operations and DB read/write if any.
The different things you wanted to monitor may require different tools. All tools I will mention below have extensive manual pages where you can find exactly how to use them.
System calls for this process/thread.
The strace command does exactly this - it lists exactly which system calls are invoked by your program. The ltrace tool is similar, but focuses on calls to library functions - not just system calls (which involve the kernel).
Open/closed sockets.
The strace/ltrace commands will list among other things socket creation, but if you want to know which sockets are open - connected, listening, and so on - right now, there is the netstat utility, which lists all the connected (or with "-a", also listening) sockets in the system, and which process they belong to.
Memory management and utilization, what block is being accessed.
Memory instructions.
Again ltrace will let you see all malloc()/free() calls, but to see exactly what memory is being access where, you'll need a debugger, like gdb. The thing is that almost everything your program does will be a "memory instruction" so you'll need to know exactly what you are looking for, with breakpoints, tracepoints, single-stepping, and so on, and usually don't just want to see every memory access in your program.
If you don't want to find all memory accesses but rather are searching for bugs in this area - like accessing memory after it's freed and so on, there are tools that help you find those more easily. One of them called ASAN ("Address Sanitizer") is built into the C++ compiler, so you can build with it enabled and get messages on bad access patterns. Another one you can use is valgrind.
Finally, if by "memory utilization" you meant to just check how much memory your process or thread is using, well, both ps and top can tell you that.
If a process is depending on the results of another one.
If a process/thread terminates, why, and was it successful?
Various tools I mentioned like strace/ltrace will let you know when the process they follow exits. Any process can print the exit code of one of its sub-processes, but I'm not aware of a tool which can print the exit status of all processes in the system.
I/O operations
There is iostat that can give you periodic summaries of how much IO was done to each disk. netstat -s gives you network statistics so you can see how many network operations were done. vmstat gives you, among other things, statistics on IO caused by swap in/out (in case this is a problem in your case).
and DB read/write if any.
This depends on your DB, I guess, and how you monitor it.
First, a little motivating background info: I've got a C++-based server process that runs on an embedded ARM/Linux-based computer. It works pretty well, but as part of its operation it creates a fairly large fixed-size array (e.g. dozens to hundreds of megabytes) of temporary/non-persistent state information, which it currently keeps on the heap, and it accesses and/or updates that data from time to time.
I'm investigating how far I can scale things up, and one problem I'm running into is that eventually (as I stress-test the server by making its configuration larger and larger), this data structure gets big enough to cause out-of-memory problems, and then the OOM killer shows up, and general unhappiness ensues. Note that this embedded configuration of Linux doesn't have swap enabled, and I can't (easily) enable a swap partition.
One idea I have on how to ameliorate the issue is to allocate this large array on the computer's local flash partition, instead of directly in RAM, and then use mmap() to make it appear to the server process like it's still in RAM. That would reduce RAM usage considerably, and my hope is that Linux's filesystem-cache would mask most of the resulting performance cost.
My only real concern is file management -- in particular, I'd like to avoid any chance of filling up the flash drive with "orphan" backing-store files (i.e. old files whose processes don't exist any longer, but the file is still present because its creating process crashed or by some other mistake forgot to delete it on exit). I'd also like to be able to run multiple instances of the server simultaneously on the same computer, without the instances interfering with each other.
My question is, does Linux have any built-it facility for handling this sort of use-case? I'm particularly imagining some way to flag a file (or an mmap() handle or similar) so that when the file that created the process exits-or-crashes, the OS automagically deletes the file (similar to the way Linux already automagically recovers all of the RAM that was allocated by a process, when the process exits-or-crashes).
Or, if Linux doesn't have any built-in auto-temp-file-cleanup feature, is there a "best practice" that people use to ensure that large temporary files don't end up filling up a drive due to unintentionally becoming persistent?
Note that AFAICT simply placing the file in /tmp won't help me, since /tmp is using a RAM-disk and therefore doesn't give me any RAM-usage advantage over simply allocating in-process heap storage.
Yes, and I do this all the time...
open the file, unlink it, use ftruncate or (better) posix_fallocate to make it the right size, then use mmap with MAP_SHARED to map it into your address space. You can then close the descriptor immediately if you want; the memory mapping itself will keep the file around.
For speed, you might find you want to help Linux manage its page cache. You can use posix_madvise with POSIX_MADV_WILLNEED to advise the kernel to page data in and POSIX_MADV_DONTNEED to advise the kernel to release the pages.
You might find that last does not work the way you want, especially for dirty pages. You can use sync_file_range to explicitly control flushing to disk. (Although in that case you will want to keep the file descriptor open.)
All of this is perfectly standard POSIX except for the Linux-specific sync_file_range.
Yes, You create/open the file. Then you remove() the file by its filename.
The file will still be open by your process and you can read/write it just like any opened file, and it will disappear when the process having the file opened exits.
I believe this behavior is mandated by posix, so it will work on any unix like system. Even at a hard reboot, the space will be reclaimed.
I believe this is filesystem-specific, but most Linux filesystems allow deletion of open files. The file will still exist until the last handle to it is closed. I would recommend that you open the file then delete it immediately and it will be automatically cleaned up when your process exits for any reason.
For further details, see this post: What happens to an open file handle on Linux if the pointed file gets moved, delete
In Unix systems, it is possible to dynamically monitor the system by reading data from /proc. I am hoping to implement this kind of monitoring in my application, by dynamically saving "current status" into a file. However, I do not want IO delay my program, so it would be best to make the file virtual, i.e. not stored into disk but actually in memory. Is there a way of doint that? Thanks for the hint!
Why not used shared memory and semaphores. Do a 'man shmget' as a starting point.
As an alternative you could make your application a socket server. Doing this way you can have it responding with status information only if being asked to (there's not even the need to keep updating a memory area with the current status) and you can also control your program from a remote machine. If the status itself is not a huge quantity of data I think this is the most flexible solution.
If also you make your application responding to an HTTP request (i don't mean handling all the http protocol possibilities, just the requests you want to support) then you can also avoid to write a client and if you want to write it anyway it's probably easier to find libraries and programmers able to do that.
Make it listening to port 80 and you could check your program over the internet getting through firewalls without efforts :-) (well... assuming the program itself can be reached from the internet, but even for that it's a simple and common thing to ask for to sysadmins).
Try FUSE. it is particularly useful for writing virtual file systems. There are already many filesystems on top of it.
I have no idea about your exact requirements, so I can only guess, but every file that under linux is put into /dev/shm is in ram. But that doesn't mean it is not doing I/O, just that the I/O is faster. If you don't want to do I/O via filedescriptors or similar, do as someone else suggested and use shared memory segments, but this is a bit harder for everyone to read. Having other programs just open and read a file, which then calls some functions in your program (which is done by /proc in kernel space) is not possible. Maybe also a filesystem socket or fifo is something that suits your needs more (e.g. when you are having a select/(e)poll routine anyways). When you have full control over the system, also tmpfs might be useful for you.
I have an application that is occasionally crashing in the release build; unfortunately, it looks like it is crashing in a 3rd party DLL. While trying to get a handle on it I've been swimming in a sea of HOW TOs and descriptions of how Windows creates crash dumps.
I was thinking about using this suggested mini-dump:
Getting a dump of a process that crashes on startup
I was planning on leaving this functionality in the code so the dump is always created for my application without having to have the PC set up beforehand. BTW, this application is not for distribution; it will be paired with our own hardware so I'm not concerned about random users having dump files building on their machines if the application happens to crash.
Additional note: all of the code is C/C++.
Is there a difference between what Dr. Watson (drwtsn32.exe) and this code will produce for me?
With Dr. Watson you'll only get the dumps when the Dr. sees you 'crashed'. Using the dumper API you'll be able to invoke it from any point in the app. Eg. you can trampoline the ordinary asserts to dump instead of showing a dialog. In my experience once you have dump support in your app you'll find it easier down the road to investigate, troubleshoot and fix various problems, simply because you can produce a full dump (or even a minidump) at any place you see fit in code.
There isn't much difference except that if you create your own minidump you have more control over the level of detail in it. By default Minidumps have the stack and some local variables, but creating your own gives you the option of creating a full memory dump also which may prove to be more useful (though this then may make collection of these dumps more problematic if the memory image is large).
If the crash happens reasonably frequently it may be worth just collecting some minidumps that drwatson (or werfault in Vista onwards) produces for you, as that may give you enough information. If it doesn't then you have the option of adding your own unhandled exception filter. Another thing that can happen is that the minidump you receive is the site of the crash rather than a first chance exception that may have arisen. Creating your own minidumps means that you're more likely to get a stack trace closer to where the problem is.
Another option, if you have a machine which exhibits the problem more often is to run ADPlus in the background -- it will sit and wait until your app crashes or throws exceptions then produce some useful log files. It does a similar same thing as the unhandled exception filter except it requires no changes to your app.
The biggest thing to watch out for is that MiniDumpWriteDump has to do memory allocation and file I/O. Calling it from inside the failed process may fail if e.g. heap structures are corrupted.
Calling MiniDumpWriteDump from a helper process works the same as using Dr. Watson, except that you have control over the dump options.
Recommended reading: loader lock deadlock in MiniDumpWriteDump
I don't think so. Although Dr Watson will generate full or mini dumps, you could use the ntsd debugger instead to get a lot more control of what data is included in the dumps.
Dr Watson's minidumps are good enough for most things, you get a call stack and variables. IF you need more, ntsd has a load of options.
The only benefit to using DrWatson is that is comes pre-installed on Windows.
I am having trouble finding a reasonable way to dump another process's memory to a file.
After extensive searching, I've been able to find a nice article at CodeProject that has *most* of the functionality I want:
Performing a hex dump of another process's memory. This does a good job of addressing permission issues and sets a good foundation.
However, with this utility I've seen that even a small process, such as an clean Notepad.exe or Calc.exe instance, can generate a dump file over 24MB in size, while the process itself runs under 20KB in memory according to TaskManager.
The article has lead me to believe that perhaps it is also dumping things in shared memory, possibly DLL space and the like. For example, a dump of Calc.exe will include sections that include method names (and presumably memory) from Kernel32.dll:
²³´µKERNEL32.dll ActivateActCtx AddAtomA AddAtomW AddConsoleAliasA AddConsoleAliasW AddLocalAlternateComputerNameA AddLocalAlternateComputerNameW AddRefActCtx AddVectoredExceptionHandler AllocConsole AllocateUserPhysicalPages AreFileApisANSI AssignProcessToJobObject AttachConsole BackupRead BackupSeek BackupWrite BaseCheckAppcompatCache BaseCleanupAppcompatCache
Is there a better way to dump the memory of another process that doesn't lead to this overhead, or perhaps an improvement upon the linked article's code that solves this problem? I want to get the memory that actually belongs to the process itself. I'd be okay with dumping the memory space of functions that are actually used in DLLs, but it seems unnecessary to dump the *entire* contents of multiple DLLs to get the running memory of the process.
I'm looking for a way to get the 30-60KB of a 30KB process, rather than 25MB for a 30KB process. Or at least closer than I can get currently.
Thanks in advance for your suggestions and guidance, it is appreciated.
Note: This is for a console utility, so GUI elements like the ones in the CodeProject article are unimportant.
You're basically asking for a user process minidump. The Windows Debug Helper library has a ready made function for this, MiniDumpWriteDump.
There is a coarse control over the amount of the detail contained in the mini dump from the MINIDUMP_TYPE parameter passed in to the function. The most basic, MiniDumpNormal, will only capture the call stack of each thread in the process. The amount of memory gets progressively more detailed with the other mini dump types.
You can also fine control the amount of information to be written into the mini dump by providing a callback to the MiniDumpWriteDump function and in the callback set the flags on the MINIDUMP_CALLBACK_OUTPUT structure.
The resulted mini dumps can be read with a debugger like Windbg or Visual Studio, or they can be processed by the various functions in the dbghelp.dll library.
Not really a "how to program it" answer, but I just found your question while looking for a tool that could do that, when I ran into PMDump:
http://ntsecurity.nu/toolbox/pmdump/
It's dead easy and simple to use, and creates correct dumps (I just tried it with some programs).