There is a tool written in C language, based on socket programming and multi-threading.
If I simply run the tool, there is no error and the tool does its work flawlessly.
But if I run the tool in debugging mode, switching between thread is raising SIGPIPE signal and thus data could not be transferred from the tool to its server.
Would handling the SIGPIPE using SIGNAL(SIGPIPE, SIG_IGN) make the thread work?
As far as I understand, this is being raised due to failing of Inter-Process communication between threads. If it is, then how can one debug such multi-threaded application?
Please suggest me the way to handle this issue.
This got resolved by carefully stepping into threads.
The tool not only creates processes but threads also thus improper gdbinit setting and stepping raises SIGPIPE signal.
This is illustrated in the other related post here.
Related
I think I've broken the rules on thread affinity somewhere in my Qt C++ application.
I'm getting lots of messages like this:
QBasicTimer::stop: Failed. Possibly trying to stop from a different thread
QBasicTimer::start: Timers cannot be started from another thread
I am using DDD/GDB. Can I insert some kind of trap which will catch these errors and give me a backtrace? Ideally suspending all other threads in the meantime?
I tried to halt on writing to stdout (b write if 1==$rdi as per this) but that didn't work.
I'm working on a embedded solution where two apps are working: one is the user interface and the other runs in the background providing data for the UI.
Recently I came across with a memory leak or similar error that is making Linux kill the secondary process, leaving the UI in a stopped situation without telling anything for the user about what is going on. I reached the problem by reading Linux's message log file and the software's print on terminal "Kill -myapp".
My question is: how could I notice such an event (and other similar) coming from the secondary software so I could properly report it to the user and log it? I mean, it's easy to have a look time to time in the process 'tree' to see if the secondary app is running and, if it's not, report a "some event happened" in the UI and it's also plausible to have a error-handler system inside the secondary app that makes it write in a log file what just happened and make the UI read that file for new entries from time to time, but how could the UI app knows with better details what is going on in such more abrupt events? (in this case, "Linux killed process", but it could be a "segmentation pipe" or any other) (and if there is another, better solution that this "constant read a log file produced by the secondary app", I'ld also like to know)
Notes: the UI is written in C++/Qt and the secondary app is in C. Although a solution using the Qt library would be welcomed, I think it would be better for the entire programming community if a more generalized solution was given.
You can create a signal handler for POSIX signals such as SIGKILL in the backend process and notify the ui using for example another signal with sigqueue. Any IPC mechanism should work, as long as it's async safe. Read more about signals: tutorial and manual
It may still be a good idea to check from the ui side periodically because the handler might not succeed.
As for a better way to check if process is alive compared to reading the log file:
Check if process exists given its pid
What is the technique to log the segmentation faults and run time errors which crash the program, through a remote logging library?
The language is C++.
Here is the solution for printing backtrace, when you get a segfault, as an example what you can do when such an error happens.
That leaves you a problem of logging the error to the remote library. I would suggest keeping the signal handler, as simple, as possible and logging to the local file, because you cannot assume, that previously initialized logging library works correctly, when segmentation fault occured.
What is the technique to log the segmentation faults and run time errors which crash the program, through a remote logging library?
From my experience, trying to log (remotely or into file) debugging messages while program is crashing might not be very reliable, especially if APP takes system down along with it:
With TCP connection you might lose last several messages while system is crashing. (TCP maintains data packet order and uses error correction, AFAIK. So if app just quits, some data can be lost before being transmitted)
With UDP connection you might lose messages because of the nature of UDP and receive them out-of-order
If you're writing into file, OS might discard most recent changes (buffers not flushed, journaled filesystem reverting to earlier state of the file).
Flushing buffers after every write or sending messages via TCP/UDP might induce performance penalties for a program that produces thousands of messages per second.
So as far as I know, the good idea is to maintain in-memory plaintext log-file and write a core dump once program has crashed. This way you'll be able to find contents of log file within core dump. Also writing into in-memory log will be significantly faster than writing into file or sending messages over network. Alternatively, you could use some kind of "dual logging" - write every debug message immediately into in-memory log, and then send them asynchronously (in another thread) into log file or over the network.
Handling of exceptions:
Platform-specific. On windows platform you can use _set_se_handlers and use it to either generate backtrace or to translate platform exceptions into c++ exceptions.
On linux I think you should be able to create a handler for SIGSEGV signal.
While catching segfault sounds like a decent idea, instead of trying to handle it from within the program it makes sense to generate core dump and bail. On windows you can use MiniDumpWriteDump from within the program and on linux system can be configured to produce core dumps in shell (ulimit -c, I think?).
I'd like to give some solutions:
using core dump and start a daemon to monitor and collect core dumps and send to your host.
GDB (with GdbServer), you can debug remotely and see backtrace if crashed.
To catch the segfault signal and send a log accordingly, read this post:
Is there a point to trapping "segfault"?
If it turns out that you wont be able to send the log from a signal handler (maybe the crash occurred before the logger has been intitialized), then you may need to write the info to file and have an external entity send it remotely.
EDIT: Putting back some original info to be able to send the core file remotely too
To be able to send the core file remotely, you'll need an external entity (a different process than the one that crashed) that will "wait" for core files and send them remotely as they appear. (possibly using scp) Additionally, the crashing process could catch the segfault signal and notify the monitoring process that a crash has occurred and a core file will be available soon.
I just came across some code which used the kill system call to send a SIGSEGV signal to an app. The rationale behind this was that this would force the app to core dump and quit. This seems so wrong to me, is this normal practice?
SIGQUIT is the correct signal to send to a program if you wish to produce a core dump. kill is the correct command line program to send signals (it is of course poorly named, since not all signals will kill the program).
Note, you should not send random signals to the program, not all of them will produce a core dump. Many of them will be handled by the program itself, either consumed, ignored, or induce other processing. Thus sending a SIGSEGV is wrong.
GCC Says:
http://www.gnu.org/s/libc/manual/html_node/Termination-Signals.html
POSIX/Unix Says:
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/signal.h.html
Yes. kill is somewhat misnamed -- it can send any signal. There are many uses for kill which don't result in the process being killed at all!
If you want to make an application dump it's core from another program, pretty much the only way to do it is via a signal. SEGV would be fine for this. Alternatively you can hook a debugger up to the program and freeze it and view it's registers and such without killing it.
If you want to dump a core from within an application there are nicer ways to do it, like via an assert().
So, no, it's not particularly wrong to send a SEGV to a program. You could also send things like SIGILL for illegal instruction, or a divide by zero signal. It's all fine.
The way to do it in Unix/Linux is to call abort() which will send SIGABORT to current process. The other option is raise() where you can specify what signal you want to send to current process.
Richard Stevens (_Advanced Programming in the UNIX Environment) wrote:
The generation of core is an implementation features of most Unix. It is not part of POSIX.1.
He lists 12 signals whose default action is to terminate with a core (ANSI: SIGABRT, SIGFPE, SIGILL, SIGSEGV, POSIX: SIGQUIT, Other: SIGBUS, SIGEMT, SIGIOT, SIGSYS, SIGTRAP, SIGXCPU, SIGXFSZ), all of them are overwritable (the two signals which aren't overwritable are SIGKILL and SIGSTOP).
I've never seen a way to generate a core which isn't the use of a default signal handler.
So if your goal is to generate a core and stop, the best is to choose a signal whose default handler does the job (SIGSEGV does the job), reset the default handler for the signal if you are using it and then use kill.
What's the best/proper method to collect log messages from multiple threads and have them all be displayed using a window? (while the threads are running).
I am currently trying to redirect stdout (cout) in to a wxTextCtrl but failing miserably when attempting to do so over multiple threads. Any help would be appreciated.
Logging has had a few major updates recently in the wxWidgets trunk, you can read about them here. One of them is to add support for logging from threads other than the main thread.
In what way is it failing? I'm not familiar with the wxTextCtrl, but unless it has built in synchronization (ie. its thread safe) that could be a big issue. The simplest way to protect a single resource like this is via a named 'mutex'. The following example is what you can do in each thread to make sure that only one accesses this resource (the output window) at a time.
// In each thread's initialization:
HANDLE mutexHandle = CreateMutex(0,FALSE,"__my_protecting_mutex__");
// Whenever you use the debug output:
WaitForSingleObject(mutexHandle, /* Timeout if you like. */ 0xFFFFFFFF );
// Do our printing here.
ReleaseMutex(mutexHandle);
// In each thread's cleanup:
CloseHandle(mutexHandle);
So this basically guarantees that only one thread can be in between the wait and the release. Now if your issue is actually routing to the wxTextCtrl, I would need some more details.
Edit: I just realized that what I posted is Windows specific, and maybe you aren't on windows! If you aren't I don't have experience with other platform's synchronization methods, but boost has some generic libraries which are not platform specific.