I am trying to debug a server application but I am running into some difficulties breaking where I need to. The application is broken up into two parts:
A server application, which spawns worker processes (not threads) to handle incoming requests. The server basically spawns off processes which will process incoming requests first-come first-served.
The server also loads plugins in the form of shared libraries. The shared library defines most of the services the server is able to process, so most of the actual processing is done here.
As an added nugget of joy, the worker processes "respawn" (i.e. exit and a new worker process is spawned) so the PIDs of the children change periodically. -_-'
Basically I need to debug a service that's called within the shared library but I don't know which process to attach to ahead of time since they grab requests ad-hoc. Attaching to the main process and setting a breakpoint hasn't seemed to work so far.
Is there a way to debug this shared library code without having to attach to a process in advance? Basically I'd want to debug the first process that called the function in question.
For the time being I'll probably try limiting the number of worker processes to 1 with no respawn, but it'd be good to know how to handle a scenario like this in the future, especially if I'd like to make sure it still works in the "release" configuration.
I'm running on a Linux platform attempting to debug this with DDD and GDB.
Edit: To help illustrate what I'm trying to accomplish, let me provide a brief proof on concept.
#include <iostream>
#include <stdlib.h>
#include <unistd.h>
using namespace std;
int important_function( const int child_id )
{
cout << "IMPORTANT(" << child_id << ")" << endl;
}
void child_task( const int child_id )
{
const int delay = 10 - child_id;
cout << "Child " << child_id << " started. Waiting " << delay << " seconds..." << endl;
sleep(delay);
important_function(child_id);
exit(0);
}
int main( void )
{
const int children = 10;
for (int i = 0; i < 10; ++i)
{
pid_t pid = fork();
if (pid < 0) cout << "Fork " << i << "failed." << endl;
else if (pid == 0) child_task(i);
}
sleep(10);
return 0;
}
This program will fork off 10 processes which will all sleep 10 - id seconds before calling important_function, the function in which I want to debug in the first calling child process (which should, here, be the last one I fork).
Setting the follow-fork-mode to child will let me follow through to the first child forked, which is not what I'm looking for. I'm looking for the first child that calls the important function.
Setting detach-on-fork off doesn't help, because it halts the parent process until the child process forked exits before continuing to fork the other processes (one at a time, after the last has exited).
In the real scenario, it is also important that I be able to attach on to an already running server application who's already spawned threads, and halt on the first of those that call the function.
I'm not sure if any of this is possible since I've not seen much documentation on it. Basically I want to debug the first application to call this line of code, no matter what process it's coming from. (While it's only my application processes that'll call the code, it seems like my problem may be more general: attaching to the first process that calls the code, no matter what its origin).
You can set a breakpoint at fork(), and then issue "continue" commands until the main process's next step is to spawn the child process you want to debug. At that point, set a breakpoint at the function you want to debug, and then issue a "set follow-fork-mode child" command to gdb. When you continue, gdb should hook you into the child process at the function where the breakpoint is.
If you issue the command "set detach-on-fork off", gdb will continue debugging the child processes. The process that hits the breakpoint in the library should halt when it reaches that breakpoint. The problem is that when detach-on-fork is off, gdb halts all the child processes that are forked when they start. I don't know of a way to tell it to keep executing these processes after forking.
A solution to this I believe would be to write a gdb script to switch to each process and issue a continue command. The process that hits the function with the breakpoint should stop.
A colleague offered another solution to the problem of getting each child to continue. You can leave "detach-on-fork" on, insert a print statement in each child process's entry point that prints out its process id, and then give it a statement telling it to wait for the change in a variable, like so:
{
volatile int foo = 1;
printf("execute \"gdb -p %u\" in a new terminal\n", (unsigned)getpid());
printf("once GDB is loaded, give it the following commands:\n");
printf(" set variable foo = 0\n");
printf(" c\n");
while (foo == 1) __asm__ __volatile__ ("":::"memory");
}
Then, start up gdb, start the main process, and pipe the output to a file. With a bash script, you can read in the process IDs of the children, start up multiple instances of gdb, attach each instance to one of the different child processes, and signal each to continue by clearing the variable "foo".
Related
The situation is: I have an external application so I don't have the source code and i can't change it. While running, the application writes logs to the stderr. The task is to write a program that check the output of it and separate some part of the output to other file. My solution is to start the app like
./externalApp 2>&1 | myApp
the myApp is a c++ app with the following source:
using namespace std;
int main ()
{
string str;
ofstream A;
A.open("A.log");
ofstream B;
B.open("B.log");
A << "test start" << endl;
int i = 0;
while (getline(cin,str))
{
if(str.find("asdasd") != string::npos)
{
A << str << endl;
}
else
{
B << str << endl;
}
++i;
}
A << "test end: " << i << " lines" << endl;
A.close();
B.close();
return 0;
}
The externalApp can crash or be terminated. A that moment the myApp gets terminated too and it is don't write the last lines and don't close the files. The file can be 60Gb or larger so saving it and processing it after not a variant.
Correction: My problem is that when the externalApp crash it terminate myApp. That mean any code after while block will never run. So the question is: Is there a way to run myApp even after the externalApp closed?
How can I do this task correctly? I interesed in any other idea to do this task.
There's nothing wrong with the shown code, and nothing in your question offers any evidence of anything being wrong with the shown code. No evidence was shown that your logging application actually received "the last lines" to be written from that external application. Most likely that external application simply failed to write them to standard output or error, before crashing.
The most likely explanation is that your external application checks if its standard output or error is connected to an interactive terminal; if so each line of its log message is followed by an explicit buffer flush. When the external application's standard output is a pipe, no such flushing takes place, so the log messages get buffered up, and are flushed only when the application's internal output buffer is full. This is a fairly common behavior. But because of that, when the external application crashes its last logged lines are lost forever. Because your logger never received them. Your logger can't do anything about log lines it never read.
In your situation, the only available option is to set up and connect a pseudo-tty device to the external application's standard output and error, making it think that's connected to an interactive terminal, while its output is actually captured by your application.
You can't do this from the shell. You need to write some code to set this up. You can start by reading the pty(7) manual page which explains the procedure to follow, at which point you will end up with file descriptors that you can take, and attach to your external application.
If you want your program to cleanly deal with the external program crashing you will probably need to handle SIGPIPE. The default behaviour of this signal is to terminate the process.
So the problem was not that when the first element of the pipe ended it terminate the second. The real problem was that the two app with pipes launched from bash script and when the bash script ended it terminated all of it child process. I solved it using
signal(SIGHUP,SIG_IGN);
that way my app executed to the end.
Thank you for all the answer at least I learned lot about the signals and pipes.
How to design a C/C++ program so that it can save some data after receiving interrupt signal.
I have a long running program that I might need to kill (say, by pressing Ctrl-C) before it finished running. When killed (as opposed to running to conclusion) the program should be able to save some variables to disk. I have several big Linux books, but not very sure where to start. A cookbook recipe would be very helpful.
Thank you.!
to do that, you need to make your program watch something, for example a global variable, that will tell him to stop what it is doing.
For example, supposing your long-running program execute a loop, you can do that :
g_shouldAbort = 0;
while(!finished)
{
// (do some computing)
if (g_shouldAbort)
{
// save variables and stuff
break; // exit the loop
}
}
with g_shouldAbort defined as a global volatile variable, like that :
static volatile int g_shouldAbort = 0;
(It is very important to declare it "volatile", or else the compiler, seeing that no one write it in the loop, may consider that if (g_shouldAbort) will always be false and optimize it away.)
then, using for example the signal API that other users suggested, you can do that :
void signal_handler(int sig_code)
{
if (sig_code == SIGUSR1) // user-defined signal 1
g_shouldAbort = 1;
}
(you need to register this handler of course, cf. here.
signal(SIGUSR, signal_handler);
Then, when you "send" the SIGUSR1 signal to your program (with the kill command for example), g_shouldAbort will be set to 1 and your program will stop its computing.
Hope this help !
NOTE : this technique is easy but crude. Using signals and global variables makes it difficult to use multiple threads of course, as other users have outlined.
What you want to do isn't trivial. You can start by installing a signal handler for SIGINT (C-c) using signal or sigaction but then the hard part starts.
The main problem is that in a signal handler you can only call async-signal-safe functions (or reentrant functions). Most library function can't be reliably considered reentrant. For instance, stdio functions, malloc, free and many others aren't reentrant.
So how do you handle this ? Set a flag in you handler (set some global variable done to 1) and look out for EINTR errors. It should be safe to do the cleanup outside the handler.
What you are trying to do falls under the rubric of checkpoint/restart.
There's several big problems with using a signal-driven scheme for checkpoint/restart. One is that signal handlers have to be very compact and very primitive. You cannot write the checkpoint inside your signal handler. Another problem is that your program can be anywhere in its execution state when the signal is sent. That random location almost certainly is not a safe point from which a checkpoint can be dropped. Yet another problem is that you need to outfit your program with some application-side checkpoint/restart capability.
Rather than rolling your own checkpoint/restart capability, I suggest you look into using a free one that already exists. gdb on linux provides a checkpoint/restart capability. Another is DMTCP, see http://dmtcp.sourceforge.net/index.html .
Use signal(2) or sigaction(2) to assign a function pointer to the SIGINT signal, and do your cleanups there.
Make your you enter only once in your save function
// somewhere in main
signal( SIGTERM, signalHandler );
signal( SIGINT, signalHandler );
void saveMyData()
{
// save some data here
}
void signalHandler( int signalNumber )
{
static pthread_once_t semaphore = PTHREAD_ONCE_INIT;
std::cout << "signal " << signalNumber << " received." << std::endl;
pthread_once( & semaphore, saveMyData );
}
If your process get 2 or more signals before you finish writing your file you'll save weird data
I am trying to spawn a process that executes a system command, while my own program still proceeds and two processes will run in parallel. I am working on linux.
I looked up online and sounds like I should use exec() family. But it doesn't work quite as what I expected. For example, in the following code, I only see "before" being printed, ,but not "done".
I am curious if I am issing anything?
#include <unistd.h>
#include <iostream>
using namespace std;
main()
{
cout << "before" << endl;
execl("/bin/ls", "/bin/ls", "-r", "-t", "-l", (char *) 0);
cout << "done" << endl;
}
[UPDATE]
Thank you for your guys comments. Now my program looks like this. Everything works fine except at the end, I have to press enter to finish the program. I am not sure why I have to press the last enter?
#include <unistd.h>
#include <iostream>
using namespace std;
main()
{
cout << "before" << endl;
int pid = fork();
cout << pid << endl;
if (pid==0) {
execl("/bin/ls", "ls", "-r", "-t", "-l", (char *) 0);
}
cout << "done" << endl;
}
You're missing a call to fork. All exec does is replace the current process image with that of the new program. Use fork to spawn a copy of your current process. Its return value will tell you whether it's the child or the original parent that's running. If it's the child, call exec.
Once you've made that change, it only appears that you need to press Enter for the programs to finish. What's actually happening is this: The parent process forks and executes the child process. Both processes run, and both processes print to stdout at the same time. Their output is garbled. The parent process has less to do than the child, so it terminates first. When it terminates, your shell, which was waiting for it, wakes and prints the usual prompt. Meanwhile, the child process is still running. It prints more file entries. Finally, it terminates. The shell isn't paying attention to the child process (its grandchild), so the shell has no reason to re-print the prompt. Look more carefully at the output you get, and you should be able to find your usual command prompt buried in the ls output above.
The cursor appears to be waiting for you to press a key. When you do, the shell prints a prompt, and all looks normal. But as far as the shell was concerned, all was already normal. You could have typed another command before. It would have looked a little strange, but the shell would have executed it normally because it only receives input from the keyboard, not from the child process printing additional characters to the screen.
If you use a program like top in a separate console window, you can watch and confirm that both programs have already finished running before you have to press Enter.
The Exec family of functions replaces the current process with the new executable.
To do what you need, use one of the fork() functions and have the child process exec the new image.
[response to update]
It is doing exactly what you told it: You don't have to press "enter" to finish the program: It has already exited. The shell has already given a prompt:
[wally#zenetfedora ~]$ ./z
before
22397
done
[wally#zenetfedora ~]$ 0 << here is the prompt (as well as the pid)
total 5102364
drwxr-xr-x. 2 wally wally 4096 2011-01-31 16:22 Templates
...
The output from ls takes awhile so it buries the prompt. If you want output to appear in a more logical order, add sleep(1) (or maybe longer) before the "done".
You're missing the part where execl() replaces your current program in memory with /bin/ls
I would suggest looking at popen() which will fork and exec a new process, then let you read or write to it via a pipe. (Or if you need read and write, fork() yourself, then exec())
If I run the classical bash forkbomb:
:(){ :&:&};:
my system hangs after a few seconds.
I tried to write a forkbomb in C, here is the code:
#include <unistd.h>
int main( )
{
while(1) {
fork();
}
return 0;
}
When I run it the system gets less responsive, but I can kill that process (even after minutes) just pressing ^C.
The above code is different from the original bash forkbomb I posted: it's something more like:
:( )
{
while true
do
:
done
}
(I didn't test it; don't know if it'd hang the system).
So I also tried to implement the original version; here the code:
#include <unistd.h>
inline void colon( const char *path )
{
pid_t pid = fork( );
if( pid == 0 ) {
execl( path, path, 0 );
}
}
int main( int argc, char **argv )
{
colon( argv[0] );
colon( argv[0] );
return 0;
}
But still nothing: I can run it and then easily kill it. It's not hanging my system.
Why?
What's so special about bash forkbombs? Is it because bash uses a lot more memory/CPU? Because bash processes call a lot more system calls (eg, to access filesystem) than mine?
That C program is tiny, seriously tiny. In addition, fork()'ing a program like that is very, very efficient. An interpreter, such as Bash, however, is much more expensive in terms of RAM usage, and needs to access the disk all the time.
Try running it for much longer. :)
The real cause for this is that in BASH the process you create is detached from the parent. If the parent process (the one you initially started) is killed, the rest of the processes live on. But in the C implementations you listed the child processes die if the parent is killed, so it's enough to bring down the initial process you started to bring down the whole tree of ever-forking processes.
I have not yet come up with a C forkbomb implementation that detaches child processes so that they're not killed if the parent dies. Links to such implementations would be appreciated.
In your bash forkbomb, you are putting new processes in new background process groups, so you are not going to be able to ^C them.
Thats basically because of the size.
When you are running the bash fork bomb it loads big monster programs into memory (with respect to your c program) each of them start hogging on your cpu resources.Definitely when big monsters start reproducing trouble comes more quickly than if bees start doing the same.
So the computer hangs immediately.However if you keep your c executable running for long it will hang the system too.Just that the time will be much much more.
If you want to compare the size of bash with size of c program check out /proc//status.
first with pid of any running instance of bash,and then with pid of any running instance of a c program
What is the best way to prevent a Linux program/daemon from being executed more than once at a given time?
The most common way is to create a PID file: define a location where the file will go (inside /var/run is common). On successful startup, you'll write your PID to this file. When deciding whether to start up, read the file and check to make sure that the referenced process doesn't exist (or if it does, that it's not an instance of your daemon: on Linux, you can look at /proc/$PID/exe). On shutdown, you may remove the file but it's not strictly necessary.
There are scripts to help you do this, you may find start-stop-daemon to be useful: it can use PID files or even just check globally for the existence of an executable. It's designed precisely for this task and was written to help people get it right.
Use the boost interprocess library to create a memory block that will be created by the process. If it already exists, it means that there is another instance of the process. Exit.
The more precise link to what you need would be this one.
#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/scoped_ptr.hpp>
int main()
{
using boost::interprocess;
boost::scoped_ptr<shared_memory_object> createSharedMemoryOrDie;
try
{
createSharedMemoryOrDie.reset(
new shared_memory_object(create_only, "shared_memory", read_write));
} catch(...)
{
// executable is already running
return 1;
}
// do your thing here
}
If you have access to the code (i.e. are writing it):
create a temporary file, lock it, remove when done, return 1; if file exists, or,
list processes, return 1; if the process name is in the list
If you don't:
create a launcher wrapper to the program that does one of the above
I do not know what your exact requirement is but I had a similar requirement; in that case I started my daemon from a Shell script ( it was a HP-UX machine) and before starting the daemon I checked if an exec by same name is already running. If it is; then don't start a new one.
By this way I was also able control the number of instances of a process.
I think this scheme should work (and is also robust against crashes):
Precondition: There is a PID file for your application (typically in /var/run/)
1. Try to open the PID file
2. If it does not exist, create it and write your PID to it. Continue with the rest of the program
3. If it exist, read the PID
4. If the PID is still running and is an instance of your program, then exit
5. If the PID does not exist or is used by another program, remove the PID file and go to step 2.
6. At program termination, remove the PID file.
The loop in step 5 ensures that, if two instances are started at the same time, only one will be running in the end.
Have a pid file and on the startup do a 'kill -0 <pid>'. Where is the value read from file. If the response is != 0 then the daemon is not alive and you might restart it
Another approach would be to bind to a port and handle the bind exception on the second attempt to start the daemon. If the port is in use then exit otherwise continue running the daemon.
I believe my solution is the simplest:
(don't use it if racing condition is a possible scenario, but on any other case this is a simple and satisfying solution)
#include <sys/types.h>
#include <unistd.h>
#include <sstream>
void main()
{
// get this process pid
pid_t pid = getpid();
// compose a bash command that:
// check if another process with the same name as yours
// but with different pid is running
std::stringstream command;
command << "ps -eo pid,comm | grep <process name> | grep -v " << pid;
int isRuning = system(command.str().c_str());
if (isRuning == 0) {
cout << "Another process already running. exiting." << endl;
return 1;
}
return 0;
}