I was reading this article about Inter-Process communication with message passing. In order to run the examples and see the it says and I quote: "should be compiled and run at the same time". Someone has any ideas how exactly should I do this?
You can create BAT file and start both programs almost simultaneously:
START first.exe
START second.exe
"should be compiled and run at the same time"
I think it is clear a program can not be run until after it is compiled (this is a minor grammatical issue, and should be ignored).
In Linux, my preferred mechanism to launch a process is popen invoked by my C++ program.
In C++, it is easy for one thread (let us call this your start process) to use popen to launch as many processes as needed for your application (call these work processes).
I would then use messages to synchronize the start up (i.e. work processes should initialize themselves, then wait (at startup) for a go message from start process). These start up messages work in the same way your application would use them. This ensures that the multiple work processes are running at the same time (but within the constraints of how many cores your system has available).
Related
I have made a program in c++ for changing the password of a system and I wanna run it for every 2 hours,then I end up with two choice in c++ ,one is Sleep(ms) and the other is using recent thread lib this_thread::sleep_for(2h)[ 2h using std::chrono_literals].
The doubt I have been wandering is, does long pausing an exe will work the way we want, is it any other better way than what i mentioned?
I have also planned to put my exe as a windows service.
any other better way than what i mentioned?
Yes.
I suggest, that you do not pause the program at. Simply do the thing, and exit.
Extract the scheduling part to a separate program. You don't even need to write this scheduler, because it already exists on most operating systems.
If you have some task that must be run periodically with long periods of waiting, you should use a program or script, that does the task and exits, and a scheduler, which handles the waiting. There're also questions you need to consider, for example:
do you need to start your task if the scheduled time was missed (due to reboot, for example)
do you allow several of your tasks to run at once, if time it takes to complete is longer than wait period
What you're trying to do is to implement a scheduler yourself. If this is what you want, then sleep is a posix function, and chrono::thread::sleep_for is cross-platform, so it's better to use the second one.
However, it's not generally recommended to implement schedulers, moreover, so simple ones.
I have a device which has an library. Some of its functions are most awesomely ill-behaved, in the "occasionally hang forever" sense.
I have a program which uses this device. If/when it hangs, I need to be able to recover gracefully and reset it. The offending calls should return within milliseconds and are being called in a loop many many times per second.
My first question is: when a thread running the recalcitrant function hangs, what do I do? Even if I litter the thread with interruption points, this happens:
boost::this_thread::interruption_point(); // irrelevant, in the past
deviceLibrary.thatFunction(); // <-- hangs here forever
boost::this_thread::interruption_point(); // never gets here!
The only word I've read on what to do there is to modify the function itself, but that's out of the question for a variety of reasons -- not least of which is "this is already miles outside of my skill set".
I have tried asynchronous launching with C++11 futures:
// this was in a looping thread -- it does not work: wait_for sometimes never returns
std::future<void> future = std::async(std::launch::async,
[this] () { deviceLibrary.thatFunction(*data_ptr); });
if (future.wait_for(std::chrono::seconds(timeout)) == std::future_status::timeout) {
printf("no one will ever read this\n");
deviceLibrary.reset(); // this would work if it ever got here
}
No dice, in that or a number of variations.
I am now trying boost::asio with a thread_group of a number of worker threads running io_service::run(). It works magnificently until the second time it times out. Then I've run out of threads, because each hanging thread eats up one of my thread_group and it never comes back ever.
My latest idea is to call work_threads.create_thread to make a new thread to replace the now-hanging one. So my second question is: if this is a viable way of dealing with this, how should I cope with the slowly amassing group of hung threads? How do I remove them? Is it fine to leave them there?
Incidentally, I should mention that there is in fact a version of deviceLibrary.thatFunction() that has a timeout. It doesn't.
I found this answer but it's C# and Windows specific, and this one which seems relevant. But I'm not so sure about spawning hundreds of extra processes a second (edit: oh right; I could banish all the calls to one or two separate processes. If they communicate well enough and I can share the device between them. Hm...)
Pertinent background information: I'm using MSVC 2013 on Windows 7, but the code has to cross-compile for ARM on Debian with GCC 4.6 also. My level of C++ knowledge is... well... if it seems like I'm missing something obvious, I probably am.
Thanks!
If you want to reliably kill something that's out of your control and may hang, use a separate process.
While process isolation was once considered to be very 'heavy-handed', browsers like Chrome today will implement it on a per-tab basis. Each tab gets a process, the GUI has a process, and if the tab rendering dies it doesn't take down the whole browser.
How can Google Chrome isolate tabs into separate processes while looking like a single application?
Threads are simply not designed for letting a codebase defend itself from ill-behaved libraries. Processes are.
So define the services you need, put that all in one program using your flaky libraries, and use interprocess communication from your main app to speak with the bridge. If the bridge times out or has a problem due to the flakiness, kill it and restart it.
I am only going to answer this part of your text:
when a thread running the recalcitrant function hangs, what do I do?
A thread could invoke inline machine instructions.
These instructions might clear the interrupt flag.
This may cause the code to be non interruptible.
As long as it does not decide to return, you cannot force it to return.
You might be able to force it to die (eg kill the process containing the thread), but you cannot force the code to return.
I hope my answer convinces you that the answer recommending to use a bridge process is in fact what you should do.
The first thing you do is make sure that it's the library that's buggy. Then you create a minimal example that demonstrates the problem (if possible), and send a bug report and the example to the library's developer. Lastly, you cross your fingers and wait.
What you don't do is put your fingers in your ears and say "LALALALALA" while you hide the problem behind layers of crud in an attempt to pretend the problem is gone.
I'm investigating as to whether there is a framework/library that will help me implement a distributed computing system.
I have a master that has a large amount of data split up into files of a few hundred megabytes. The files would be chunked up into ~1MB pieces and distributed to workers for processing. Once initialized, the processing on each worker is dependent on state information obtained from the previous chunk, so workers must stay alive throughout the entire process, and the master needs to be able to send the right chunks to the right workers. One other thing to note is that this system is only a piece of a larger processing chain.
I did a little bit of looking into MPI (specifically Open MPI), but I'm not sure if it is the right fit. It seems to be geared to sending small messages (a few bytes), though I did find some charts that show it's throughput increases with larger files (up to 1/5 MB).
I'm concerned that there might not be a way to maintain the state unless it was constantly sent back and forth in messages. Looking at the structure of some MPI examples, it looked like master (rank 0) and workers (ranks 1-n) were a part of the same piece of and their actions were determined by conditionals. Can I have the workers stay alive (maintaining state) and wait for more messages to arrive?
Now that I'm writing this I'm thinking it would work. The rank 1...n section would just be a loop with a blocking receive followed by the processing code. The state would be maintained in that loop until a "no more data" message was received at which point it would send back the results. I might be beginning to grasp the MPI structure here...
My other question about MPI is how to actually run the code. Remember that this system is part of a larger system, so it needs to be called from some other code. The examples I've seen make use of mpirun, with which you can specify how the number of processors, or a hosts file. Can I get the same behavior by calling my MPI function from other code?
So my question is is MPI the right framework here? Is there something better suited to this task, or am I going to be doing this from scratch?
MPI seems reasonable option for your task. It uses the SPMD architecture, meaning you have the same program executing simultaneously on possibly distributed or even heterogeneous system. So the choice of process with rank 0 being the master and others being the workers is not mandatory, you can choose other patterns.
If you want to provide state for your application, you can use a constantly living MPI application and master process sending commands to worker ones over time. You probably should also consider saving that state to disk in order to provide more robustness to failures.
Running of an MPI process is done initially by mpirun. For example, you create some program program.c, then compile it using mpicc -o program program.c. Then you have to run mpirun -np 20 ./program <params> to run 20 processes. You will have 20 independent processes each having its own rank, so further progress is upon your application. The way these 20 processes will be distributed among nodes/processors is controlled by things like hostfile etc, should look at the documentation more closely.
If you want your code to be reusable, i.e. runnable from another MPI program, you generally should at least learn what MPI Communicator is and how to create/use one. There're articles on the net, keywords being "Creating MPI library".
If the code using your library is not to be in MPI itself, it's no huge problem, your program in MPI is not limited to MPI in communication. It just should communicate inside it's logic through MPI. You can call any program using mpirun, unless it tries calls to MPI library, it won't notice that it's being run under MPI.
If you are getting up and running with a cluster and mpi, then I recommend having a look at boost mpi. Its a c++ wrapper over an underlying mpi library (such as openmpi or mpich2). I found it very useful.
Your idea of sending messages back and forward, with each node requesting a new message when it is finished until a handshake saying "no more messages" is provided sounds a good one. I had a similar idea, and got a simple version up and running. I just put it on github for you in case you want to have a look. https://github.com/thshorrock/mpi_manager. Most of the code is in the header file:
https://github.com/thshorrock/mpi_manager/blob/master/include/mpi_manager/mpi_manager.hpp
Note, this was just a bit of code that was used to get me up and running, its not fully documented, and not a final version but its fairly short, works fine for my purposes and should provide a starting point for you.
Have a look at FastFlow. They use a data flow model to process data. It is extremely efficient if this model is suitable for you.
RayPlatform is a MPI framework for C++. You need to define plugins for your application (like modules in Linux).
RayPlatform is licensed under the LGPLv3.
Link: https://github.com/sebhtml/RayPlatform
It is well documented also.
An example application using RayPlatform: https://github.com/sebhtml/RayPlatform-example
edit: added link
I'm writing a program (A genetic algorithm implementation) which executes another program using "system" method to calculate fitness. The problem is that another program sometimes hangs for unlimited amount of time. How can I execute a program with some time limit from C++.
Both POSIX and c++ solutions are appreciated. And more or less this will be run once application so solution doesn't have to be very elegant.
I'm running Linux CentOS distribution and am testing on Cygwin. For compiler I use gcc 4.1.2 with boost library.
Any help is apreciated
Instead of system, execute the program with the fork/exec idiom. Before the exec, set RLIMIT_CPU to a maximum value with setrlimit in the child.
Make sure the child does not ignore SIGXCPU (which is very unlikely).
You could create a timer (with boost timer for example) and then try to kill the child process... this assume that you use fork and exec to launch all your child, and you stored each pid.
If this 'another' program is yours or you have sources under public license it's better probably to make it not a separate program but a separate thread in the main program. In this case you can easily control it.
If this 'another' program is yours or you have sources under public license but don't want (or can't) follow the suggestion above may be it is easier to fix the program to prevent hanging.
Shitty method:
do fork(), remeber PID, call exec*("my-prog", ...)
create thread in the main program with timer.
when time fires kill the process using kill() and PID remembered.
I've often heard that using system("PAUSE") is bad practice and to use std::cin.get() instead. Now my understanding of system calls is that they take a string which they enter into a system command line and talk with the OS, so PAUSE is a DOS command that pauses the output in the command window. I assume this works similarly with Mac and unix with different keywords, and using system calls is discouraged because of a lack of cross OS compatibility. (If I'm wrong with any of this, please correct me)
my question is this: When is it appropriate to use system() calls? How should they be applied? When should they NOT be applied?
system("PAUSE") is certainly less than ideal. using a call to system creates a subprocess, which on windows is fairly expensive and in any case not terribly cheap on any operating system. On embedded systems the memory overhead is significant.
If there is any way to do it without much pain natively then do it. In the case of waiting for the user to press a single button, cin.get() will be very hard to beat. In this case, your applications process will just block on stdin, setting only a few flags visible to the kernel, and most importantly, allocates no new memory and creates no new scheduling entities, not even an interrupt handler.
Additionally, it will work the same on all operating systems with all c++ compilers, since it uses only a very basic feature of a very standard part of the language, rather than depend on anything the OS provides.
EDIT: predicting your concern that it doesn't matter if it's expensive because the whole idea is to pause. Well, first off, if its expensive, then it's going to hurt performance for anything else that might be going on. Ever notice (on windows) when one application is launching, other, already open apps become less responsive too? Additionally, your user might not be a live human, but rather another program working on behalf of a human user (Say, a shell script). The script already knows what to do next and can pre-fill stdin with a character to skip over the wait. If you have used a subprocess here, the script will experience a (noticeable to a human) delay. If the script is doing this hundreds (or hundreds of millions!) of times, a script that could take seconds to run now takes days or years.
EDIT2: when to use system(): when you need to do something that another process does, that you can't do easily. system() isn't always the best candidate because it does two things that are somewhat limiting. First, the only way to communicate with the subprocess is by command line arguments as input and return value as output. The second is that the parent process blocks until the child process has completed. These two factors limit the cases in which system is useable.
on unixy systems, most subprocesses happen with fork because it allows the same program to continue in the same place as two separate processes, one as a child of the other (which is hardly noticeable unless you ask for it from the OS). On Linux, this is especially well optimized, and about as cheap as creating a pthread. Even on systems where this is not as fast, it is still very useful (as demonstrated by the apache process-pool methodology) (unavailable on windows/link to unix docs)
other cases (on windows too!) are often handled by popen or exec family of functions. popen creates a subprocess and a brand new pipe connecting to the subprocesses' stdin or stdout. Both parent and child processes can then run concurrently and communicate quite easily. (link to windows docs/link to unix docs)
exec* family of functions (there are several, execl, execv and so on) on the other hand causes the current program to be replaced by the new program. The original program exits invisibly and the new process takes over. When then new process returns, it will return to whatever called the original process, as if that process had returned at that point instead of vanishing. The advantage of this over exit(system("command")) is that no new process is created, saving time and memory (though not always terribly much) (link to windows docs /link to unix docs)
system could plausibly be used by some scripted tool to invoke several steps in some recipe action. For example, at a certain point, a program could use system to invoke a text editor to edit some configuration file. It need not concern itself too much with what happens, but it should certainly wait until the user has saved and closed the editor before continuing. It can then use the return value to find out if the editing session was successful, in the sense that the editor actually opened the requested file (and that the editor itself existed at all!), but will read the actual results of the session from the edited file directly, rather than communicating with the subprocess. (link to windows docs/link to unix docs)
System calls are sent to the shell or command line interpreter of the OS (dos, bash, etc) and its up to the shell to do what it wants with this command.
You would avoid using these kind of calls as it would reduce your programs portability to work with other operating systems. I would think only when you are absolutely sure that your code is targeting a specific OS that you should use such calls.
But my question is this: When is it appropriate to use system() calls? How should they be applied?
When you can't do the thing you're trying to do with your own code or a library (or the cost of implementing it outweighs the cost of launching a new process to do so). system() is pretty costly in terms of system resources compared to cin.get(), and as such it should only be used when absolutely necessary. Remember that system() typically launches both an entire new shell and whatever program you asked it to run, so thats two new executables being launched.
By the way, system() call should never be used with binaries with SUID or SGID bit set, quoting from the man page:
Do not use system() from a program with set-user-ID or set-group-ID
privileges, because strange values for some environment variables
might be used to subvert system integrity. Use the exec(3) family of
functions instead, but not execlp(3) or execvp(3). system() will not,
in fact, work properly from programs with set-user-ID or set-group-ID
privileges on systems on which /bin/sh is bash version 2, since bash 2
drops privileges on startup.
system() is used to ask the operating system to run a program.
Why would your program want the operating system to run a program? Well there are cases. Sometimes an external program or operating system command can perform a task that is hard to do in your own program. For example, an external program may operate with elevated privileges or access propriety data formats.
The system() function, itself, is fairly portable but the command string you pass it is likely to be very platform-specific -- though the command string can be pulled from local configuration data to make it more platform-agnostic.
Other functions like fork(), exec*(), spawn*() and CreateProcess() will give you much more control over the way you run the external program, but are platform-specific and may not be available on your platform of choice.
system("PAUSE") is an old DOS trick and is generally considered to be fairly grotty style these days.
As far as i know system("PAUSE") is a windows only thing, and that is why it is frowned upon.