In The Linux Programming Interface by Kerrisk
In the discussion so far, we have talked about using pipes
for communication between a parent and a child process. However,
pipes can be used for communication between any two (or more) related
processes, as long as the pipe was created by a common ancestor before
the series of fork() calls that led to the existence of the processes.
In distributed systems,
does the relationship of parent-child processes or "related" processes exist between processes running on different machines in the distributed systems?
does pipe exist as a way for communication between processes running on different machines in the distributed systems?
Thanks.
A concept of parent-child processes is not practical to design against in distributed environment, since a single process is not reliable.
Two patterns that are commonly used instead:
Service discovery, a way for a process detect other processes it depends upon.
RPC, to communicate with other processes over the network.
Related
I am new in multi-processing. I am creating an app in modern c++ that communicates with PLCs by modbus TCP. Thus, I have created my library to achieve that and I run the communication process on a different thread and the main thread communicates with it by a shared queue and events. It works well.
Now, I would like to design another app that has a different purpose that would sometime run in parallel to the first one. It still needs to communication with the PLCs. Should I also implement it with multithreading, opening a new canal of communication in a thread? Or should I implement some unique communication process and that every executables would connect to via some pipe? What is the difference between the two and is there a logic to use/exclude one of those solutions?
I see advantages in the multi-threading solution:
If the client fails, it only affects one program until reconnexion,
Shared memory is useful,
It is already implemented in my case
But I also see several advantages for the multi-processing approach. One communication client could be used by many apps. Also among them, it appears to be better for optimization.
I thank you in advance for your help.
I'm tasked with designing a small daemon (on Debian Linux) which will use a blackbox libfoo.so to communicate with an external EFT terminal. There are several, identical EFT terminals (around 100), and one libfoo.so instance can only communicate with a single terminal. There is an init call which essentially binds the instance to a terminal.
We're mainly using Java in our company, but this probably calls for a C++ implementation. The programming language is not yet defined.
As we'll need to handle concurrent communication with multiple terminals (around maybe 10 concurrent threads), we'll need to load several instances of the libfoo.so. I'm looking for design principles how to solve such a requirement (dlopen will only load an SO once, same thing for JNI). Do I need to spawn child processes? Copy/paste the SO and call it libfoo_1.so, libfoo_2.so etc. (aargh!) Are there other solutions?
Thanks
Simon
If the library has no API, meaning it runs its code using the .init mechanism, then you have no better choice than forking a parent process and dlopen the library in the child processes.
This is pretty simple actually, as long as you remember to wait for your child processes to terminate when needed.
If you need to handle communication between your parent and child processes, there are several Inter-process Communication methods available such as pipes.
If I use erlang as say a spawner process, it does major functions things that are not too speed critical like communicating with a server and handling server-client communication messages.
Then I choose to spawn a process in erlang and run c/c++ code from it, will this make my code faster?
Will erlang more efficiently handle multithreading than an equivalent in c/c++?
If I were to spawn many c nodes from erlang, would they stack on a single core or would erlang handle the multithreading. This is the main point I am wondering about.
You talk about two different concepts in your question: Erlang processes running C/C++ code and C nodes.
C or C++ code run from inside Erlang is not scheduled at all. In fact, it will block the Erlang scheduler for the current CPU core, so having long run times in C-land will most likely mess up your (Erlang) scheduling.
For example, on a quad core processor Erlang will by default create 4 scheduler threads, each which will occupy one core. Any process running C code will block the scheduler it is assigned to until that code has finished executing.
When running C nodes, you're totally on your own. An Erlang node does not know about the scheduling of a C node at all, it only cares about it's own scheduling. Of course, you could create your own scheduling such as dedicating one core to the C node and three to the Erlang node or something similar.
Erlang only manages concurrency using its own lightweight processes, which are not thread based. It usually only runs a few threads that service the many thousands of processes it can spawn, which are usually only one thread per CPU/core when SMP is enabled.
It definitely doesn't know anything about any external threads, and any external processes will just add communication overhead and complexity of managing memory and resources in the external process.
I have a very simple interface which needs to communicate between processes. It's currently implemented in a very simple manner (all single proc):
bool GetFoo(struct Foo *outFoo);
bool GetBar(struct Bar *getBar);
Such as:
Foo foo;
if (!GetFoo(&foo))
{
ReportError();
}
GetFoo fills out the "Foo" data structure with pure data (that is, no pointers - it's purely blitable data).
I need to convert this in-process function call to be between two processes on the same machine (in this case it's always the same machine). Is there a commonly followed idiom for cross-process calls in C++ on Windows? Is there some kind of intra-process communication supported by Windows? Should I be using shared memory instead?
One note: I don't want to take a dependency on anything other than the Windows APIs if at all possible.
You have many choices, but in my personal experience the most popular/easy to use ones are: sockets & pipes.
See here for all IPC options available for Windows.
I'm not sure what the most common is -- to truly answer that we'd have to have some kind of polling. That said, the most flexible way would probably be to expose the methods via DCOM.
A common method would be RPC, it can be implemented in various ways for instance as Billy mentioned using COM` (or DCOM if the processes are residing on different machines).
Although you may want to think about not doing direct RPC calls and instead have a named pipe between your processes which is used for the communication.
There are a number of ways to communicate between processes on the same computer in Windows. Which one works best depends on the relationship between the two processes. Is one process expected to start the other? In that case an out-of-process COM server would probably work best, since you can restart the other process if it is not already running.
If performance is critical, then shared memory will give you the most control the speed of passing the data between your processes.
One thing to think about is the failure semantics of running multiple processes. What does the calling process do if the callee is not there?
Both can be used for communicating between different processes,
what's the difference?
Windows has two kinds of pipes: anonymous pipes and named pipes. Anonymous pipes correspond (fairly) closely to Unix pipes -- typical usage is for a parent process to set them up to be inherited by a child process, often connected to the standard input, output and/or error streams of the child. At one time, anonymous pipes were implemented completely differently from named pipes so they didn't (for one example) support overlapped I/O. Since then, that's changed so an anonymous pipe is basically just a named pipe with a name you don't know, so you can't open it by name, but it still has all the other features of a named pipe (such as the aforementioned overlapped I/O capability).
Windows named pipes are much more like sockets. They originated with OS/2, where they were originally the primary mechanism for creating client/server applications. They were originally built around NetBIOS (i.e., used NetBIOS both for addressing and transport). They're tightly integrated with things like Windows authentication, so you can (for example) have a named pipe server impersonate the client to restrict the server to doing things the client would be able to do if logged in directly. More recently, MS has gone to some trouble to get rid of the dependence on NetBIOS, but even though they can now use IP as their transport (and DNS for addressing, IIRC) they're still used primarily for Windows machines. The primary use on other machines is to imitate Windows, such as by running Samba.
(Shamelessly cribbed from http://www.perlmonks.org/?node_id=180842)
Pipes are fast and reliable, because they are implemented in memory on a single host where both communicating processes run. Sockets are slower and less reliable, but are much more flexible since they allow communication between processes on different hosts.
(Off the top of my head)
Pipe: A tube with a small bowl at one end; used for smoking tobacco
Socket: Receptacle where something (a pipe, probe or end of a bone) is inserted
Anyways:
"A major difference between pipes and
sockets is that pipes require a common
parent process to set up the
communications channel. A connection
between sockets can be set up by two
unrelated processes, possibly residing
on different machines."
Sockets would use some sort of IP protocol like TCP/IP or UDP, thus would be slower, but your code'd be more portable if you'd need to communicate over a network. There is a third Shared mem approach and forth Mach ports (in this case I am not sure about it would work with Windows )
They both do the same function, the only difference is that pipes are more efficient as they are closest one can get to the barebones of internets. Sockets are an abstraction built on top of series of tubes (pipes) as a result they are slower (just as java is slower than native assembly code).