I want to stream some data from my program for other subscribers (other programs). These programs can use these data as streaming event.
What I want?
How it is done generally?
Any libraries or papers pointing to technique and pros/cons?
Security related Ideas?
I will dig more information on my own even if I get a small hint.
Example :
Program A : Object A changed =======> Program B : Report Change in Object A
Two things are generally used: sockets/pipes which are just your basic byte streams, and message passing which is a bit more complex, made for parallel use and horizontal scalability.
I am not sure if it is a bit advanced, but have a look at boost::asio
http://www.boost.org/doc/libs/1_39_0/doc/html/boost_asio/overview/core/basics.html
Best I found:
Google Protobuf
Facebook Thrift
Pros:
Helps with establishing the formats for streaming
Fast
Easy to build
Cons:
List item
Other top-level design issues (bandwidth control, cancelation) have to go on top.
Broadly it sounds like you're trying to perform inter-process communication, aka IPC.
In the tags to your question you refer to Windows. This link provides a broad list of the IPC options within Windows.
The tag list for your question also says platform-independent. Either a socket based solution or a Pipe based solution will mostly standard across a large number of platforms that you're likely to develop for. You can either use sockets directly or use one of the numerous cross-platform wrappers, eg. ZeroMQ and Boost, to hide some of the detail.
It's not clear from the question whether Program A and Program B are running on the same machine? If not then using sockets is a better approach.
Related
We have two applications: one uses C/Fortran, and the other one is using C++.
We want the two applications to exchange data preferably without a buffer file,
and we want to ensure synchronization between the two applications as they have two different time steps.
Any helpful ideas are deeply appreciated.
Simplest way would be to use message passing via sockets, TCP or UDP depending on your needs.
How to pass data using sockets in C
There are several ways of Inter-process communication, see this link comparison of IPC methods.
The choice here depends on several variables, such as: Are these applications running on different machines vs one, are they on different operating systems, how fast, and how reliable, and the complexity of the implementation.
My favorite in this case is Named Pipes, it is easy to implement and it is reliable Named Pipes:
I am not sure what is your OS and requirements but if both applications run on the same Linux machine you could also take a look at DBUS.
Another way to go is to use webservices like gSOAP. You can use it together with Eclipse to edit the WSDL and maybe SoaMoa for testing the webservice.
Both of these options are quite high level so you don't have to work directly with the sockets and serialization/deserialization.
What should be very easy to use would be zeromq for your case. Also a plus would be the great documentation.
I'm writing a distributed system wherein each node interfaces with local applications via some RESTful API, supports extensions and runtime customization, etc. It's a bit like an Enterprise Service Bus over the wide area, but with a lot else going on which is not related to the question at hand.
I've read a little about both MPI and Asio - originally I was set on Asio, but then I found MPI, and now again I'm thinking Asio is the better solution for me. MPI seems to provide a lot I don't need and a higher level of abstraction than I want - for example, I only need point-to-point communication, and it is important to me to be in control of when and what data is transmitted. (e.g. I have already designed a packet structure that I would conform to ideally)
So my primary question: it is worth it to start from a lower level with Asio, or should I try to graft MPI onto what i'm looking for? Further, are there 'skeleton applications' available which use MPI or Asio which would aid development? (Actually I am 100% new to C++.. ;) Or, does it make sense to use them in tandem?
For more perspective, maybe it's worth noting that I already have implemented the bulk of this project in Perl using Perl Object Environment, which itself is just an asynchronous event system with a ton of networking libraries.
Also, if it makes a difference, I would ideally use threads for this.
Honestly though I have not used Boost at all yet, as I hinted above, so any input is appreciated.
I should start by saying that I know nothing about Asio, but from the 2 minute scan of the website and the description of your problem, it sounds like while either would probably work for you, Asio might be simpler. MPI is really designed less for general purpose network communication and more for running applications where the set of processes and applications is a little more static. While it provides a client/server style interface if desired, it's not the main focus of the library.
MPI is also more difficult to use if you already have a packet structure designed. MPI is great for abstracting away the need to worry about packets, location of processes, etc. but if you've already taken all of that into account in you application, you have already done the hard work.
There have been at least one other discussion of Asio vs. MPI that you can take a look at (For distributed applications, which to use, ASIO vs. MPI?) to get more opinions too.
I want one application on a linux host to call procedures from applications on Win7x64 in a VM. I guess that the VM runs on the linux host does not matter since it should use sockets. How do approach this plan? Are there any libraries for that in the internet?
Edit:
Well I took am look at all of them. XMLRPC seems to be some kind of predecessor of SOAP. Prototbuf seem to me like having the focus on easy serialization of large objects. So my decision falls on SOAP. But now I am searching for a nice-to-use c++ binding. I read a few pages of the apache axis2 manual but its everything else but nice-to-use, and anyway in C.
Another question in my specific case is: Isn't it easier to send just send some textual numbers plus parameters to identify the procedures to call and reply in textual form, as I know the signature of the called procedures?
Edit2:
As SOAP is just a standard and RPC is just curious, (Imagined something like function pointers over IP :D) ,imho none of them all is a solution. But ZeroMQ is indeed a progress.
There is no magic to RPC. I would suggest having a look at a combination of ZeroMQ and Google protobuf. ZeroMQ is a very easy to use messaging system (your communication layer). You would use the REQ/REP pattern. Google protobuf is used to describe and serialize/deserialize your messages. Both libraries are cross-platform and even cross language (ruby, python, c++, etc. etc.)
Have you taken a look at SOAP? It has pros and cons but may meet your needs.
The fact that your Windows box is VM shouldn't make a difference providing that it's up and running, and so long as the system you choose treats the servers as logically separate sockets normally would, your solution will remain flexible.
One decent alternative is XMLRPC, which runs on top of HTTP. It's simpler than SOAP, at least. I've used it for Java-Python communication, and it was not much code then, but don't know any C++ libraries to recommend, so I'll just say http://en.wikipedia.org/wiki/XML-RPC and its C++ section to that.
Edit after edited question: For a really simple solution, just use plain HTTP, mapping request path to function and giving parameters with GET or POST. Return value in HTTP response data as simply as you can: plain text for number or string, for more complex return value a binary blob if binary compatibility can be ensured (same CPU, same word size, plain struct, using compiler struct packing options if needed), or as json (or even XML, but then you are starting to reinvent SOAP/XMLRPC...).
I've been playing around with Boost::Asio and checking out it's capability when it comes to n independent users collaborating to compute a function on shared inputs.
At one point I need to get each user to send to (resp. receive from) all the other users some data.
I guess effectively I'm looking for "active" establishment by each user opposed to the common client-server examples seen all over the internet.
I've got working code that allows a user to send data to another user. I'm having trouble seeing how I should get all users to be simultaneously sending and receiving from each other. I also can't seem to generalise multiple async_connects at once.
Are there any suggestions as to how this sort of protocol should be implemented? Should I be looking at using threads to manage both sending and receiving?
If you on Unix-based OS, such as Linux, I would start with much less complex solution using C programming language (or C++ if you need that), but without using Boost library. Just use plain old select syscall instead. See manpage and tutorial manpage.
Only after that I'd switch to something fancy like Boost.
See also this tutorial about BSD sockets programming.
I've been working on a few C++ projects now which involve doing some simple IPC using window messages. In a number of cases, some extra data is passed with the window messages by putting the data into a shared memory segment and then passing the pointer into the shared memory with the SendMessage call. Re-doing this all the time is annoying, but before resolving this dull repetition by inventing yet another IPC system I'd like to ask:
Is there an existing framework which satisfies the criteria?:
Written in C or C++ (we're using MSVC here)
As few dependencies as possible; in the best case, it's just a few source files which use plain C++ and Windows standard libraries and which can be compiled directly into the application/library.
Works on Windows XP and newer
Is built on window messages plus a shared memory segment
Proper error reporting would be highly desireable (remote process is gone, remote process doesn't understand given message, argument mismatch, etc.)
For what it's worth, COM is not really an option for us since it's so painful to work with it (unless you start using all kinds of wrappers on top of it which we'd like to avoid). I don't really need interfaces and all that stuff; just a convenient way to send messages with (in the best case arbitrary) arguments back and forth with a bit of error handling. Also, I discarded DBUS for doing so much more than I need.
I've had success using a memory mapped file for interprocess communication. I like it mainly because it's simple, fast, and available on any version of windows you're likely to come across (well, i doubt it will work on Win9x, but....)
There's a basic article at http://msdn.microsoft.com/en-us/library/ms810613.aspx (writtin in 1993!) that shows how to use them.
Although does not meet all of your criteria, ZeroMQ (http://www.zeromq.org/) might be worth looking at. It is small, simple and fast. Also it gives you message passing semantics which may help depending on the type of applications you are using
This question is almost 3 years old at the time of this answer, but I can't believe nobody has formally suggested Boost.Interprocess.
Light-weight, IPC, and C++ wrappings for WINAPI mechanisms are available.