Looking to implement an an event based system for a project across multiple linux processes. Essentially, I want to be able to log an event and then notify multiple processes about the event (and also log via rsyslog). I have done this in the past with domain sockets and some custom code, but does anyone know of a better way utilizing Boost or a similar library?
Even better would be a subscription based model where only certain processes would receive certain events.
You have lots of options:
ZeroMQ.
It is high-speed, asynchronous library and contains lots of messaging patterns you can use, e.g. PUB-SUB
C++ Actor Framework -- C++11 actor model implementation
Dataflow.Signals and Boost.Asio -- in case you want to stick to BOOST-based solution. An example can be found here
Related
I am designing a C++ app which consists of largely independent plugins or modules which produce from time to time results useful for other plugins. For example, analysis module comes across some useful piece of data and sends it to action modules. Each module will run in its own thread; this is so analysis modules can continue gathering data while action modules handle the data at the rate they can.
I am looking for a suitable message passing architecture/design pattern. This stackoverflow thread gives some suggestions, but I'm not sure a plain interface will work in multi-threaded environment.
I was thinking of having some sort of channel based architecture, where each module broadcasts something on the channel and whichever module is interested in it - listens to. If there are some ready made libraries under liberal licence - the better.
I've been using ACE (Adaptive Communication Environment) for thread management, TCP/UDP communications, mutex relationships and programming.
ACE is a highly portable collection of invocations of platform core patterns. Best of all, it's free, open source, and currently under active development.
http://www.cs.wustl.edu/~schmidt/ACE.html
For your application, I would recommend looking specifically at class "ACE_Task_Base" to provide multi-threading, and "ACE_Reactor" to register all the handlers for your asynchronous callback architecture.
You might be interested to take a look at ZeroMQ library that acts like a concurrency framework also. Using this library your components would communicate with each other by sending messages to named ZeroMQ sockets. There are many sockets types (TCP, IPC, inproc) and several patterns available for request-reply and publish-subscribe messaging.
How does ZeroC ICE compare to 0MQ? I know that 0MQ/Crossroads and DDS are very similar, but cant seem to figure out where ICE comes in.
I need to quickly implement a system that offloads real-time market-data from C++ to C#, as a first phase of my project. The next phase will be to implement an Event Based architecture with an underlying Pub/Sub design.
I am willing to use TCP.. but the the system is currently running on a single 24 core server.. so an IPC option would be nice. From what I understand ICE is only TCP, while DDS and 0mq have an IPC option.
Currently ,I am leaning towards using Protobuf with either ICE or Crossroads IO. Got turned off from the OpenSplice DDS website. Ive done lots research on the various options, was originally considering OpenMPI + boost:mpi, but there does not seem to be MPI for .NET.
My question is:
How does ICE compare to 0MQ? I cant wrap my head around this. Was unable to find anything online that compares the two.
thanks in advance.
........
More about my project:
Currently using CMAKE C++ on Windows, but the plan is to move to CentOS at some point. An additional desired feature is to store the tic data and all the messages in a "NoSql" database such as Hbase/Hadoop or HDF5. Do any of these middleware/messaging/pub-sub libraries have any database integration?
Some thoughts about ZeroC:
Very fast; Able to have multiple endpoints; Able to load balance on the endpoints; Able to reconnect to a different endpoint in case one of the node goes down. This is transparent to the end user; Has good tool chain (IceGrid, IceStorm, IceBox, etc); Distributed, high availability, multiple failover, etc
Apart from that, I have used it for hot swapping code modules (something similar to Erlang) by having the client create the proxy with multiple endpoints, and later on bring down each endpoint for a quick upgrade one by one. With the transparent retry to a different endpoint, I could have the system up and running the whole time i did an upgrade. Not sure if this is an advertised feature or an unadvertised side-effect :)
Overall, it is very easy to scale out your servers if need be using ZeroC Ice.
I know ZeroMQ provides a fantastic set of tools and messaging patterns and I would keep using it for my pet projects. However, The problem that i see is that it is very easy to go overboard and lose track of all your distributed components. This is a must have in a distributed environment. How will you know where your clients/server are when you need to upgrade? If one of components down the chain does not receive a message, how to identify where the issue is? the publisher? the client? or any one of the bridges (REP/REQ, XREP/XREQ, etc) in between?
Overall, ZeroC provides a much better toolset and ecosystem for enterprise solutions.
And it is open source :)
Jaybny,
ZMQ:
If you want real good performance and the only job for Phase 1 of your job is to move data from C++ to C#, then Zmq is the best option.
Having a pub/sub model for event driven architecture is also something that Zmq can help you with, with its in-built messaging pattern.
Zmq also supports your IPC requirements in this case. Eg: you can have one instance of your application that consumes 24 cores by multithreading and communicating via IPC.
ZeroC Ice:
Ice is a RPC framework very much like CORBA.
Eg.
Socket/ZMQ - You send message over the wire. Read it at the other end, parse the message, do some action, etc.
ZeroC Ice - Create a contract between client and server. Contract is nothing but a template of a class. Now the client calls a proxy method of that class, and the server implements/actions it and returns the value. Thus, int result = mathClass.Add(10,20) is what the client calls. The method, parameters, etc is marshalled and sent to the server, server implements the Add method, returns the result, and the client gets 30 as the result. Thus on the client side, the api is nothing but a proxy for a servant running on a remote host.
Conclusion:
ZeroC ICE has some nice enterprisy features which are really good. However, for your project requirements, ZMQ is the right tool.
Hope this helps.
For me.. the correct answer was Crossroads I/O . It does everything I need.. but still unable to pub/sub when using protobufs... im sure ZeroC ICE is great for distributed IPC, but 0MQ/Crossroads, gives you the added flexibility to use Inter-Thread-Communication.
Note: on windows, 0mq does not have IPC.
So, all in all, the crossroads fork of 0mq is the best. but you will have to roll your own windows/ipc (or use tcp::127..) , and publisher side topic filtering features for pub/sub.
nanomsg, from the guy who wrote crossroads and 0mq (i think).
http://nanomsg.org/
I am looking for a logging facility for a multiprocess application which also contains multiple threads with in each process.
My current application is only multi-threaded and uses ACE_Message_Queue for sending logging events to the logger thread (The actual Log message is shared between application threads and the logger thread through a global array).
My new application contains multiple processes with multiple threads with in each process. How can I achieve a decent logging functionality in this scenario? Also looking to get rid of ACE in favor of Boost. My new application is supposed to run on Linux,Mac and Windows.
Thank you in advance.
Boost.Log (v2) is very easy to set-up and pretty comprehensive. It is not in the boost library yet but it will be soon. You can use it as a simple logger, or write filters and customize the logging process and output. I am using it for a project and very very happy with it. See this question.
Try to use log4cplus. It is an up-to date logging library allows using from multiple processes.
I'v a blog to show my logging practice about this question: http://peihanw.blogspot.com/2012/08/my-answer-to-stackoverflows-question.html
Hi I'm working on a c++ project that I'm trying to keep OS independent and I have two processes which need to communicate. I was thinking about setting up a 3rd process (possibly as a service?) to coordinate the other two, asynchronously.
Client 1 will tell the intermediate process when data is ready, and send the data to it. The intermediate process will then hold this data until client 2 tells it that it is ready for the data. If the intermediate process has not received new data from client 1, it will tell client 2 to wait.
Since I am trying to keep this OS independent I don't really know what to use. I have looked into using MPI but it doesn't really seem to fit this purpose. I have also looked into Boost.ASIO, Named Pipes, RPC's and RCF. Im currently programming in Windows but I'd like to avoid using the WIN_API so that the code could potentially be compiled in Linux.
Here's a little more detail on the two processes.
We have a back end process/model (client 1) that will receive initial inputs from a GUI (client 2, written in Qt) via the intermediate process. The model will then proceed to work until the end condition is met, sending data to the server as it becomes ready. The GUI will ask the intermediate process for data on regular intervals and will be told to wait if the model has not updated the data. As the data becomes available from the model we also want to be able to keep any previous data from the current session for exporting to a file if the user chooses to do so (i.e., we'll want the GUI to issue a command to the interface to export (or load) the data).
My modification privleges of the the back end/model are minimal, other than to adhere to the design outlined above. I have a decent amount of c++ experience but not much parallel/asynchronous application experience. Any help or direction is greatly appreciated.
Standard BSD TCP/IP socket are mostly platform independent. They work with some minor differences on both windows and Unices (like linux).
PS windows does not support AF_UNIX sockets.
I'd checkout the boost.interprocess library. If the two processes are on the same machine it has a number of different ways to communicate between processes, and do so in an platform independent manner.
I am not sure if you have considered the messaging system but if you are sending structured data between processes you should consider looking at google protocol buffers.
These related to the content of the messaging (what is passed) rather than how they are passed.
boost::asio is platform independent although it doesn't imply C++ at both ends. Of course, when you are using C++ you can use boost::asio as your form of transport.
I looking for cross platform multithread message queue implementation on c++ (not slot/signal) . Better if it based on subject-observer pattern.
ZeroMQ looks like it may be what you are looking for.
It is well documented with lots of examples, such as this one: http://www.zeromq.org/blog:multithreaded-server , which may be what you are trying to implement.
Take a look to the "ISL" open source project (stands for an "Internet Server Library", C++), which SVN-repository is located on http://svn.storozhilov.com/isl/ - isl::AbstractMessageBroker class is a good candidate for a basement of your job. This is quite simple but extensible skeleton for any message broker subsystem (DBus, JMS, AMQP, etc.). Each client is served by 2 threads from the pre-started thread's pool: one is for receiving message from the transport and processing message and another is for sending message to transport. So, actually in order to implement your messaging system you have to override at least following three virtual methods:
isl::AbstractMessageBroker::receiveMessage(...);
isl::AbstractMessageBroker::processMessage(...);
isl::AbstractMessageBroker::sendMessage(...);
Example of use is in trunk/examples/EchoMessageBroker directory. Responses client with echoed message, terminates connection on "bye\r\n" message, terminates itself on SIGINT.
You can try out Apache ActiveMQ. http://activemq.apache.org. Quite robust.We use it for a FIX messaging platform, quite responsive and easy to configure also.
Have a look at Intel's Open Source lib Threading Building Blocks. They are cross-platform and last time I looked they had lock-free containers.