Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I'm building a small system that contains many parts and I want to use a message pub/sub service to communicate between parts.
I read about some message queue services like RabbitMQ and ZeroMQ but I feel they are too complicated and feel like it was born for distributed system. All parts of my system will be written in C++/Linux and place on a small Raspberry Pi CPU, so I dont need feature like scalable, cross-platform, other language clients ...
Can you guys give me some advice about services or libraries that fit my need?
It's not that hard to do yourself actually.
First of all you need to define the protocol to be used. It can be very simple; like just a message type field, a payload size field, and the actual payload. The message types you need SUBSCRIBE, UNSUBSCRIBE and PUBLISH. The payload for the SUBSCRIBE and UNSUBSCRIBE messages is the name of a channel to subscribe to/unsubscribe from. The payload for the PUBLISH message is the channel name and the actual data (together with the size of the data of course).
To connect all subscribers you need a central server. All subscribers/publishers needs to connect to this server. The server program keeps a collection of queues, one for each channel. When a subscribe or publish message arrives to the server for a channel that doesn't exist, create a new message queue for this channel. For each channel the server also needs a collection of all clients subscribes to that channel. When a publish message arrives to the server, it's added to the end of the queue for the channel in question. While a channel queue is not empty, send a copy of it to all subscribers for that channel, and when all have received it then the message can be removed from the queue.
The hard part of the server is probably going to be the communication part. The easy part will be all queues and collections, as you can use the C++ standard containers for all of them (e.g. std::queue for the actual queue, std::unordered_map for channels, and std::vector for the collection of connected clients.)
The clients are very simple, all the need to do is being able to send the subscription and publish messages to the server, and receive the publish messages from the server. The hard part will once again be the actual communication part.
Postscript:
I've never actually built such a system my self, all of the above was just directly of the top of my head. An experienced programmer shouldn't need more than a couple of hours to implement the basics, maybe a couple of days for an inexperienced one.
For the communication you could use e.g. Boost ASIO, maybe use one threads per channel. And you can use something like Boost property tree to construct/parse JSON or XML messages.
However, all of this is kind of reinventing the wheel, when you could probably start using one of the existing systems like RabbitMQ in a couple of hours, saving you a lot of time (and a lot of bugs!)
As far as lightweight servers go, Redis supports pub/sub commands.
The Redis code itself is extremely tight (only a couple files), it's single-threaded (use an event-loop), and the memory consumption is quite low (compared to other Queing systems I have seen).
I know it is late but may be useful for others. I implemented a basic pub/sub in C++ using boost.
CppPubSub
Usage is very simple. From one end publish your data (generic map) on a channel and other side subscribe for same channel and receive the generic map again.
// you should create a singleton object of NotificationService class, make it accessible throughout your application.
INotificationService* pNotificationService = new NotificationService();
// Subscribe for the event.
function<NotificationHandler> fnNotificationHandler = bind(&SubscriberClass::NotificationHandlerFunction, this, std::placeholders::_1);
subscriptionToken = pNotificationService->Subscribe("TEST_CHANEL", fnNotificationHandler);
// Publish event
NotificationData _data;
_data["data1"] = "Hello";
_data["data2"] = "World";
pNotificationService->Publish("TEST_CHANEL", _data);
// Unsubscribe event.
pNotificationService->Unsubscribe(subscriptionToken);
Related
I am looking to use ZeroMQ to facilitate IPC in my embedded systems application, however, I'm not able to find many examples on using multiple 0MQ socket types in the same process.
For example, say I have a process called "antenna_mon" that monitors an antenna. I want to be able to send messages to this process and get responses back - a classic REQ-REP pattern. However, I also have a "cm" process, that publishes configuration changes to subscribers. I want antenna_mon to also subscribe to antenna configuration changes - PUB-SUB.
I found this example of reading from multiple sockets in the same process, but it seems sub optimal, because now you no longer block waiting for messages, you inefficiently check for messages constantly and go back to sleep.
Has anyone encountered this problem before? Am I just thinking about it wrong? Maybe I should have two threads - one for CM changes, one for REQ-REP servicing?
I would love any insights or examples of solving this type of problem.
Welcome to the very nature of distributed computing!
Yes, there are new perspectives one has to solve, once assembling a Project for a multi-agent domain, where more than one process works and communicates with it's respective peers ad-hoc.
A knowledge base, acquired from a soft Real-Time System or embedded systems design experience will help a lot here. If none such available, some similarities might be also chosen from GUI design, where a centerpiece is something like a lightweight .mainloop() scheduler, and most of the hard-work is embedded into round-robin polled GUI-devices and internal-state changes or external MMI-events are marshalled into event-triggered handlers.
ZeroMQ infrastructure gives one all the tools needed for such non-blocking, controllably poll-able ( scaleable, variable or adaptively ad-hoc adjustable poll-timeouts, not to overcome the given, design defined, round-trip duration of the controller .mainloop() ) and transport-agnostic, asynchronously operated, message dispatcher ( with thread-mapped performance scaling & priority tuning ).
What else one may need?
Well, just imagination and a lot of self-discipline to adhere the Zero-Copy, Zero-Sharing and Zero-Blocking design maxims.
The rest is in your hands.
Many "academic" examples may seem trivial and simplified, so as to illustrate just the currently discussed, or a feature demonstrated in some narrow perspective.
Not so in the real-life situations.
As an example, my distributed ML-engine uses a tandem of several PUSH/PULL pipelines for moving state data updates transfers and prediction forcasts + another PUSH/PULL for remote keyboard + a reversed .bind()/.connect() on PUB/SUB for easy broadcasting of distributed agents' telemetry to a remote centrally operated syslog and some additional PAIR/PAIR pipes, as processing requires.
( nota bene: one shall always bear in mind, that robust and error-resilient systems ought avoid to use a default REQ/REP Scaleable Formal Communication Pattern, as there is non-zero probability of falling the pairwise-stepped REQ/REP dual-FSA into an unsalvageable deadlock. Do not hesitate to read more about this smart tool. )
I have seen several questions about this, but none have answers I found satisfactory. This question, zeromq pattern: pub/sub with guaranteed delivery in particular is similar, though I am open to using any other zeromq mechanism to achieve the same effect.
My question is, is there any way to send messages in a fan-out pattern like publisher-subscriber in ZeroMQ with the assurance that the messages will be delivered? It seems as though a Dealer with zero-copy could do this okay, but it would be much messier than pub-sub. Is there a better option? What are the drawbacks of doing it this way besides having to write more code?
Reason for needing this:
I am writing a code to analyze data coming from instrumentation. The module which connects to the instrumentation needs to be able to broadcast data to other modules for them to analyze. They, in turn, need to broadcast their analyzed data to output modules.
At first glance pub-sub with ZeroMQ seemed perfect for the job, but messages get dropped if any subscriber slows down and hits the high watermark. In the case of this system, it is not acceptable for messages to be dropped at only a fraction of the modules because of event continuity. All the modules need to analyze an event for the output to be meaningful. However, if no modules received the messages for an event, that would be fine. For this reason, it would be okay to block the publisher (the instrumentation module) if one of the analysis modules hit the high watermark.
I suppose another alternative is to deal with missed messages after the fact, but that just wastes processing time on events that would be discarded later.
EDIT:
I guess thinking about this further, I currently expect a message sent = message delivered because I'm using inproc and communicating between threads. However, if I were to send messages over TCP there is a chance that the message could be lost even if ZeroMQ does not drop it on purpose. Does this mean I probably need to deal with dropped messages even if I use a blocking send? Are there any guarantees about message delivery with inproc?
In general, I think there's no way of providing a guarantee for pub/sub on its own with 0MQ. If you really need completely reliable messaging, you're going to have to roll your own.
Networks are inherently unreliable, which is why TCP does so much handshaking just to get packets across.
As ever, it's a balance between latency and throughput. If you're prepared to sacrifice throughput, you can do message handshaking yourself - perhaps using REQ/REP - and handle the broadcasting yourself.
The 0MQ guide has some ideas on how to go about at least part of what you want here.
I agree with SteveL. If you truly need 100% reliability (or close to it), ZeroMq is probably not your solution. You're better off going with commercial messaging products where guaranteed message delivery and persistence are addressed, otherwise, you'll be coding reliability features in ZeroMq and likely pull your hair out in the process. Would you implement your own app server if you required ACID compliance between your application and database? Unless you want to implement your own transaction manager, you'd buy WebLogic, WebSphere, or JBoss to do it for you.
Does this mean I probably need to deal with dropped messages even if I
use a blocking send?
I'd stay away from explicitly blocking anything, it's too brittle. A synchronous sender could hang indefinitely if something goes wrong on the consumption side. You could address this using polling and timeouts, but again, it's brittle and messy code; stick with asynchronous.
Are there any guarantees about message delivery with inproc?
Well, one thing is guaranteed; you're not dealing with physical sockets, so any network issues are eliminated.
This question comes up on search engines, so I just wanted to update.
You can stop ZeroMQ from dropping messages when using PUB sockets. You can set the ZMQ_XPUB_NODROP option, which will instead raise an error when the send buffer is full.
With that information, you could create something like a dead letter queue, as mentioned here, and keep trying to resend with sleeps in between.
Efficiently handling this problem may not be possible currently, as there does not appear to be a way to be notified when the send buffer in ZeroMQ is no longer full, which means timed sleeps / polling may be the only way to find out if the send queue has room again so the messages can be published.
I am coding a workload scheduler. I would like my piece of software to be a peer-to-peer scheduler, ie. a node only knows some neighbours (other nodes) and use them to reach other nodes.
Each node would have its own weighted-routing table to send messages to other peers (basically based on the number of hops), ie. "I want the master to give me my schedule" or "is resource A available on node B ?" : which neighbor is the closest to my target ?
For instance I have written my own routing protocol using XML-RPC (xmlrpc-c) and std::multimaps / std::maps.
I am thinking of using ZeroMQ to optimze my data streams :
queueing can reduce the network load between peers ;
subscriptions can be used to publish upgrades.
As a consequence :
I would need to open as many sockets as I would create new types of connections ;
Each node would need to be a client, a server, a publisher, a subscriber, a broker and a directory ;
I am not sure that my "peer-to-peer architecture" is compatible with the main purpose of ZeroMQ.
Do you think that ZeroMQ can be a helpful concept to use ?
It would be helpful to know exactly what you mean by "routing protocol".
That sounds like you mean the business logic of routing to a particular peer.
Knowing more fully what you're looking to achieve with ZeroMQ would also be helpful.
Have you read the ZeroMQ Guide?
ZeroMQ is a pretty different beast and without spending some time to play with it, you'll
likely find yourself confused. As a bonus, reading the guide will also help you answer
this question for yourself, since you know your requirements better.
ZeroMQ was designed to build robust distributed and multi-threaded applications. Since distributed applications can often take the form of "peer-to-peer", ZeroMQ could indeed be a good fit for your needs.
My situation: I would like to create a hobby project for improving my C++ involving real-time/latency programming.
I have decided I will write a small Java program which will send lots of random stock prices to a client, where the client will be written in C++ and accept all the prices.
I do not want the C++ client to have to poll/have a while loop which continuously checks for data even if there is none.
What options do I have for this? If it's easier to accomplish having a C++ server then that is not a problem.
I presume for starters I will have to use the boost ASIO package for networking?
I will be doing this on windows 7.
Why not just have the Java server accept connections and then wait for some duration of time. e.g. 10 seconds. Within that time if data becomes available, send it and close the connection.
Then the C++ client can have a thread which opens a connection whenever the previous one has completed.
That should give quite low latency without creating connections very often when there is no new data.
This is basically the Comet web programming model, which is used for many applications.
Think about how a web server receives data. When a URL is accessed the data is pushed to the server. The server need not poll the client (or indeed know anything about the client other than its a service pushing bytes towards it).
You could use a Java servlet to accept the data over HTTP and write the code in this fashion. Similarly, boost::asio has a server example that should get you started. Under the hood, you could enable persistent HTTP so that the connections aren't opened / closed frequently. This'll make the coding model much simpler.
I do not want the C++ client to have to poll/have a while loop which
continuously checks for data
Someone HAS to.
Need not be you. I've never used boost ASIO, but it might provide a callback registration. If yes, then just register a callback function of yours with boost, boost would do the waiting and give you a call back when it gets some data.
Other option is of course that you use some functions which are synchronous. Like (not a real function) Socket.read() which blocks the thread until there is data in the socket or it's closed. But in this case you're dedicating a thread of your own.
--edit--
Abt the communication itself. Just pick any IPC mechanism (sockets/pipes/files/...), someone already described one I think. Once you send the data, the data itself is "encoded" and "decoded" by you, so you can create your own protocol. E.g. "%%<STOCK_NAME>=<STOCK_PRICE>##" where "%%", = and ## (markers to mark start, mid and end) that you add on sender side and remove on receiver side to get stock name and price.
You can develop the protocol further based on your needs. Like you can also send buy/sell recommendation or, text alert msgs with major stock exchange news. As long as your client and server understand how the data is "encoded" you're good.
Finally, if you want to secure teh communication (and say you're not using some secure layer (SSL)) then you can encrypt the data. But that's a different chapter. :)
HTH
I'm working on a little client that interfaces with a game server. The server sends messages to the connected client over HTTP. Its relatively easy to parse the text messages coming into the client and form responses to send back.
Now what I'm trying to figure out is how to break up the process. I want to have a thread receiving the messages, parsing them into some data object, and placing them into an "incoming" queue to be processed. Then another thread reads messages from this queue and processes them (the brains or AI of the client) and makes responses back to the server.
I want to have the thread that watches the incoming data to do process the text (break up the messages, pull the important data out, etc.) so the AI thread doesn't have that overhead. But the problem is that the server can send a couple hundred different types of messages to the client (what the client can see, other players, if you are firing etc). I want to package this data into a neat little structure so the AI can handle it quickly, and the AI can be rewritten easily.
But how do I write a function that can pull something off a queue and know what type of message it is (so I know what data is contained within the message)?
Example messages:
ALIVE (tells you if you are alive)
It has only one data object, the current game time
DAM (tells if you are damaged)
Has a whole bunch of data, who damaged you, how much, what gun it is, if you can see them, etc.
It is possible to make an object that can handle all of these different message types and be interpreted by a single function? Very few messages have common attributes, so I don't think inheriting or just making one really big message class would be very good...
I'm not looking for a full solution here, just point me in the right direction and hopefully I'll be able to learn a bit on the way :-)
Basically what you're asking about is called a protocol: how data is exchanged and interpreted. Traditionally you'd define your own (and odds are they'd tend to start out rather naive -- sending plain text data with newlines to indicate the end of a command, or something like that). After a while you begin to realize that more is needed (how do you handle binary data? how do you handle errors? etc, etc)
Fortunately there are libraries out there to make life easier for you. These days I tend to favor simple RPC-like libraries for most of my needs. Examples include protocol buffers (by Google), Apache Thrift (by Facebook) and Apache Avro.