I am implementing a c++ application which involves multiply users(for example. 128 users) with asymmetric roles(they all have different jobs). In such scenario, every user has to communicate with each other. Thus each pair of users need a bi-direction (virtual) communication channel between them.
There are three popular messaging patterns in this application.
Exchange: each user i has a message m_ij for user j != i to send. The length of m_ij is a public constant value. These messages m_ij are independent and have no relation with each other. This is something like "everyone has something for everyone".
Distribute: a (per-determined) user i_0 has a message m_j for ever other user j!=i_0. The length of the messages is a public constant value. It is a little similar to broadcast but the receivers are not receiving the same message.
Gather: a (per-determined) user i_0 receives a message m_j from ever other user j!=i_0. The length of the messages is a public constant value. This is very similar to vote mechanism.
Besides, there are also a small amount of two-party communication between some of the users.
The round-trip cost is very sensitive in the application. Thus a one-roundtrip implementation for these communication patterns is very desirable.
Besides, the bandwidth cost of the application is very high thus a non-blocking implementation is almost a must-have.
I first tried the classic server/client socket (https://www.geeksforgeeks.org/socket-programming-cc/) by having multiply ports and deploy a server/client pair between every two users. However it turns out to be a failure.
I also investgate the ZMQ library. But to my poor understanding I have to somehow handle "routing" on my own, which I am not capable of.
Nanomsg is another candidate to go with but none of the patterns it provides seem to match the requirements.
So, could anyone provide any idea about this challenge? Thanks in advance!
Related
I've been working on a Cards Against Humanity type game in which there are several, well defined message-sending patterns between clients and the server:
The server waits for the prompts & answers to be generated, then sends them over to the client.
The clients wait for the prompts to be received, then everyone besides the czar chooses their answer(s) and sends them to the server.
The server waits for the answers to be received and sends them back to everyone.
And so on.
However I've noticed that I have a bunch of super long functions that behave sort of similarly in that they receive some data, modify some state and notify other users through sockets, a lot of boilerplate basically.
So the pattern that I noticed here is:
someone waits for something to happen (so you need a synchronization mechanism, I used std::condition_variables)
do something (generate some data/prompt the user for input)
send stuff over to other participants
I thought of defining a class that basically has those 3 components - a condition variable, an std::function that modifies stuff, and a list of sockets to notify and the data to send them. I'm not really aware of any networking-application design patterns - is there an abstraction that fits what I've described in any way?
I am trying to solve the following problem and was wondering what is the best approach to apply?
I'd like to setup a versioned communication via ZeroMQ, which effectively means that any client first makes a handshake stating the version of its messaging protocol and than all subsequent requests are forwarded only to specific set of workers, i.e. the ones which can understand this protocol.
I saw the example of Router/Dealer but there forwarding occurs always to all workers.
IMO this is something like a session, which is established based on handshake and all future requests are made in a particular context. Can this be done with ZeroMQ?
I understand that I can send back some ID to the client and ask it to put in all future requests, but would like to avoid that kind of intrusiveness.
Just a side note: I implement this approach in C++. I don't mind if you answer represents a general idea taking in account features available in either in ZeroMQ C API or cppzmq wrapper. No need to write a fully fledged solution, just how it might be done.
Yes, this seems doable:
With a full respect not to enter into "in-band" signalling via ID/multipart message-processing, one may build the wish-to-have infrastructure using a mix of static and dynamic use of as-is ZeroMQ resources.
Step 0: your central authority handles initial "client" contact / handshaking / identity validation
Step 1: each "client" receives a set of directions, as it's identity/version got approved, based upon 0)
Step 2: ad-hoc instructed "client" may { .connect() | .bind() } with appropriate access-point
Step 3: as an architecture bonus, this can be enjoyed as distributed platform with re-negotiation(s) and node re-discoveries for even more robust, scalable-performance and raised security motivated scenarios
Our own imagination is the only ceiling:
In a few word, one may soon forget about the standard Scalable Formal Communication Patterns, these serve as rather a set of building blocks for ad-hoc defined architectures. That is the biggest power of the ZeroMQ or nanomsg to realise.
May read more on advanced use-cases in this ( and check the book there ... ).
I am looking to use ZeroMQ to facilitate IPC in my embedded systems application, however, I'm not able to find many examples on using multiple 0MQ socket types in the same process.
For example, say I have a process called "antenna_mon" that monitors an antenna. I want to be able to send messages to this process and get responses back - a classic REQ-REP pattern. However, I also have a "cm" process, that publishes configuration changes to subscribers. I want antenna_mon to also subscribe to antenna configuration changes - PUB-SUB.
I found this example of reading from multiple sockets in the same process, but it seems sub optimal, because now you no longer block waiting for messages, you inefficiently check for messages constantly and go back to sleep.
Has anyone encountered this problem before? Am I just thinking about it wrong? Maybe I should have two threads - one for CM changes, one for REQ-REP servicing?
I would love any insights or examples of solving this type of problem.
Welcome to the very nature of distributed computing!
Yes, there are new perspectives one has to solve, once assembling a Project for a multi-agent domain, where more than one process works and communicates with it's respective peers ad-hoc.
A knowledge base, acquired from a soft Real-Time System or embedded systems design experience will help a lot here. If none such available, some similarities might be also chosen from GUI design, where a centerpiece is something like a lightweight .mainloop() scheduler, and most of the hard-work is embedded into round-robin polled GUI-devices and internal-state changes or external MMI-events are marshalled into event-triggered handlers.
ZeroMQ infrastructure gives one all the tools needed for such non-blocking, controllably poll-able ( scaleable, variable or adaptively ad-hoc adjustable poll-timeouts, not to overcome the given, design defined, round-trip duration of the controller .mainloop() ) and transport-agnostic, asynchronously operated, message dispatcher ( with thread-mapped performance scaling & priority tuning ).
What else one may need?
Well, just imagination and a lot of self-discipline to adhere the Zero-Copy, Zero-Sharing and Zero-Blocking design maxims.
The rest is in your hands.
Many "academic" examples may seem trivial and simplified, so as to illustrate just the currently discussed, or a feature demonstrated in some narrow perspective.
Not so in the real-life situations.
As an example, my distributed ML-engine uses a tandem of several PUSH/PULL pipelines for moving state data updates transfers and prediction forcasts + another PUSH/PULL for remote keyboard + a reversed .bind()/.connect() on PUB/SUB for easy broadcasting of distributed agents' telemetry to a remote centrally operated syslog and some additional PAIR/PAIR pipes, as processing requires.
( nota bene: one shall always bear in mind, that robust and error-resilient systems ought avoid to use a default REQ/REP Scaleable Formal Communication Pattern, as there is non-zero probability of falling the pairwise-stepped REQ/REP dual-FSA into an unsalvageable deadlock. Do not hesitate to read more about this smart tool. )
Looking for the best approach to sending the same message to multiple destinations using TCP/IP sockets. I'm working with an existing VS 2010 C++ application on Windows. Hoping to use a standard library/design pattern approach that has many of the complexities already worked out if possible.
Here's one approach I'm thinking about.. One main thread retrieves messages from a database and adds them to some sort of thread safe queue. The application also has one thread for each client socket connection to some destination server. Each one of these threads would read from the thread safe queue, and send the message over a tcp/ip socket.
There may be better/simpler/more robust approaches than this one though..
The issues I have to be concerned about mostly are latency. The destinations could be anywhere, and there may be significant latency between one socket connection and another.
The messages must go in an exact FIFO order to all the destinations.
Also one destination will be considered the primary destination.. all messages must get to this destination, no exceptions. For the other destinations, i.e. non-primary, the messages are just copies and it's not absolutely critical if the non-primary destinations do not receive a few messages. At any point, one of the non-primary destinations could become the primary destination. If one of the destinations falls too far behind, then that thread would need to catch up to the primary destination, but skipping some messages.
Looking for any suggestions. Preliminary research so far, my situation appears to be something akin to a single producer and multiple consumers pattern, or possibly master-worker pattern in Java.
I need to implement this in C++ on Windows, and the application must use tcp/ip sockets using an existing defined protocol.
Any help at all would be greatly appreciated.
You need exactly two threads, one that saturates the IO channel to the database and another that saturates the IO channel to the network leading to the 12 servers. Unless you have multiple network interfaces (which you should think about!) you don't send things faster by using multiple threads. Also, since you don't have multiple threads taking care of the network, you don't have to sync them.
What you definitely need to know about is select(). In the case of WinSock, also take a look at WSAEventSelect/WaitForMultipleObjects. Basically, you take a message from the queue and then send it to all clients when they're ready. select() tells you when one of a set of sockets is ready to accept data, so you don't waste time waiting or block trying to send data. What you need to come up with is a schema to reconnect after broken connections, when to drop messages to lagging clients etc. Also, in case the throughput to the different targets varies a lot, you need to think about handling multiple messages in parallel. If they are small (less than a network packet's payload) it makes sense combining them anyway to avoid overhead.
I hope this short overview helps getting you started, otherwise I can elaborate on the details.
I've already developed some online games (like chess, checkers, risk clone) using server side programming (PHP and C++) and Flash (for the GUI). Now, I'd like to develop some kind of game portal (like www.mytopia.com). In order to do so, I must decide what is a good way to structure my server logic.
At first I thought in programming separated game servers for each game. In this way, each game will be an isolated program that opens a specific port to the client. I thought also in creating different servers to each game room (each game room allow 100 clients connected on the same time). Of course I'd use database to link everything (like highscores, etc).
Then, I guess it is not the best way to structure a game portal server. I'm reading about thread programming and I think that is the best way to do it. So, I thought in doing something like a connection thread that will listen only to new connection clients (that way every type of game client will connect in only one port), validate this client (login) and then tranfer this client to the specific game thread (like chess thread, checkers thread, etc). I'll be using select (or variants) to handle the asynchronous clients (I guess the "one thread per client" is not suited this time). This structure seems to be the best but how do I make the communication between threads? I've read about race conditions and global scope variables, so one solution is to have a global clients array (vector or map) that need to be locked by connection thread or game thread everytime it is changed (new connection, logout, change states, etc). Is it right?
Has anyone worked in anything like this? Any recommendations?
Thanks very much
A portal needs to be robust, scalable and extensible so that you can cope with larger audiences, more games/servers being added, etc. A good place to start is to look into the way MMOs and distributed systems are designed. This might help too: http://onlinegametechniques.blogspot.com/
Personally, I'd centralise the users by having an authentication server, then a separate game server for each game that validates users against the authentication server.
If you use threads you might have an easier time sharing data but you'll have to be more careful about security for exactly the same reason. That of course doesn't address MT issues in general.
TBH I've been doing a voip system where the server can send out many streams and the client can listen to many streams. The best architecture I've come up with so far is just to bind to a single port and use sendto and recvfrom to handle communications. If i receive a valid connect packet from a client on a new address then I add the client to an internal list and begin sending audio data to them. The packet receive and response management (RRM) all happens in one thread. The audio, as it becomes ready, then gets sent to all the clients from the audio thread. The clients respond saying they received the audio and that gets handle on the RRM thread. If the client fails to respond for longer than 30 seconds then I send a disconnect and remove the client from my internal list. I don't need to be particularly fault tolerant.
As for how to do this in a games situation my main thought was to send a set of impulse vectors (the current one and 'n' previous ones). This way if the client moves out of sync it can check how out of sync it is by checking the last few impulses it should have received for a given object. If it doesn't correspond to what its got then it can either correct or if it is too far out of sync it can ask for a game state reset. The idea being to try and avoid doig a full game state reset as it is going to be quite an expensive thing to do.
Obviously each packet would be hashed so the client can check the validity of incoming packets but it also allows for the client to ignore an invalid packet and still get the info it needs in the next update and thus helping prevent the state reset.
On top of that its worth doing things like keeping an eye on where the client is. There is no point in sending updates to a client when the client is looking in the other direction or there is something in the way (ie the client can't see the object its being told about). This also limits the effectiveness of a wallhack packet sniffing the incoming packets. Obviously you have to start sending things a tad before the object becomes visible, however, or you will get things popping into existence at inconvenient moments.
Anyway ... thats just some random thoughts. I have to add that I've never actually written a multiplayer engine for a game so I hope my musings help ya a bit :)