I'm trying to implement Algorithm 2 (page 4) from this scientific paper (https://www.cse.ust.hk/~yike/pods10-cdsample.pdf) in Flink and I would be glad if someone helped me with some tips on how to approach this task (which State should I use, maybe Mapstate?, how to implement broadcast in Flink, etc.).
These algorithms need two-way communication between the coordinator and the workers. To achieve this with the DataStream API you'll need to have some sort of pub-sub channel between the workers and the coordinator so the workers can message the coordinator, and then the coordinator can broadcast its updates out to the workers.
There's an example of how to use Flink's Broadcast State Pattern in https://github.com/ververica/flink-training-exercises/blob/master/src/main/java/com/ververica/flinktraining/examples/datastream_java/broadcast/BroadcastState.java; hopefully that will help you get started.
Alternatively, it might be more natural to implement this use case with Flink's Stateful Functions API. The workers could each register themselves with the coordinator, which could remember their addresses and then message them all as needed.
As for the type of state the coordinator should use, yes, I think MapState is probably the best choice (assuming I've grokked the algorithm approximately correctly).
Related
I am looking to use ZeroMQ to facilitate IPC in my embedded systems application, however, I'm not able to find many examples on using multiple 0MQ socket types in the same process.
For example, say I have a process called "antenna_mon" that monitors an antenna. I want to be able to send messages to this process and get responses back - a classic REQ-REP pattern. However, I also have a "cm" process, that publishes configuration changes to subscribers. I want antenna_mon to also subscribe to antenna configuration changes - PUB-SUB.
I found this example of reading from multiple sockets in the same process, but it seems sub optimal, because now you no longer block waiting for messages, you inefficiently check for messages constantly and go back to sleep.
Has anyone encountered this problem before? Am I just thinking about it wrong? Maybe I should have two threads - one for CM changes, one for REQ-REP servicing?
I would love any insights or examples of solving this type of problem.
Welcome to the very nature of distributed computing!
Yes, there are new perspectives one has to solve, once assembling a Project for a multi-agent domain, where more than one process works and communicates with it's respective peers ad-hoc.
A knowledge base, acquired from a soft Real-Time System or embedded systems design experience will help a lot here. If none such available, some similarities might be also chosen from GUI design, where a centerpiece is something like a lightweight .mainloop() scheduler, and most of the hard-work is embedded into round-robin polled GUI-devices and internal-state changes or external MMI-events are marshalled into event-triggered handlers.
ZeroMQ infrastructure gives one all the tools needed for such non-blocking, controllably poll-able ( scaleable, variable or adaptively ad-hoc adjustable poll-timeouts, not to overcome the given, design defined, round-trip duration of the controller .mainloop() ) and transport-agnostic, asynchronously operated, message dispatcher ( with thread-mapped performance scaling & priority tuning ).
What else one may need?
Well, just imagination and a lot of self-discipline to adhere the Zero-Copy, Zero-Sharing and Zero-Blocking design maxims.
The rest is in your hands.
Many "academic" examples may seem trivial and simplified, so as to illustrate just the currently discussed, or a feature demonstrated in some narrow perspective.
Not so in the real-life situations.
As an example, my distributed ML-engine uses a tandem of several PUSH/PULL pipelines for moving state data updates transfers and prediction forcasts + another PUSH/PULL for remote keyboard + a reversed .bind()/.connect() on PUB/SUB for easy broadcasting of distributed agents' telemetry to a remote centrally operated syslog and some additional PAIR/PAIR pipes, as processing requires.
( nota bene: one shall always bear in mind, that robust and error-resilient systems ought avoid to use a default REQ/REP Scaleable Formal Communication Pattern, as there is non-zero probability of falling the pairwise-stepped REQ/REP dual-FSA into an unsalvageable deadlock. Do not hesitate to read more about this smart tool. )
Pardon me if this is a basic / well known pattern, but I've been researching and can't find a solution. Maybe I'm searching for the wrong terms.
I have one producer akka actor, which pumps several small jobs involving about 50 KBytes each. And I am currently dispatching these jobs to consumer akka actors via a round robin router.
The root of the problem is the producer is much faster than the consumer actors.
I've been rolling my own solutions, adding in wait states and measuring average times of the processes to keep a steady rythm of jobs without blowing up the actor message queues. But this involves some synchronization between the actors and I don't think this is the akka way of doing it....it's not loose, it's a closed loop and has feedback, causing synchronization issues(determining how fast to send messages requires feedback from consumers). And if I don't throttle it, the queues blow up in 2 to 3 minutes.
I assume I'm doing this wrong.
So what's the appropriate pattern / method to use in the case when you have a producer that is faster than the consumer actors? Will I need a middleman, like a message queue or a database?
Thanks in advance for any pointers.
Akka Streams was created to solve exactly that problem (among other things). Take a look at it, it handles back-pressure between faster/slower stages of such processing pipelines.
It also can integrate well with Actors using Sinks/Sources explained here Akka Streams: Integrating with Actore.
Yes Akka Streams is the solution. Thank you Viktor Klang and Konrad 'ktoso' Malawski for your response.
This post might provide you quick help for development point of view.
https://blog.redelastic.com/diving-into-akka-streams-2770b3aeabb0#.z0yoceoox
I am trying to develop a p2p distributed concurrent programming solution in C++. For higher throughput I have observed that "to and fro" concurrent and scalable network solution is required as in actor model provided by AKKA. Unfortunately there is no reliable solution for actor model available in C++. libcppa is available but only in Beta quality. Is there any scalable and concurrent solution do same in zeroMQ or Boost.asio etc? Programatically I want to do following very fast and in scalable fashion.
Suppose I have a cluster of N nodes which contain separate set of objects. When a node require a remote object it sends a request and get backs a object as a response. Currently I do it in following manner:
Request req = Network.request(Node x, objectId id);
req.waitforResponse( ); //Implemented through Conc-Hashmap of requests and wait & signal
Object obj = req.response().getObject();
// Do sth with object.
I don't want to implement last two lines of codes myself. I want it to be done by networking library itself. Best solution will be if I can get "req" object in some sort of "Future" data structure which will block only on calling getObject(). I think it is a very generic problem and there should be some good open source solution available for it. If not please suggest some more efficient solution other than Concurrent hashmap (C++ Intel threads) based approach. Unfortunately my good amount of googling on it did not pay much.
As no one answered it. I am just updating solution which I used. I used MsgConnect with my thread pool to handle the events and callbacks.
[as a small context provider: I am new to networking and ZERO-MQ, but I did spend quite a bit of time on the guide and examples]
I have the following challenge (done in C++, but irrelevant to the question). I have a single source that generates tasks. I have multiple engines that need to process those tasks, and send back the result.
First attempt:
I created a client with a ZMQ_PUSH socket. The engines have a ZMQ_PULL socket. To get the answers back to the client, I created the reverse: a ZMQ_PUSH on the workers and a ZMQ_PULL on the client. It worked out of the box. Only to find out that after some time the client ran out of memory since I was pushing way more requests than the workers could process. I need some backpressure.
Second attempt:
I added a counter on the client that took care of only pushing when no more than say 1000 tasks were 'in progress'. The out of memory issue was solved, since I was never having more than 1000 'in progress' tasks. But ... some workers were slower than others. Since PUSH/PULL uses fair queueing, the amount of work for that slow worker kept increasing and increasing...until the slowest worker had all 1000 requests queued and the others were starved. I was not using my workers effectively.
Now, what architecture could I use that solves the issue of 'workers with different speed'? Is the 'count the number of in progress tasks' approach a good way of balancing the number of pushed requests? Or is there a way I can PUSH tasks to the workers, and the pushing blocks on a predefined point? Can I do that with HWM?
I am sure this problem is of such a generic nature that I should be able to easily deal with this. Can anyone point me in the right direction?
Thanks!
we used the Paranoid Pirate Protocol http://rfc.zeromq.org/spec:6,
but in case of many very small jobs, where the overhead of communication might be high, a credit-based flow control pattern might be more efficient. http://unprotocols.org/blog:15
in both cases it is necessary for the requester to directly assign jobs to individual workers. this is abstracted away of course and, depending on the use-case, could be made available as a sync call, which returns when all tasks have been processed.
I am coding a workload scheduler. I would like my piece of software to be a peer-to-peer scheduler, ie. a node only knows some neighbours (other nodes) and use them to reach other nodes.
Each node would have its own weighted-routing table to send messages to other peers (basically based on the number of hops), ie. "I want the master to give me my schedule" or "is resource A available on node B ?" : which neighbor is the closest to my target ?
For instance I have written my own routing protocol using XML-RPC (xmlrpc-c) and std::multimaps / std::maps.
I am thinking of using ZeroMQ to optimze my data streams :
queueing can reduce the network load between peers ;
subscriptions can be used to publish upgrades.
As a consequence :
I would need to open as many sockets as I would create new types of connections ;
Each node would need to be a client, a server, a publisher, a subscriber, a broker and a directory ;
I am not sure that my "peer-to-peer architecture" is compatible with the main purpose of ZeroMQ.
Do you think that ZeroMQ can be a helpful concept to use ?
It would be helpful to know exactly what you mean by "routing protocol".
That sounds like you mean the business logic of routing to a particular peer.
Knowing more fully what you're looking to achieve with ZeroMQ would also be helpful.
Have you read the ZeroMQ Guide?
ZeroMQ is a pretty different beast and without spending some time to play with it, you'll
likely find yourself confused. As a bonus, reading the guide will also help you answer
this question for yourself, since you know your requirements better.
ZeroMQ was designed to build robust distributed and multi-threaded applications. Since distributed applications can often take the form of "peer-to-peer", ZeroMQ could indeed be a good fit for your needs.