Akka configuration to distribute Map workload across different network nodes? - mapreduce

Question

I prepared a working Akka/Java implementation of MapReduce akka_mapreduce_example that I currently use in multiple projects.
I now would like to distribute the Map step workload across many separate network nodes and I'm wondering what I have to change in the Akka configuration to achieve exactly that. I would need configuration changes in the "Master" machine (the one triggering the MapReduce that also runs the reduce step) and the "Slave" machines that help distribute the Map workload.
My current application.conf works for a single machine and is quite simple. I'm hoping that this requires only configuration changes and not code changes, or?

Answer 1

Depending on your code, all that is required are configuration changes. You'll need something like this in your config to set remoting up:
akka {
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = "127.0.0.1"
port = 2552
}
}
}
Then you need to tell Akka which of your actors will be created on the remote node:
akka {
actor {
deployment {
/sampleActor {
remote = "akka.tcp://sampleActorSystem#127.0.0.1:2553"
}
}
}
}
For more info you can look up the excellent documentation. I linked the docs for the 2.2.1 version, if you use a different version, make sure to look in the docs for that version. There tend to be changes between the version, and while things from an older version will still work in a newer version (they might be deprecated though), the other way round is obviously a problem.
You'll notice I said "Depending on your code". What I mean by that, is that all your messages that will be send to the remote node need to be Serializable, and that you do not use any static members in your actors.
If you want more flexibility you can check out the brand new Clustering support. This will give you a nice flexible, dynamic Peer-To-Peer system which you can scale up and down as you want. You might need some slight code changes for that.
Hope that helps.

Answer 2

The workload can be distributed across multiple nodes but you have to choose on two modes of processing i.e. either pull or push based processing.
Both have pros and cons. But pull is more attractive as it provides fault tolerance and you can track assignment of work to worker actor's. To get started look at http://blog.goconspire.com/post/64901258135/akka-at-conspire-part-5-the-importance-of-pulling.
The sample working code is at https://github.com/typesafehub/activator-akka-distributed-workers.
Handling fast producer and slow consumer problem akka streaming's back pressure can be used.
For dynamic creation of work actor's in case of extreme load on system's, you have to come up with your own design. But the solution says to add more nodes to the cluster to distribute work to new nodes.