Akka: How to reconnect to restarted slave? - akka

Question

I have two docker containers running localy, one is master, the second is slave, communicating over akka remote. Slave can go OOM from time to time for certain messages, in which case docker gracefully restarts it..
The code looks a little bit like this:
object Master {
def main() {
...
val slave =
typedActorOf(TypedProps[Slave], resolveRemoteAtor(..))
val dispatcher =
typedActorOf(TypedProps(classOf[Dispatcher], new DispatcherImpl(slave)))
val httpServer =
typedActorOf(TypedProps(classOf[HTTPServer], new HTTPServerImpl(dispatcher)))
}
}
class Slave() { def compute() = ... }
class Dispatcher(s: Slave) { def compute() = s.compute() }
The problem is, that the master shutdowns the connection with the slave, once it becomes unavailable due to OOM, and it never renews it:
[ERROR] from a.r.EndpointWriter - AssociationError akka.tcp://MasterSystem#localhost:0] -> [akka.tcp://SlaveSystem#localhost:1]: Error [Shut down address: akka.tcp://SlaveSystem#localhost:1] [akka.remote.ShutDownAssociation: Shut down address: akka.tcp://SlaveSystem#localhost:1 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ]
[INFO] from a.r.RemoteActorRef - Message [akka.actor.TypedActor$MethodCall] from Actor[akka://MasterSystem/temp/$c] to Actor[akka.tcp://SlaveSystem#localhost:1/user/Slave#1817887555] was not delivered. [1] dead letters encountered.
So my question is, how can I force the master to reconnect with the slave once the slave restarts and send all the pending messages, that were not possible to deliver during the time it was down?

Answer 1

I'd recommend using Akka Cluster over remoting directly, for this and in general as well, cluster will allow you to listen for membership events so that you can react on a node leaving and reappearing.
Making guarantees around delivery of messages requires some extra thought though. This section of the docs is good to read for better understanding the issues around it.