Akka - keeping reference to client - akka

Question

All,
I'm dealing with what seems like a simple case, but one that poses some design challenges:
There's a local actor system with a client, which reaches out to a
remote system that runs the bulk of the business logic.
The remote system will have a fixed IP address, port, etc. -
therefore, one can use the context.actorSelection(uri) strategy to
get a hold of the ActorRef for the current incarnation of the actor
(or a group of routees behind a router).
The remote system, being a server, shouldnt be in the business of
knowing the location of the client.
Given this, it's pretty straightforward to propagate messages from the client to the server, process them, and send a message back to the client. Even if there are several steps, one can propagate the responses through the hierarchy until one reaches the top-level remote actor that the client called, which will know who the sender was on the client side.
Let's say on the server side, we have a Master actor that has a router of Worker actors. You can have the worker respond directly to the client, since the message received from the Client by the Master can be sent to the Worker via the router as "router.tell(message, sender)" instead of "router ! message." Of course, you can also propagate responses from the Worker to the Master and then to the Client.
However, let's say the Worker throws an exception. If its parent (the Master) is its supervisor and it handles the Workers' failures, the Master can do the usual Restart/Resume/Stop. But let's say that we also want to notify the Client of the failure, e.g. for UI updating purposes. Even if we handle the failure of the Worker via the Master's SupervisorStrategy, we won't know who the original caller was (the Client) that had the processing request payload at the time when the Master intercepted the Worker's failure.
Here's a diagram
Client (local) -> Master (remote) -> Router (remote) -> Worker (remote)
Worker throws an exception, Master handles it. Now the Master can restart the Worker, but it doesn't know which Client to notify, in case there are several, their IP addresses change, etc.
If there's one Client and the Client has a host/port/etc. known to the server, then one could use context.actorSelection(uri) to look up the client and send it a message. However, with the server not being in the business of knowing where the Client is coming from (#3), this shouldn't be a requirement.
One obvious solution to this is to propagate messages from the Client to the Worker with the Client's ActorRef in the payload, in which case the Master would know about whom to send the failure notification to. It seems ugly, though. Is there a better way?
I suppose the Client can have the Workers on DeathWatch, but the Client shouldn't really have to know the details of the actor DAG on the server. So, I guess I'm coming back to the issue of whether the message sent from the Client should contain not just the originally intended payload, but also the ActorRef of the Client.
Also, this brings another point. Akka's "let it crash" philosophy suggests that the actor's supervisor should handle the actor's failures. However, if we have a Client, a Master (with a router) and a Worker, if the Worker fails, the Master can restart it - but it would have to tell the Client that something went wrong. In such a case, the Master would have to correlate the messages from the Client to the Workers to let the Client know about the failure. Another approach is to send the ActorRef of the Client along with the payload to the Worker, which would allow the Client to use the standard try/catch approach to intercept a failure, send a message to the client before failing, and then throw an exception that would be handled by the Master. However, this seems against Akka's general philosophy. Would Akka Persistence help in this case, since Processors track message IDs?
Thanks in advance for your help!
Best,
Marek

Answer 1

Quick answer, use this:
def preRestart(reason: Throwable, message: Option[Any]): Unit
More elaborate answer that gives no easy answers (as I struggle with this myself):
There are several ideas on how you can achieve what you need.
The question you asked should the worker answer the client or the master. Well that depends.
Let's assume that client sends you some work W1 and you pass it to the worker. The worker fails. Now the question is, if that work was important? If so, the master should still hold the reference to the W1 as it will probably retry the attempt in some near future. Maybe it was some data that should be persisted and the connection to database was lost for a second?
It the work was not important you may just set a timeout on the client that the operation was unsuccesfull and you're done. This way you will lost the exception details. But maybe it does not matter? You only want to check the logs afterwards and you just give a '500 Server Error' response.
This is not as easy to answer as it seems at first.

Answer 2

One possibility is to side-step most of this complexity by changing your approach. This may or may not be feasible for your use case.
For example, if the Exception can be anticipated and it is not of a sort that requires a restart of the worker actor, then don't let the master supervisor handle it. Simply build an appropriate response for that Exception (possible the Exception itself, or something containing the Exception), and send that to the client as a normal response message. You could send a scala Try message, for example, or create whatever messages make sense.
Of course, there are non-expected Exceptions, but in this case, the actor dealing with the UI can simply time-out and return a general error. Since the exception is unexpected, you probably wouldn't be able to do better than a general UI error anyway (e.g. a 500 error if the UI is HTTP-based), even if the exception was propagated to that layer. One downside of course is that the timeout will take longer to report the problem to the UI than if the error was propagated explicitly.
Lastly, I don't think there is anything wrong at all with sending ActorRef's as part of the payload, to handle this case from within the master actor as you suggested. I believe ActorRef was designed explicitly with the intent of sending them between actors (including remote actors). From the ScalaDoc of ActorRef:
Immutable and serializable handle to an actor, which may or may not reside on the local host or inside the same ActorSystem.
...
ActorRefs can be freely shared among actors by message passing.