The use case is this:
An Actor is bind to spray IO - receiving and handling all inbound HTTP requests coming through a specified port.
For each inbound request the actor needs to send an outbound asynchronous http request to a different external endpoint, get back an inbound response and send a response back to originating party.
Using spray's client sendReceive returns a future. This means the actor will continue to handle the next inbound message on it's mailbox without waiting for a response of the outbound request it just sent, in the same time the response for the outbound request might arrive and execute on the Future callback, since it is not queued on the actor's mailbox it might be executed in parallel breaking the idea of an actor being executed by only one thread in a given time.
I wonder how this use case can be handled without breaking the actor thread encapsulation, how can an actor make use of spray-client (for sending/receiving asynchronous http events) in an actor safe way?
It is perfectly safe to complete with the future, not the actual value in spray-routing, so for instance, you can do the following:
get {
comlete {
val resultFuture: Future[Result] = ...
val resultFuture.onComplete {....}
resultFuture
}
}
Of course, you will need to make sure that you handle timeouts and error conditions as well.
The question is which thread executes the callback, if it is not queued on the actor's mailbox it could be a parallel execution to the actor receive handling, which might break its thread encapsulation...
To my understanding, there is the same issue with akka actor 'ask' method which returns a Future, they provide a warning not to execute operations on the actor's mutable state from within the callback since it may cause synchronization problems. see: http://doc.akka.io/docs/akka/snapshot/scala/actors.html
"Warning:
When using future callbacks, such as onComplete, onSuccess, and onFailure, inside actors you need to carefully avoid closing over the containing actor’s reference, i.e. do not call methods or access mutable state on the enclosing actor from within the callback. This would break the actor encapsulation and may introduce synchronization bugs and race conditions because the callback will be scheduled concurrently to the enclosing actor. Unfortunately there is not yet a way to detect these illegal accesses at compile time."
Related
Imagine I have a JobsEndpoint class, which contains a JobSupervisor class which then has two child actors, RepositoryActor and StreamsSupervisorActor. The behavior for different requests to this top level JobSupervisor will need to be performed in the appropriate child actor. For example, a request to store a job will be handled exclusively in the RepositoryActor, etc...
My question, then, is if it is an anti-pattern to pass the request context through these actors via the messages, and then complete the request as soon as it makes sense?
So instead of doing this:
Request -> Endpoint ~ask~> JobSupervisor ~ask~> RepositoryActor
Response <- Endpoint <- JobSupervisor <-|return result
I could pass the RequestContext in my messages, such as StoreJob(..., ctx: RequestContext), and then complete it in the RepositoryActor.
I admittedly haven't been using Akka long but I see a few opportunities for improvement.
First, you are chaining "ask" calls which block threads. In some cases it's unavoidable but I think in your case, it is avoidable. When you block threads, you're potentially hurting your throughput.
I would have the Endpoint send a message with it's ActorRef as a "reply to" field. That way, you don't have to block the Endpoint and JobSupervisor actors. Whenever Repository actor completes the operation, it can send the reply directly to Endpoint without having to traverse middlemen.
Depending on your messaging guarantee needs, the Endpoint could implement retrying and de-duplicating if necessary.
Ideally each actor will have everything it needs to process a message in the message. I'm not sure what your RequestContext includes but I would consider:
1) Is it hard to create one? This impacts testability. If the RequestContext is difficult to create, I would opt for pulling out just the needed members so that I could write unit tests.
2) Can it be serialized? If you deploy your actor system in a cluster environment, then you'll need to serialize the messages. Messages that are simple data holders work best.
With ActorPublisher deprecated in favor of GraphStage, it looks as though I have to give up my actor-managed state for GraphStateLogic-managed state. But with the actor managed state I was able to mutate state by sending arbitrary messages to my actor and with GraphStateLogic I don't see how to do that.
So previously if I wanted to create a Source to expose data that is made available via HTTP request/response, then with ActorPublisher demand was communicated to my actor by Request messages to which I could react by kicking off an HTTP request in the background and send responses to my actor so I could send its contents downstream.
It is not obvious how to do this with a GraphStageLogic instance if I cannot send it arbitrary messages. Demand is communicated by OnPull() to which I can react by kicking off an HTTP request in the background. But then when the response comes in, how do I safely mutate the GraphStateLogic's state?
(aside: just in case it matters, I'm using Akka.Net, but I believe this applies to the whole Akka streams model. I assume the solution in Akka is also the solution in Akka.Net. I also assume that ActorPublisher will also be deprecated in Akka.Net eventually even though it is not at the moment.)
I believe that the question is referring to "asynchronous side-channels" and is discussed here:
http://doc.akka.io/docs/akka/2.5.3/scala/stream/stream-customize.html#using-asynchronous-side-channels.
Using asynchronous side-channels
In order to receive asynchronous events that are not arriving as stream elements (for example a completion of a future or a callback from a 3rd party API) one must acquire a AsyncCallback by calling getAsyncCallback() from the stage logic. The method getAsyncCallback takes as a parameter a callback that will be called once the asynchronous event fires.
I have an actor processing messages and storing its results via asynchronous API (ReactiveMongo). IE when computation is completed actor is asking ReactiveMongo to store computation result and that call is non blocking.
How can I stop actor processing next messages until last ReactiveMongo request feature will be completed? Also mailbox should be able to receive incoming messages.
Blocking solution
Simple and wrong answer: you can do this by blocking the actor, just call Await (or whatever similar method in the language do you use).
It is wrong because Do not block inside the actor.
Not blocking solution
Master\Worker pattern is a good for this problem: http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2
So your worker actor will send the "Work Done" message after ReactiveMongo request feature completion. Then master actor will send new "Do this work" message to the worker.
According to the Akka docs for PoisonPill:
You can also send an actor the akka.actor.PoisonPill message, which will stop the actor when the message is processed. PoisonPill is enqueued as ordinary messages and will be handled after messages that were already queued in the mailbox.
Although the usefulness/utility of such a feature may be obvious to an Akka Guru, to a newcomer, this sounds completely useless/reckless/dangerous.
So I ask: What's the point of this message and when would one ever use it, for any reason?!?
We use a pattern called disposable actors:
A new temporary actor is created for each application request.
This actor may create some other actors to do some work related to the request.
Processed result is sent back to client.
All temporary actors related to this request are killed. That's the place where PoisonPill is used.
Creating an actor implies a very low overhead (about 300 bytes of RAM), so it's quite a good practise.
I am using Akka.NET to implement an actor system in which some actors are created on demand and are deleted after a configurable idle period (I use Akka's "ReceiveTimeout" mechanism for this). Each of these actors is identified by a key, and there should not exist two actors with the same key.
These actors are currently created and deleted by a common supervisor. The supervisor can be asked to return a reference to the actor matching a given key, either by returning an existing one or creating a new one, if an actor with this key doesn't exist yet. When an actor receives the "ReceiveTimeout" message, it notifies the supervisor who in turn kills it with a "PoisonPill".
I have an issue when sending a message to one of these actors right after it has been deleted. I noticed that sending a message to a dead actor doesn't generate an exception. Worse, when sending an "Ask" message, the sender remains blocked, waiting indefinitely (or until a timeout) for a response that he will never receive.
I first thought about Akka's "Deatchwatch" mechanism to monitor an actor's lifecycle. But, if I'm not mistaken, the "Terminated" message sent by the dying actor will be received by the monitoring actor asynchronously just like any other message, so the problem may still occur in between the target actor's death and the reception of its "Terminated" message.
To solve this problem, I made it so that anyone asking the supervisor for a reference to such an actor has to send a "close session" message to the supervisor to release the actor when he doesn't need it anymore (this is done transparently by a disposable "ActorSession" object). As long as there are any open sessions on an actor, the supervisor will not delete it.
I suppose that this situation is quite common and am therefore wondering if there isn't a simpler pattern to follow to address this kind of problem. Any suggestion would be appreciated.
I have an issue when sending a message to one of these actors right after it has been deleted. I noticed that sending a message to a dead actor doesn't generate an exception.
This is by design. You will never receive an exception upon attempting to send a message - it will simply be routed to Deadletters and logged. There's a lot of reasons for this that I won't get into here, but the bottom line is that this is intended behavior.
DeathWatch is the right tool for this job, but as you point out - you might receive a Terminated message after you already sent a message to that actor.
A simpler pattern than tracking open / closed sessions is to simply use acknowledgement / reply messages from the recipient using Ask + Wait + a hard timeout. The downside of course is that if your recipient actor has a lot of long-running operations then you might block for a long period of time inside the sender.
The other option you can go with is to redesign your recipient actor to act as a state machine and have a soft-terminated or terminating state that it uses to drain connections / references with potential senders. That way the original actor can still reply and accept messages, but let callers know that it's no longer available to do work.
I solved this problem with entity actors created through Akka's Cluster Sharding mechanism:
If the state of the entities are persistent you may stop entities that are not used to reduce memory consumption. This is done by the application specific implementation of the entity actors for example by defining receive timeout (context.setReceiveTimeout). If a message is already enqueued to the entity when it stops itself the enqueued message in the mailbox will be dropped. To support graceful passivation without losing such messages the entity actor can send ShardRegion.Passivate to its parent Shard. The specified wrapped message in Passivate will be sent back to the entity, which is then supposed to stop itself. Incoming messages will be buffered by the Shard between reception of Passivate and termination of the entity. Such buffered messages are thereafter delivered to a new incarnation of the entity.