I have small mess in my head
Avoid having poisoned mailbox
http://doc.akka.io/docs/akka/2.4.2/general/supervision.html
The new actor then resumes processing its mailbox, meaning that the
restart is not visible outside of the actor itself with the notable
exception that the message during which the failure occurred is not
re-processed.
My case: actor receives the "command" to run smth. Actor tries to reach remote service. Service is unavailable. Exception is thrown. I want actor to keep going contact remote server. I don't wan't actor to skip input command which caused exception. Will Resume help me to force actor to keep going?
override val supervisorStrategy: SupervisorStrategy =
OneForOneStrategy(maxNrOfRetries = -1, withinTimeRange = 5.minutes) {
case _: RemoteServiceIsDownException => Resume
case _ => Stop
}
By Resume, I mean retry the invocation that caused the exception to be thrown. I suspect that akka Resume means keep actor instance, but not retry failed invocation
Does akka persistence means durable mailboxes?
Extending first case. Actor tries to reach remote service. Now actor is persistent. SupervisorStrategy forces Actor to continue to contact remote service. The whole JVM shuts down. Akka app is restarted. Will Actor Resume from the point where it tired desperately reach remote service?
Does akka persistence means At least once semantics?
Actor receives message. Then JVM crashes. Will parent re-receive message it was processing during crush?
Expanding my comment:
Will Resume help me to force actor to keep going? ... By Resume, I mean retry the invocation that caused the exception to be thrown. I suspect that akka Resume means keep actor instance, but not retry failed invocation
No, I do not believe so. The Resume directive will keep the actor chugging along after your message processing failure. One way to retry the message though, is to actually just use Restart and to take advantage of an Actor's preRestart hook:
override def preRestart(t: Throwable, msgBeforeFailure: Option[Any]): Unit = {
case t: RemoteServiceIsDownException if msgBeforeFailure.isDefined =>
self ! msgBeforeFailure.get
case _: =>
}
When the actor crashes, it will run this hook and offer you an opportunity to handle the message that caused it to fail.
Does akka persistence means durable mailboxes?
Not necessarily, using a persistent actor just means that the actor's domain events, and subsequently, internal state is durable, not the mailbox it uses to process messages. That being said, it is possible to implement a durable mailbox pretty easily, see below.
Does akka persistence means At least once semantics?
Again not necessarily, but the toolkit does have a trait called AtleastOnceDelivery to let you achieve this (and durable mailbox)!
See http://doc.akka.io/docs/akka/current/scala/persistence.html#At-Least-Once_Delivery
Related
We have been using Akka BackoffSupervisor for a few cluster singleton actors that are critical for the system and should always be present. These actors are persistent so in case DB failure they need to be restarted. Our code used OnFailure condition. However when the child persistence failed due the database error both the child and backoff supervisor were stopped and not restarted.
Should we use OnStop instead of OnFailure? It's not explained when each of the should be used and since the child actor fails in case of the database error our assumption was that we could use OnFailure. But perhaps since an actor is unconditionally terminated on persitence error OnFailure is never invoked and OnStopp should be used instead?
And the answer is "yes": in case the supervised actor is stopped unconditionally, it triggers the event OnStop that can be monitored to start a new incarnation of it.
Im trying to implement Saga (Process Manager) with PersistentActor which updates multiple entities / aggregate roots to achieve (eventual) consistency without ACID transaction.
Let's say, in the middle of the process, between Process Manager received message and started processing it, jvm crashed.
Or application is being redeployed at this very moment.
Will Process Manager's mailbox be recovered after jvm (along with actor system) restart?
The mailboxes are not recovered in case of JVM crash. It is the sender's responsability to ensure that its message was received/persisted. Between persistent actors, use At-Least-Once Delivery.
Given that you're already using PersistentActor, it's reasonable to leverage its AtLeastOnceDelivery functionality for what you need. AtLeastOnceDelivery essentially ensures that the persistent sender actor will keep sending messages to the receiver at a configurable frequency until it receives confirmation from the receiver (or from some termination routine for canceling ongoing delivery attempts).
As with general PersistentActor (with or without AtLeastOnceDelivery), messages that are persisted will be re-sent should the JVM crash. In the case of AtLeastOnceDelivery, persisted messages will be re-sent upon system recovery until receipt of confirmation. Akka provides method deliver to send each message tagged with a consecutive monotonically increasing deliveryId and, upon receiving a confirmation with the corresponding deliveryId from the receiver, uses method confirmDelivery to signal successful delivery of the message.
The following snippet, which is part of the sample code from relevant Akka doc, highlights the key AtLeastOnceDelivery logic in the sender actor class (which extends PersistentActor with AtLeastOnceDelivery). Note that the same persistence handler updateState is used to persist events corresponding to deliver and confirmDelivery for consistent state recovery:
override def receiveCommand: Receive = {
case s: String =>
persist(MsgSent(s))(updateState)
case Confirm(deliveryId) =>
persist(MsgConfirmed(deliveryId))(updateState)
}
override def receiveRecover: Receive = {
case evt: Evt => updateState(evt)
}
def updateState(evt: Evt): Unit = evt match {
case MsgSent(s) =>
deliver(destination)(deliveryId => Msg(deliveryId, s))
case MsgConfirmed(deliveryId) =>
confirmDelivery(deliveryId)
}
I would like to ask for pattern when it is required to init actor state from database. I have a DAO which return Future[...] and normaly to be non blocking message should be send to actor on future complete. But this case is different. I cannot receive any message from actor mailbox before initialization complete. Is the only way to block actor thread while waiting for database future complete?
The most trivial approach would be to define two receive methods, e.g. initializing and initialized, starting with def receive = initializing. Inside your inizializing context, you could simply send back a message like InitializationNotReady to tell the other actor that he should try it again later. After your actor is initialized, you switch your context with context become initialized to the new state, where you can operate normally.
After all, another good aproach could be to have a look at Akka Persistence. It enables stateful actors to persist their internal state so that it can be recovered when an actor is started, restarted after a JVM crash or by a supervisor, or migrated in a cluster.
In you case, you can restore your state from a database, since their are multiple storage options for Akka persistence. you can find them here. After that recovery, you can receive messages like you're used to it.
The use case is this:
An Actor is bind to spray IO - receiving and handling all inbound HTTP requests coming through a specified port.
For each inbound request the actor needs to send an outbound asynchronous http request to a different external endpoint, get back an inbound response and send a response back to originating party.
Using spray's client sendReceive returns a future. This means the actor will continue to handle the next inbound message on it's mailbox without waiting for a response of the outbound request it just sent, in the same time the response for the outbound request might arrive and execute on the Future callback, since it is not queued on the actor's mailbox it might be executed in parallel breaking the idea of an actor being executed by only one thread in a given time.
I wonder how this use case can be handled without breaking the actor thread encapsulation, how can an actor make use of spray-client (for sending/receiving asynchronous http events) in an actor safe way?
It is perfectly safe to complete with the future, not the actual value in spray-routing, so for instance, you can do the following:
get {
comlete {
val resultFuture: Future[Result] = ...
val resultFuture.onComplete {....}
resultFuture
}
}
Of course, you will need to make sure that you handle timeouts and error conditions as well.
The question is which thread executes the callback, if it is not queued on the actor's mailbox it could be a parallel execution to the actor receive handling, which might break its thread encapsulation...
To my understanding, there is the same issue with akka actor 'ask' method which returns a Future, they provide a warning not to execute operations on the actor's mutable state from within the callback since it may cause synchronization problems. see: http://doc.akka.io/docs/akka/snapshot/scala/actors.html
"Warning:
When using future callbacks, such as onComplete, onSuccess, and onFailure, inside actors you need to carefully avoid closing over the containing actor’s reference, i.e. do not call methods or access mutable state on the enclosing actor from within the callback. This would break the actor encapsulation and may introduce synchronization bugs and race conditions because the callback will be scheduled concurrently to the enclosing actor. Unfortunately there is not yet a way to detect these illegal accesses at compile time."
I am using Akka.NET to implement an actor system in which some actors are created on demand and are deleted after a configurable idle period (I use Akka's "ReceiveTimeout" mechanism for this). Each of these actors is identified by a key, and there should not exist two actors with the same key.
These actors are currently created and deleted by a common supervisor. The supervisor can be asked to return a reference to the actor matching a given key, either by returning an existing one or creating a new one, if an actor with this key doesn't exist yet. When an actor receives the "ReceiveTimeout" message, it notifies the supervisor who in turn kills it with a "PoisonPill".
I have an issue when sending a message to one of these actors right after it has been deleted. I noticed that sending a message to a dead actor doesn't generate an exception. Worse, when sending an "Ask" message, the sender remains blocked, waiting indefinitely (or until a timeout) for a response that he will never receive.
I first thought about Akka's "Deatchwatch" mechanism to monitor an actor's lifecycle. But, if I'm not mistaken, the "Terminated" message sent by the dying actor will be received by the monitoring actor asynchronously just like any other message, so the problem may still occur in between the target actor's death and the reception of its "Terminated" message.
To solve this problem, I made it so that anyone asking the supervisor for a reference to such an actor has to send a "close session" message to the supervisor to release the actor when he doesn't need it anymore (this is done transparently by a disposable "ActorSession" object). As long as there are any open sessions on an actor, the supervisor will not delete it.
I suppose that this situation is quite common and am therefore wondering if there isn't a simpler pattern to follow to address this kind of problem. Any suggestion would be appreciated.
I have an issue when sending a message to one of these actors right after it has been deleted. I noticed that sending a message to a dead actor doesn't generate an exception.
This is by design. You will never receive an exception upon attempting to send a message - it will simply be routed to Deadletters and logged. There's a lot of reasons for this that I won't get into here, but the bottom line is that this is intended behavior.
DeathWatch is the right tool for this job, but as you point out - you might receive a Terminated message after you already sent a message to that actor.
A simpler pattern than tracking open / closed sessions is to simply use acknowledgement / reply messages from the recipient using Ask + Wait + a hard timeout. The downside of course is that if your recipient actor has a lot of long-running operations then you might block for a long period of time inside the sender.
The other option you can go with is to redesign your recipient actor to act as a state machine and have a soft-terminated or terminating state that it uses to drain connections / references with potential senders. That way the original actor can still reply and accept messages, but let callers know that it's no longer available to do work.
I solved this problem with entity actors created through Akka's Cluster Sharding mechanism:
If the state of the entities are persistent you may stop entities that are not used to reduce memory consumption. This is done by the application specific implementation of the entity actors for example by defining receive timeout (context.setReceiveTimeout). If a message is already enqueued to the entity when it stops itself the enqueued message in the mailbox will be dropped. To support graceful passivation without losing such messages the entity actor can send ShardRegion.Passivate to its parent Shard. The specified wrapped message in Passivate will be sent back to the entity, which is then supposed to stop itself. Incoming messages will be buffered by the Shard between reception of Passivate and termination of the entity. Such buffered messages are thereafter delivered to a new incarnation of the entity.