I have a problem, where the second invocation of my program gives unpredictable behaviour.
I am trying to ensure a graceful shutdown of an akka-remoting application.
I have two actor systems, a local actor system and a remote actor system.
The local actor system
creates two actors, one on the local system and one on the remote
shuts himself down
I first start the remote actor system and then run the local actor system.
When i run this the first time, everything is okay, and the local actor system shuts down (the remote is still up). But if i run this a second time (without restarting the remote system), the behaviour is different and the two actor systems start heart beating. The local does not shut down.
The minimal code to replicate the behaviour is below
Local Actor System
object ActorAsSink_7 extends App {
val system = ActorSystem("system")
val localActor = system.actorOf(MyActor_4.props)
val remoteActor = system.actorOf(MyActor_4.props, "remote_agent")
localActor ! PoisonPill
}
Remote Actor System
object RemoteActorSystem extends App {
import system.dispatcher
implicit val system = ActorSystem("remote-actorsystem")
// actors in this actor system are created remotely
println("hello.. remote agent is up")
}
This is part of a much larger code base, and this is the minimal i was able to find to replicate the issue.
Why is my local actor system behaving differently and not shutting down in the second invocation?
The local actor does the following
object MyActor_4 {
val props = Props[MyActor_4]
}
class MyActor_4 extends Actor with ActorLogging {
def receive = { case x: Any => log.error("unexpected message in reaper: " + x) }
override def postStop = { println ("shutting down...") ; context.system.terminate(); }
}
Remoting is working fine, but for this shutdown issue.
As per my understanding below code line would not stop remoteActor.
localActor ! PoisonPill
Because PoisonPill stops the sender actor and all its child actors.
Here remoteActor is not a child actor of localActor.
could you share the logs where both actors of local actor system stop in first run ?
Related
What does Akka's Inbox.create(actorsystem) actually do? Will it create a new mailbox for all the actors as we have created the inbox on the system? And what does inbox.watch(actorref) do?
Can you explain what inbox.send and inbox.receive actually do?
From the source code for Inbox:
/**
* An Inbox is an actor-like object which is interrogated from the outside.
* It contains an actor whose reference can be passed to other actors as
* usual and it can watch other actors’ lifecycle.
*/
The Inbox object is a way to communicate with an actor from outside an actor. Let's say we wanted to send messages to and receive replies from an actor named myActor, and we wanted to do this from outside an actor. (The below code samples are adapted from the documentation.)
// this code is not inside an actor
ActorRef myActor = ???
final Inbox inbox = Inbox.create(system);
inbox.send(myActor, "ping");
Inbox.create(system) creates a system-level actor under the covers. inbox.send(myActor, "ping"); sends the message "ping" to myActor with Inbox's internal actor as the sender. Because the Inbox's actor is the sender, it can get a reply from myActor with inbox.receive:
try {
assert inbox.receive(Duration.create(1, TimeUnit.SECONDS)).equals("pong");
} catch (java.util.concurrent.TimeoutException e) {
// timeout
}
inbox.watch registers the Inbox's actor to be notified via DeathWatch when myActor has been terminated:
inbox.watch(myActor);
myActor.tell(PoisonPill.getInstance(), ActorRef.noSender());
try {
assert inbox.receive(Duration.create(1, TimeUnit.SECONDS)) instanceof Terminated;
} catch (java.util.concurrent.TimeoutException e) {
// timeout
}
Inbox.create(system) does not create a new mailbox for all the actors in a system. Don't let the word "inbox" confuse you. Also, Akka enables by default a dispatcher that creates one mailbox per actor; the Inbox object doesn't change that.
I have an actor which is orchestrating database updates. I need to ensure that each operation gets executed only after the previous one gets completed.
This because operation B will reuse the result of operation A.
Here the code I wrote for the actor.
class DbUpdateActor(databaseOperations: DBProvider) extends Actor {
implicit val ec:ExecutionContext = context.system.dispatcher
def receive: Receive = {
case newInfo : UpdateDb =>
val future = Future {
// gets the current situation from DB
val status = databaseOperations.getSituation()
// do db update
databaseOperations.save(something)
}
future onComplete {
case Success(result: List[Int]) =>
//
case Failure(err: Throwable) =>
//
}
}
}
The code works fine for a single operation. If I fire two updates then the second one is executed asynchronously so it gets started before the first one has completed.
I was reading about different types of mailbox, not sure if having a different one would help.
Any suggestion?
One option you can explore would be to remove that Future and allow that blocking db code to be run within the actor. Then, use a separate dispatcher (perhaps a PinnedDispatcher) to fire-wall this blocking code off from the main actor system's dispatcher, giving it its own thread to run on. By blocking in the body and removing that Future, you will ensure proper sequential execution of the actor's mailbox. A rough sketch of the changes to make that work are as follows:
object DbUpdateActor{
def props(databaseOperations:DBProvider) =
Props(classOf[DbUpdateActor], databaseOperations).
withDispatcher("db-update-dispatcher")
}
class DbUpdateActor(databaseOperations: DBProvider) extends Actor {
def receive: Receive = {
case newInfo : UpdateDb =>
val status = databaseOperations.getSituation()
databaseOperations.save(something)
}
}
Then, as long as you had the following dispatcher configured in your actor system config:
db-update-dispatcher {
executor = "thread-pool-executor"
type = PinnedDispatcher
}
And you started up the db update actor like so:
val updater = system.actorOf(DbUpdateActor.props(databaseOperations))
Then you should be all set setting this actor up to run that blocking code in a way that won't negatively affect the throughput of the main dispatcher.
How about this: start operation A in a child; when the child is complete it sends the parent a message saying it completed. Then you can start operation B, either in the existing or a new child.
From: https://www.chrisstucchio.com/blog/2013/actors_vs_futures.html
suggests that this is safe:
class FooCounter extends Actor {
var count: Long = 0
def receive = {
case Foo => { count += 1}
case FooCountRequest => { sender ! count }
}
}
Isn't it possible that there will be multiple simultaneous calls to receive, making the value of count uncertain.
My understanding is that the only way this could ever be safe would be if the receive call on this object was made mutex with itself.
The receive method is never called simultaneously by multiple threads. Messages residing in an Actor's mailbox are processed one-at-a-time by the receive method. Multiple other Actors, or functions outside of the ActorSystem, can concurrently enqueue messages to the Actor's mailbox but the ActorSystem eventually orders the messages. From the docs:
Enqueuing happens in the time-order of send operations, which means
that messages sent from different actors may not have a defined order
at runtime due to the apparent randomness of distributing actors
across threads.
The receive method's serial processing is guaranteed by the fact that you never actually get an Actor value (which has a receive) from the ActorSystem. Rather, you only get an ActorRef which does not have receive method:
val actorSystem = akka.actor.ActorSystem()
//not an Actor but an ActorRef
val actorRef : ActorRef = actorSystem actorOf Props[FooCounter]
actorRef.receive(Foo) //COMPILE TIME ERROR!
The only way to "invoke" the receive method is to send a message to the ActorRef:
actorRef ! Foo //non-blocking, enqueues a Foo object in the mailbox
Relating back to your question: the ActorSystem acts as a pseudo-mutex for all Actor instances.
Therefore, the code in your example is absolutely safe and the state will only be accessed by one message at any given time.
Totally agree with Ramon. You can think of it have a mail box outside your house(Actor) and mail are coming to your mail box though your address (ActorRef) and you only have 1 single person at your home to take care your mail one at a time.
Besides, for more functional style and maintain immutability of code. I would do the following instead:
class FooCounter extends Actor {
def _receive(count: Long): Receive = {
case Foo =>
context.become(_receive(count + 1))
case FooCountRequest =>
sender() ! count
}
def receive = _receive(0L)
}
For this simple example, there are no difference between mine and yours. But when system become more complex, my code are less error prone.
I have a Scala unit test for an Akka actor. The actor is designed to poll a remote system and update a local cache. Part of the actor's design is that it doesn't attempt to poll while it's still processing or awaiting the result of the last poll, to avoid flooding the remote system when it experiences a slowdown.
I have a test case (shown below) which uses Mockito to simulate a slow network call, and checks that when the actor is told to update, it won't make another network call until the current one is complete. It checks the actor has not made another call by verifying a lack of interactions with the remote service.
I want to eliminate the call to Thread.sleep. I want to test the functionality of the actor without relying on waiting for a hardcoded time, in every test run, which is brittle, and wastes time. The test can poll or block, waiting for a condition, with a timeout. This will be more robust, and will not waste time when the test is passing. I also have the added constraint that I want to keep the state used to prevent extra polling var allowPoll limited in scope, to the internals of the PollingActor.
is there a way force a wait until the actor is finished messaging itself? If there's a way I can wait until then before trying to assert.
is it necessary to send the internal message at all? Couldn't I maintain the internal state with a threadsafe datastructure, such as java.util.concurrent.AtomicBoolean. I have done this and the code appears to work, but I'm not knowledgeable enough about Akka to know if it's discouraged -- a colleague recommended the self message style.
is there better, out-of-the-box functionality with the same semantics? Then I would opt for an integration test instead of a unit test, though I'm not sure if it would solve this problem.
The current actor looks something like this:
class PollingActor(val remoteService: RemoteServiceThingy) extends ActWhenActiveActor {
private var allowPoll: Boolean = true
def receive = {
case PreventFurtherPolling => {
allowPoll = false
}
case AllowFurtherPolling => {
allowPoll = true
}
case UpdateLocalCache => {
if (allowPoll) {
self ! PreventFurtherPolling
remoteService.makeNetworkCall.onComplete {
result => {
self ! AllowFurtherPolling
// process result
}
}
}
}
}
}
trait RemoteServiceThingy {
def makeNetworkCall: Future[String]
}
private case object PreventFurtherPolling
private case object AllowFurtherPolling
case object UpdateLocalCache
And the unit test, in specs2, looks like this:
"when request has finished a new requests can be made" ! {
val remoteService = mock[RemoteServiceThingy]
val actor = TestActorRef(new PollingActor(remoteService))
val slowRequest = new DefaultPromise[String]()
remoteService.makeNetworkCall returns slowRequest
actor.receive(UpdateLocalCache)
actor.receive(UpdateLocalCache)
slowRequest.complete(Left(new Exception))
// Although the test calls the actor synchronously, the actor calls *itself* asynchronously, so we must wait.
Thread.sleep(1000)
actor.receive(UpdateLocalCache)
there was two(remoteService).makeNetworkCall
}
The way we have chosen to solve this for now is to inject the equivalent of an observer into the actor (piggybacking on an existing logger which wasn't included in the listing in the question). The actor can then tell the observer when it has transitioned from various states. In the test code we perform an action then wait for the relevant notification from the actor, before continuing and making assertions.
In the test we have something like this:
actor.receive(UpdateLocalCache)
observer.doActionThenWaitForEvent(
{ actor.receive(UpdateLocalCache) }, // run this action
"IgnoredUpdateLocalCache" // then wait for the actor to emit an event
}
// assert on number of calls to remote service
I don't know if there's a more idiomatic way, this seems like a reasonable suggestion to me.
Imagine a straight-forward supervision hierarchy. The child dies. The father decides to Restart the child. When Restarted, the postRestart and friends are called, but what if the father had decided to resume the child? Does the child actor know that he is being resumed? And btw. does the father have access to the message that caused the exception in his child?
Resume means “nothing really happened, carry on” and in this spirit the child is not even informed. It is a directive which should rarely be used.
The parent does only get the failure itself (i.e. the Throwable), not the message which caused the problem, because that would invite you to entangle the logic of parent and child beyond what is healthy.
The term resume means continue processing messages and is referred to at two points in the documentation.
The first is used in response to an exception state:
As per akka documentation:
As described in Actor Systems supervision describes a dependency relationship between actors: the supervisor delegates tasks to subordinates and therefore must respond to their failures. When a subordinate detects a failure (i.e. throws an exception), it suspends itself and all its subordinates and sends a message to its supervisor, signaling failureDepending on the nature of the work to be supervised and the nature of the failure, the supervisor has a choice of the following four options:
Resume the subordinate, keeping its accumulated internal state
Restart the subordinate, clearing out its accumulated internal state
Terminate the subordinate permanently
Escalate the failure, thereby failing itself
Note that RESTART actually KILLS the original actor. The term resume is used again here meaning to continue processing messages.
As per the akka documentation.
The precise sequence of events during a restart is the following:
- suspend the actor (which means that it will not process normal messages until resumed), and recursively suspend all children
- call the old instance’s preRestart hook (defaults to sending termination requests to all children and calling postStop)
- wait for all children which were requested to terminate (using context.stop()) during preRestart to actually terminate; this—like all actor operations—is non-blocking, the termination notice from the last killed child will effect the progression to the next step
- create new actor instance by invoking the originally provided factory again
- invoke postRestart on the new instance (which by default also calls preStart)
- send restart request to all children which were not killed in step 3; restarted children will follow the same process recursively, from step 2
- resume the actor
You can have the failure bubble up to the Supervisor if you properly set up that kind of behavior in the supervisorStrategy of the supervisor. A little example to show that behavior:
import akka.actor.Actor
import akka.actor.Props
import akka.actor.ActorSystem
object SupervisorTest {
def main(args: Array[String]) {
val system = ActorSystem("test")
val master = system.actorOf(Props[Master], "master")
master ! "foo"
Thread.sleep(500)
val worker = system.actorFor("/user/master/foo")
worker ! "bar"
}
}
class Master extends Actor{
import akka.actor.OneForOneStrategy
import akka.actor.SupervisorStrategy._
import scala.concurrent.duration._
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1 minute) {
case _: Exception => Escalate
Escalate
}
override def preRestart(ex:Throwable, msg:Option[Any]) = {
println("In master restart: " + msg)
}
def receive = {
case msg:String =>
context.actorOf(Props[Worker], msg)
}
}
class Worker extends Actor{
override def preRestart(ex:Throwable, msg:Option[Any]) = {
println("In worker restart: " + msg)
}
def receive = {
case _ =>
throw new Exception("error!!")
}
}
You can see in the Master actor (the supervisor in my example), I am choosing to Escalate a failure of type Exception. This will cause the failure to bubble up to the preRestart in the Master actor. Now I was expecting the msg param to preRestart to be the original offending message that went to the worker actor, but it wasn't. The only way I got that to show was be also overriding the preRestart of the child actor. In my example, you will see the print outs from both the supervisor and child, in that order.