Terminating a flailing akka actor - akka

I have the following actor setup, using Akka actors (2.10)
A -spawn-> B -spawn-> C
A -sendWork-> B -sendWork-> C
C -sendResults-> A (repeatedly)
However, at some point A notices that it should change the workload sent to B/C because C is sending a large number of messages that turn out to be useless. However, in such situations C's inbox seems to be very full, and/or C may be blocked.
How can A tell B to shutdown C immediately? Losing the state and messages of both B and C is acceptable, so destroying them and spawning new ones is an option.

Given the actors are started the way you described, then using stop in the right way will do what you require. According to the docs, calling stop will both:
1) stop additional messages from going into the mailbox (sent to deadletter)
2) take the current contents of the mailbox and also ship that to deadletter (although this is based on mailbox impl, but the point is they won't be processed)
Now if the actor will need to completely finish the message it's currently processing before it's all the way stopped, so if it's "stuck", stopping (or anything for that matter) won't fix that, but I don't think that's the situation you are describing.
I pulled a little code sample together to demonstrate. Basically, A will send a message to B to start sending work to C. B will flood C with some work and C will send the results of that work back to A. When a certain number of responses have been received by A, it will trigger a stop of B and C by stopping B. When B is completely stopped, it will then restart the process over again, up to 2 total times because it stops itself. The code looks like this:
case object StartWork
case class DoWork(i:Int, a:ActorRef)
case class WorkResults(i:Int)
class ActorA extends Actor{
import context._
var responseCount = 0
var restarts = 0
def receive = startingWork
def startingWork:Receive = {
case sw # StartWork =>
val myb = actorOf(Props[ActorB])
myb ! sw
become(waitingForResponses(myb))
}
def waitingForResponses(myb:ActorRef):Receive = {
case WorkResults(i) =>
println(s"Got back work results: $i")
responseCount += 1
if (responseCount > 200){
println("Got too many responses, terminating children and starting again")
watch(myb)
stop(myb)
become(waitingForDeath)
}
}
def waitingForDeath:Receive = {
case Terminated(ref) =>
restarts += 1
if (restarts <= 2){
println("children terminated, starting work again")
responseCount = 0
become(startingWork)
self ! StartWork
}
else{
println("too many restarts, stopping self")
context.stop(self)
}
}
}
class ActorB extends Actor{
import concurrent.duration._
import context._
var sched:Option[Cancellable] = None
override def postStop = {
println("stopping b")
sched foreach (_.cancel)
}
def receive = starting
def starting:Receive = {
case sw # StartWork =>
val myc = context.actorOf(Props[ActorC])
sched = Some(context.system.scheduler.schedule(1 second, 1 second, self, "tick"))
become(sendingWork(myc, sender))
}
def sendingWork(myc:ActorRef, a:ActorRef):Receive = {
case "tick" =>
for(j <- 1 until 1000) myc ! DoWork(j, a)
}
}
class ActorC extends Actor{
override def postStop = {
println("stopping c")
}
def receive = {
case DoWork(i, a) =>
a ! WorkResults(i)
}
}
It's a little rough around the edges, but it should show the point that cascading the stop from B through to C will stop C from sending responses back to A even though it still had messages in the mailbox. I hope this is what you were looking for.

Related

Akka actors output not as expected

Following akka documentation, for example in section 'The actor lifecycle', for me output is not as documented. What I get is:
first started
second started
second stopped
Code is:
object StartStopActorMain extends App {
val first = ActorSystem(StartStopActor1(), "firstActor")
first ! "stop"
}
object StartStopActor1 {
def apply() =
Behaviors.setup(context => new StartStopActor1(context))
}
class StartStopActor1(context: ActorContext[String]) extends
AbstractBehavior[String](context) {
println("first started")
context.spawn(StartStopActor2(), "second")
override def onMessage(msg: String): Behavior[String] =
msg match {
case "stop" => Behaviors.stopped
}
override def onSignal: PartialFunction[Signal, Behavior[String]] = {
case PostStop =>
println("first stopped")
this
}
}
object StartStopActor2 {
def apply() =
Behaviors.setup(context => new StartStopActor2(context))
}
class StartStopActor2(context: ActorContext[String]) extends
AbstractBehavior[String](context) {
println("second started")
override def onMessage(msg: String): Behavior[String] = Behaviors.unhandled
override def onSignal: PartialFunction[Signal, Behavior[String]] = {
case PostStop =>
println("second stopped")
this
}
}
Anything I am missing here? I copied code from there itself.
With the amount of information you provide it is impossible to answer your question. But my best guess is that your JVM exits before the first actor has a change to print its stop message.
Edit
It may also be that the Akka documentation is wrong: the first actor replaces its behavior with Behaviors.stopped, thus the PostStop signal is not delivered to the StartStopActor1 behavior but to the stopped behavior. I remember implementing it this way a few years back, with the rationale that the PostStop hook is not necessary when the actor voluntarily terminates: any code that you would want to run for PostStop can also be run before returning Behaviors.stopped.

Does Akka Decider have access to the full failure scenario?

New to Akka. Creating a new Scala class that extends SupervisorStrategy gives me the following template to work with:
class MySupervisorStrategy extends SupervisorStrategy {
override def decider: Decider = ???
override def handleChildTerminated(context: ActorContext, child: ActorRef,
children: Iterable[ActorRef]): Unit = ???
override def processFailure(context: ActorContext, restart: Boolean,
child: ActorRef, cause: Throwable, stats: ChildRestartStats, children: Iterable[ChildRestartStats]): Unit = ???
}
I'm looking for a way to access:
The Throwable/Exception that was thrown from the child actor
The child actor ActorRef that threw the exception
The message that was passed to the child actor that prompted the exception to be thrown
I think the Decider (which is actually a PartialFunction[Throwable,Directive]) gets passed the Throwable whenever the child throws the exception, but I'm not seeing where I could get access to #2 and #3 from my list above. Any ideas?
Update
From the posted fiddle, it looks like a valid Decider is:
{
case ActorException(ref,t,"stop") =>
println(s"Received 'stop' from ${ref}")
Stop
case ActorException(ref,t,"restart") =>
println(s"Received 'restart' from ${ref}")
Restart
case ActorException(ref,t,"resume") =>
println(s"Received 'resume' from ${ref}")
Resume
}
Above, I see all three:
The exception that was thrown by the child
The child (ref) that threw the exception
The message that was sent to the child originally (that caused the exception to be thrown)
It looks like there's nothing in that Decider that needs to be defined inside that Supervisor class. I'd like to pull the Decider logic out into, say, MyDecider.scala and find a way to refactor the Supervisor so that its supervisorStrategy uses an instance of MyDecider, so maybe something similar to:
class Supervisor extends Actor {
import akka.actor.OneForOneStrategy
import akka.actor.SupervisorStrategy._
import scala.concurrent.duration._
var child: ActorRef = _
override val supervisorStrategy =
OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1 minute, decider = myDecider)
...
}
For #2 you could access the sender "if the strategy is declared inside the supervising actor"
If the strategy is declared inside the supervising actor (as opposed to within a companion object) its decider has access to all internal state of the actor in a thread-safe fashion, including obtaining a reference to the currently failed child (available as the sender of the failure message).
The message is not made available so the only option is to catch your exception and throw a custom one with the message that was received.
Here is a quick fiddle
class ActorSO extends Actor {
def _receive: Receive = {
case e =>
println(e)
throw new RuntimeException(e.toString)
}
final def receive = {
case any => try {
_receive(any)
}
catch {
case t:Throwable => throw new ActorException(self,t,any)
}
}
}
Update
A Decider is just a PartialFunction so you can pass it in the constructor.
object SupervisorActor {
def props(decider: Decider) = Props(new SupervisorActor(decider))
}
class SupervisorActor(decider: Decider) extends Actor {
override val supervisorStrategy = OneForOneStrategy()(decider)
override def receive: Receive = ???
}
class MyDecider extends Decider {
override def isDefinedAt(x: Throwable): Boolean = true
override def apply(v1: Throwable): SupervisorStrategy.Directive = {
case t:ActorException => Restart
case notmatched => SupervisorStrategy.defaultDecider.apply(notmatched)
}
}
object Test {
val myDecider: Decider = {
case t:ActorException => Restart
case notmatched => SupervisorStrategy.defaultDecider.apply(notmatched)
}
val myDecider2 = new MyDecider()
val system = ActorSystem("stackoverflow")
val supervisor = system.actorOf(SupervisorActor.props(myDecider))
val supervisor2 = system.actorOf(SupervisorActor.props(myDecider2))
}
By doing so, you won't be able to access supervisor state like the ActorRef of the child that throw the exception via sender() (although we are including this in the ActorException)
Regarding your original question of accessing from the supervisor the child message that cause the exception, you can see here (from akka 2.5.3) that the akka developers choose to not make it available for decision.
final protected def handleFailure(f: Failed): Unit = {
// ¡¡¡ currentMessage.message is the one that cause the exception !!!
currentMessage = Envelope(f, f.child, system)
getChildByRef(f.child) match {
/*
* only act upon the failure, if it comes from a currently known child;
* the UID protects against reception of a Failed from a child which was
* killed in preRestart and re-created in postRestart
*/
case Some(stats) if stats.uid == f.uid ⇒
// ¡¡¡ currentMessage.message is not passed to the handleFailure !!!
if (!actor.supervisorStrategy.handleFailure(this, f.child, f.cause, stats, getAllChildStats)) throw f.cause
case Some(stats) ⇒
publish(Debug(self.path.toString, clazz(actor),
"dropping Failed(" + f.cause + ") from old child " + f.child + " (uid=" + stats.uid + " != " + f.uid + ")"))
case None ⇒
publish(Debug(self.path.toString, clazz(actor), "dropping Failed(" + f.cause + ") from unknown child " + f.child))
}
}

Order of ActorContext changes

I noticed that actor at first sent message about state change and later really has been changed this state. It's correct?
class MyActor extends Actor {
def receive = idle(Set.empty)
def idle(isInSet: Set[String]): Receive = {
case Add(key) =>
// sending the result as a message back to our actor
validate(key).map(Validated(key, _)).pipeTo(self)
// waiting for validation
context.become(waitForValidation(isInSet, sender()))
}
def waitForValidation(set: Set[String], source: ActorRef): Receive = {
case Validated(key, isValid) =>
val newSet = if (isValid) set + key else set
// sending acknowledgement of completion
source ! Continue
Here occurs sending notification
// go back to idle, accepting new requests
context.become(idle(newSet))
and later changed state
case Add(key) =>
sender() ! Rejected
}
def validate(key: String): Future[Boolean] = ???
}
// Messages
case class Add(key: String)
case class Validated(key: String, isValid: Boolean)
case object Continue
case object Rejected
You should probably consider moving become() before pipeTo(self) if you want the actor to receive the message in the waitForValidation state:
context.become(waitForValidation(isInSet, sender()))
validate(key).map(Validated(key, _)).pipeTo(self)
I agree that piping the message will put it in the queue, and by the time the object gets to processing it the object should be in the new state, but most of the examples I have seen call the become before piping just to be on the safe side.

AKKA Shutdown Pattern doesn't work

After I read this excellent blog written by the AKKA team, Shutdown Patterns in AKKA 2, I run the code and it indeed works.
But when I do another experiment with a slight change, throwing out an exception in workers, then this pattern won't work. Since it's reasonable that workers may throw any kinds of exceptions during working, right?
The following are my code, two files:
Reaper.scala, copied from the article mentioned above:
import akka.actor.{Actor, ActorRef, Terminated}
import scala.collection.mutable.ArrayBuffer
object Reaper {
// Used by others to register an Actor for watching
case class WatchMe(ref: ActorRef)
}
abstract class Reaper extends Actor {
import Reaper._
// Keep track of what we're watching
val watched = ArrayBuffer.empty[ActorRef]
// Derivations need to implement this method. It's the
// hook that's called when everything's dead
def allSoulsReaped(): Unit
// Watch and check for termination
final def receive = {
case WatchMe(ref) =>
context.watch(ref)
watched += ref
case Terminated(ref) =>
watched -= ref
if (watched.isEmpty) allSoulsReaped()
}
}
TestWorker.scala
import akka.actor.{SupervisorStrategy, Props, ActorSystem, Actor}
import Reaper._
class TestReaper extends Reaper {
def allSoulsReaped(): Unit = context.system.shutdown()
override val supervisorStrategy = SupervisorStrategy.stoppingStrategy
}
// The reaper sends this message to all workers to notify them to start work
case object StartWork
class TestWorker extends Actor {
def receive = {
case StartWork =>
// do real work ...
throw new IllegalStateException("Simulate uncaught exceptions during work")
}
}
object TestWorker {
def main(args: Array[String]) : Unit = {
val system = ActorSystem("system")
val reaper = system.actorOf(Props[TestReaper])
val worker1 = system.actorOf(Props[TestWorker])
val worker2 = system.actorOf(Props[TestWorker])
reaper ! WatchMe(worker1)
reaper ! WatchMe(worker2)
Thread.sleep(3000) // make sure WatchMe will be delivered before StartWork
worker1 ! StartWork
worker2 ! StartWork
system.awaitTermination()
}
}
This program will hang forever.
It seems the reaper cannot receive Terminated messages if workers throw uncaught exceptions
Can somebody tell my why? Many thanks in advance!
Correct answer by #mattinbits:
The reason that this program hangs forever is that in my code TestWorker are not children of TestReaper even TestReaper calls context.watch(ref).
context.watch() doesn't mean becoming a child. context.watch(ref) just means that TestReaper will get notified when a TestWorker actor dies.
SupervisorStrategy and context.watch() are two different things. SupervisorStrategy will only have influence on all children actors.
Putting override val supervisorStrategy = SupervisorStrategy.stoppingStrategy in the TestReaper won't make TestWorker stopped when exceptions happen inside TestWorker. Instead we need to change the SupervisorStrategy of TestWorkers parent actor. Since all actors in the code above are created by system.actorOf(), they are children of the Guardian Actor /user, so actually we need to change the supervision strategy of the /user actor , by adding akka { actor { guardian-supervisor-strategy = "akka.actor.StoppingSupervisorStrategy" } } in application.conf
However, it's better to use another actor as the supervision actor, just like what #mattinbits does in his code.
It is not enough to watch the actors, you also have to make sure the actors stop (which is the condition under which Terminated gets sent).
By default, when an actor throws an exception, the strategy is to restart it. You need to give the actors a supervisor which will apply the Stop directive.
Have a look at the following, both tests pass (with Reaper unchanged from your version above):
import java.util.concurrent.TimeoutException
import Reaper.WatchMe
import akka.actor.SupervisorStrategy.Stop
import akka.actor._
import akka.testkit.{TestProbe, TestKit}
import org.scalatest.{BeforeAndAfterAll, Matchers, WordSpecLike, WordSpec}
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
case object StartWork
class TestReaper extends Reaper {
def allSoulsReaped(): Unit = context.system.shutdown()
}
class TestWorker extends Actor {
def receive = {
case StartWork =>
// do real work ...
throw new IllegalStateException("Simulate uncaught exceptions during work")
}
}
class TestParent(reaper: ActorRef, probe: ActorRef) extends Actor {
def receive = {
case "Start" =>
val worker1 = context.actorOf(Props[TestWorker])
val worker2 = context.actorOf(Props[TestWorker])
reaper ! WatchMe(worker1)
reaper ! WatchMe(worker2)
worker1 ! StartWork
worker2 ! StartWork
}
override def supervisorStrategy = OneForOneStrategy() {
case ex: IllegalStateException =>
probe ! "Stopped a worker"
Stop
}
}
class TestSupervision extends TestKit(ActorSystem("Test"))
with WordSpecLike
with Matchers
with BeforeAndAfterAll{
"Supervision" should {
"Stop the actor system when the parent stops the workers" in {
val reaper = system.actorOf(Props[TestReaper])
val probe = TestProbe()
val parent = system.actorOf(Props(new TestParent(reaper, probe.ref)))
parent ! "Start"
probe.expectMsg("Stopped a worker")
probe.expectMsg("Stopped a worker")
import system.dispatcher
val terminatedF = Future {
system.awaitTermination()
}
Await.ready(terminatedF, 2 seconds)
}
}
override def afterAll(){
system.shutdown()
}
}
class TestLackSupervision extends TestKit(ActorSystem("Test2"))
with WordSpecLike
with Matchers
with BeforeAndAfterAll{
"Lack of Supervision" should {
"Not stop the actor system when the workers don't have an appropriate parent" in {
val reaper = system.actorOf(Props[TestReaper])
val worker1 = system.actorOf(Props[TestWorker])
val worker2 = system.actorOf(Props[TestWorker])
reaper ! WatchMe(worker1)
reaper ! WatchMe(worker2)
import system.dispatcher
val terminatedF = Future { system.awaitTermination()}
a [TimeoutException] should be thrownBy Await.ready(terminatedF, 2 seconds)
}
}
override def afterAll(){
system.shutdown()
}
}
By default when actors throw an exception they are restarted. Since supervision strategy is applied from parent to child, the TestParent exists to enforce the Stop directive on the children. Your original code would not work for this reason.
If you want top level actors (Those launched with system.actorOf) to stop on an exception, you could set the configuration property akka.actor.guardian-supervisor-strategy = "akka.actor.StoppingSupervisorStrategy" but in my example I prefer to use a parent actor since actor hierarchies are a normal way to organise supervision in Akka.
To run as an app, do something similar to the following:
object Main extends App {
val system = ActorSystem("Example")
val reaper = system.actorOf(Props[TestReaper])
val dummyProbe = system.actorOf(Props(new Actor{
def receive = {
case "Stopped a worker" => println("Stopped a worker")
}
}))
val parent = system.actorOf(Props(new TestParent(reaper, dummyProbe)))
parent ! "Start"
system.awaitTermination()
}
To stop the exceptions from being printed on the command line and muddying the output, change the supervision strategy as follows:
override def supervisorStrategy = OneForOneStrategy(loggingEnabled = false) {...}

Akka Actors logging processing time

I have a set of Akka Actors and I give about a couple of hundreds of messages to each one of them. I want to track how much time each instance of that Actor took to process all the messages that it received. What I'm doing currently is to have a state in the Actor instance as:
var startTime
var firstCall
I set both the variables when the Actor instance is first called. Is there another way that I could use to track the processing time for my Actor instances? I want to avoid having a local state in my Actor instance.
This is a good use case for context.become.
Remember than a receive block in an Akka actor is just a PartialFunction[Any, Unit], so we can wrap that in another partial function. This is the same approach taken by Akka's builtin LoggingReceive.
class TimingReceive(r: Receive, totalTime: Long)(implicit ctx: ActorContext) extends Receive {
def isDefinedAt(o: Any): Boolean = {
r.isDefinedAt(o)
}
def apply(o: Any): Unit = {
val startTime = System.nanoTime
r(o)
val newTotal = totalTime + (System.nanoTime - startTime)
log.debug("Total time so far: " + totalTime + " nanoseconds")
ctx.become(new TimingReceive(r, newTotal))
}
}
object TimingReceive {
def apply(r: Receive)(implicit ctx: ActorContext): Receive = new TimingReceive(r, 0)
}
Then you can use it like this:
class FooActor extends Actor {
def receive = TimingReceive {
case x: String => println("got " + x)
}
}
After each message, the actor will log the time taken so far. Of course, if you want to do something else with that variable, you'll have to adapt this.
This approach doesn't measure the time the actor is alive of course, only the time taken to actually process messages. Nor will it be accurate if your receive function creates a future.