I have very simple application (2 actors) that sends file through socket. Application uses Spring Boot and Akka, and after transfer is finished, nothing else happens. However, I have noticed high CPU usage, even after transfer (like 50% on 4 core machine, so full 2 cores). I have captured snapshot of active threads, and about 1/3 of them belongs to Akka.
What is Akka doing, and why does it take so much cpu?
http://pastebin.com/CF9yUvuL
To answer your question generally: Akka creates a thread pool which is used to do asynchronous message processing. So if you write code like the following, your application would never terminate since Akka keeps on running.
class Root extends Actor {
// create some child actors to do some work
override def receive: Receive = Actor.emptyBehavior
}
object Main {
def main(args: Array[String]): Unit = {
val system = ActorSystem("main")
system.actorOf(Props(new Root))
}
}
So usually if you need to terminate your application you would define a message which signals that all work has been done and send this message to your root actor. The root actor can then call context.system.terminate() to terminate the system.
Coming back to your question, I don't understand why you would want to combine Akka with Spring Boot. Your thread dump, I'm not sure whether the problem is related to Akka at all. It looks like the threads are hanging inside HttpClient. Are you sure you have freed and closed all open connections?
Related
In my akka application , i am using a main actor as controller which receives commands and delegates it to processor actor .Processor actor upon finishing (which takes around 2 mins to finish each task) , passes the message to controller and then controller actor , sends message to Database actor for persistence. Both processor actor and db database actor are managed using router each with say 5 routees. I am using default dispatcher and all other akka configuration are default only .Now below is the scenario.
The controller actor is receiving around 100 messages which are passed to processor actor , i can see from the log that the processor has finished processing some messages (around 5) and passed on completion message to controller actor. But database actor starts executing after some 5 mins. During this 5 mins , however the processor actor is processing its pending messages. So its not like the application is idle.
When the volume of message is less , the flow from controller -> processor -> controller -> db actor is almost instantaneously and there is hardly any lag.
I dont want this lag after processing , DB execution should happen as soon as processing is finished. But seems threads are busy executing processor task.How can i overcome this situation , ideally i want the turn around time of my task execution to be less , but due to above behavior i am not able to achieve it.
By default, all Akka actors use the same executor which is limited to use 64 threads maximum. From https://doc.akka.io/docs/akka/current/general/configuration-reference.html :
# This will be used if you have set "executor = "default-executor"".
# If an ActorSystem is created with a given ExecutionContext, this
# ExecutionContext will be used as the default executor for all
# dispatchers in the ActorSystem configured with
# executor = "default-executor". Note that "default-executor"
# is the default value for executor, and therefore used if not
# specified otherwise. If no ExecutionContext is given,
# the executor configured in "fallback" will be used.
default-executor {
fallback = "fork-join-executor"
}
and fork-join-executor config:
# This will be used if you have set "executor = "fork-join-executor""
# Underlying thread pool implementation is java.util.concurrent.ForkJoinPool
fork-join-executor {
# Min number of threads to cap factor-based parallelism number to
parallelism-min = 8
# The parallelism factor is used to determine thread pool size using the
# following formula: ceil(available processors * factor). Resulting size
# is then bounded by the parallelism-min and parallelism-max values.
parallelism-factor = 1.0
# Max number of threads to cap factor-based parallelism number to
parallelism-max = 64
# Setting to "FIFO" to use queue like peeking mode which "poll" or "LIFO" to use stack
# like peeking mode which "pop".
task-peeking-mode = "FIFO"
}
The problem could be related to blocking calls in processor actors. Akka assigns separate threads from a pool of 64 to handle these blocking calls in processor actors and waits for one of them to complete message processing to be able to handle messages for other actors. This might cause the time lag that you observe between actors.
A key aspect on which Akka is based is that systems should remain responsive at all times. If you used the same dispatcher/thread pool for the blocking DB operations or processing messages as your main Akka routing infrastructure, it is possible that all the Akka threads could be occupied by the processing actors or DB operations, and your system would effectively be deadlocked until one of the blocking operations is completed. This might not be a problem for a simple system on a single JVM that only performs this task, but when it is scaled, it might cause a lot of problems.
In cases like yours where you cannot avoid blocking, a dedicated dispatcher for blocking operations should be used. This link talks about this aspect(though it is titled for Akka-Http, it can be generalized). You can create two types of dispatchers to handle the two different blocking operations. I also, think that you should throttle your blocking requests as to not overwhelm your system(use dispatchers for throttling). You could also implement buffers within your actors to deal with backpressure situations.
EDIT
The controller mailbox has 100 messages and 5 messages are taken and delegated to processor actors. Each processor actor takes 2 mins of time and sends the response back to the controller and the response gets queued in the controller's mailbox. But before processing these messages, the controller needs to process messages which were added before these messages effectively increasing the service time for processing the messages for controller. The lag is the culmination of this whole process. As soon as the controller got the response message, it delegated to the actor.I think that the processing time increases as the messages increase.
Let me know if it helps!!
I'm new to the Akka framework and I'm building a group chat application with it. My application may have 10 million actor instances (an actor instance for each group chat) of the same type, only 5% of which are highly active, and 60% of which can be idle (not receiving any messages) for days.
I want to know:
Is there any best practice to identify these idle actors?
What is the best practice to deal with them? Is stopping them enough?
Is there any best practice to identify these idle actors?
An actor's ActorContext has a setReceiveTimeout method that defines an inactivity threshold for the actor: if the actor hasn't received a message in the given amount of time, then an akka.actor.ReceiveTimeout message is sent to the actor. For example:
import akka.actor.{ Actor, ReceiveTimeout }
import scala.concurrent.duration._
class ChatActor extends Actor {
context.setReceiveTimeout(2 hours)
def receive = {
case ReceiveTimeout =>
// do something
// other case clauses
}
}
The above ChatActor will receive a ReceiveTimeout message if it hasn't received a message for two hours. (However, as the documentation states: "the receive timeout might fire and enqueue the ReceiveTimeout message right after another message was enqueued; hence it is not guaranteed that upon reception of the receive timeout there must have been an idle period beforehand as configured via this method.")
What is the best practice to deal with them?
It's a good idea to stop inactive actors; otherwise you could have a memory leak. Here are a few approaches for stopping these actors:
The inactive actor throws an exception, which is handled in a supervisor strategy defined in the actor's parent. In the supervisor strategy, the parent stops the idle actor (e.g., via context stop sender()).
The inactive actor sends its self reference to a "reaper" actor that collects references to idle actors and culls (i.e., stops) these actors on a periodic basis (perhaps using a scheduler).
The inactive actor stops itself (via context stop self).
More information about stopping actors is found here.
Is stopping them enough?
When an actor is stopped, its ActorRef essentially becomes invalid. From the documentation:
After stopping an actor, its postStop hook is called, which may be used e.g. for deregistering this actor from other services. This hook is guaranteed to run after message queuing has been disabled for this actor, i.e. messages sent to a stopped actor will be redirected to the deadLetters of the ActorSystem.
At this point, the underlying actor instance to which the now-stale ActorRef points is eligible for garbage collection. In other words, an actor must be stopped in order for it to be eligible for garbage collection. Therefore, in regard to freeing up memory, stopping the actor is enough. You could also remove the invalid ActorRef itself after the actor has been stopped. Note that removing an ActorRef does not automatically stop the actor:
It is important to note that Actors do not stop automatically when no longer referenced, every Actor that is created must also explicitly be destroyed.
Is there any best practice to identify these idle actors?
The only way is to make each actor to keep the time when it was active last time. Then, to speedup investigation of the longest inactive actor, you can organize an index-like structure, e.g. PriorityQueue. Then a dedicated actor periodically awakes and cleans that structure from actors which are idle longer than some predefined period of time.
What is the best practice to deal with them? Is stopping them enough?
An idle actor does not consume any resources except core memory. If you have plenty of memory, the best practice is to do nothing. If you want to save that memory, store actor in database (after some period of inactivity), and then read it from there by demand.
I have written some actor classes and I find that I have to get a handle into the lifecycle of these entities. For example whenever my actor is initialized I would like a method to be called so that I can setup some listeners on message queues (or open db connections etc).
Is there an equivalent of this? The equivalent I can think of is Spring's InitialisingBean and DisposableBean
This is a typical scenario where you would override methods like preStart(), postStop(), etc. I don't see anything wrong with this.
Of course you have to be aware of the details - for example postStop() is called asynchronously after actor.stop() is invoked while preStart() is called when an Actor is started. This means that potentially slow/blocking things like DB interaction should be kept to a minimum.
You can also use the Actor's constructor for initialization of data.
As Matthew mentioned, supervision plays a big part in Akka - so you can instruct the supervisor to perform specific stuff on events. For example the so-called DeathWatch - you can be notified if one of the actors "you are watching upon" dies:
context.watch(child)
...
def receive = {
case Terminated(`child`) => lastSender ! "finished"
}
An Actor is basically two methods -- a constructor, and onMessage(Object): void.
There's nothing in its lifecycle that naturally provides for "wiring" behavior, which leaves you with a few options.
Use a Supervisor actor to create your other actors. A Supervisor is responsible for watching, starting and restarting Actors on failure -- and therefore it is often valuable to have a Supervisor that understands the state of integrated systems to avoid continously restarting. This Supervisor would create and manage Service objects (possibly via Spring) and pass them to Actors.
Use your preferred Initialization technique at the time of Actor construction. It's tricky but you can certainly combine Spring with Actors. Just be aware that should a Supervisor restart your actor, you'll need to be able to resurrect its desired state from whatever content you placed in the Props object you used to start it in the first place.
Wire everything on-demand. Open connections on demand when an Actor starts (and cache them as necessary). I find I do this fairly often -- and I let the Actor fail when its connections no longer work. The supervisor will restart the Actor, which will recreate all connections.
Something important things to remember:
The intent of Actor model is that Actors don't run continuously -- they only run when there are messages provided to them. If you add a message listener to an Actor, you are essentially adding new threads that can access that actor. This can be a problem if you use supervision -- a restarted actor may leak that thread and this may in turn cause the actor not to be garbage collected. It can also be a problem because it introduces a race condition, and part of the value of actors is avoiding that.
An Actor that does I/O is, from the perspective of the actor system, blocking. If you have too many Actors doing I/O at the same time, you will exhaust your Dispatcher's thread pool and lock up the system.
A given Actor instance can operate on many different threads over its lifetime, but will only operate on one thread at a time. This can be confusing to some messaging systems -- for example, JMS' Spec asserts that a Session not be used on multiple threads, and many JMS interpret this as "can only run on the thread on which it was started." You may see warnings, or even exceptions, resulting from this.
For these reasons, I prefer to use non-actor code to do some of my I/O. For example, I'll have an incoming message listener object whose responsibility is to take JMS messages off a queue, use them to create POJO messages, and send tells to the Actor system. Alternately, I'll use an Actor, but place that actor on a custom Dispatcher that has thread pinning enabled. This assures that that Actor will only run on a specific thread and won't block up the system that other non-I/O actors are using.
If say the code that my actor uses (a code I have no control over) throws an unhandled exception, could that result into the whole actor system process to crash or each actor is running in some kind of special container?
To clarify more, in my use case, I want each actor to load (at run time) some user written code/lib and call some interface methods on them. These libs maybe buggy and can potentially result in my actor system os process to die or halt or something like that. I mean what if the code that actor calls does something that halt (like accessing a remote resource by a buggy client or a dead loop) or even call Enviroment.exit() or something of bad nature.
I mean if my requirement is to allow each actor to load code that I do not have control over, how can I guard my actor system against them? Do I even have to do this?
One way that I can think the whole actor system OS process guard itself against these third party code is to run each actor inside some kind of a container or event have one actor system per actor on the local machine that my actor controls? Do I have to go this far or akka already takes care of this for me and any failure at actor level would not jeopardize the whole actor system and its process??
If the JVM process dies, the JVM process dies. You get around that by using Akka Cluster so you can observe and react to node failures.
How can I gracefully restart an entire actor system? In other words, stop all the actors in the system with a gracefulStop (or at least the important ones!)
Basically I want to, in effect, restart the application during a test, without the overhead of starting up a new separate JVM each time.
I'm using Akka 2.0.5.
Whenever you have implemented an actor system and it is lacking one function, don’t hesitate to add one more actor to take care of that: in this case create one actor whose job is it to send graceful stop requests to your important actors and await their termination. That actor can then shut down the actor system, while from the outside you awaitTermination. If you created that actor with system.actorOf(Props[Terminator], "terminator"), then you can shut it down with
system.actorFor("/user/terminator") ! MakeItStop // or whatever message you choose
try system.awaitTermination(1.minute) // or however long it may take
catch {
case _: TimeoutException => system.shutdown() // as a last resort
}
If it's for unit test you can just stop the ActorSystem and start a new one.
val system1 = ActorSystem("system1")
system1.shutdown() // This is async and will return before the system actually stops.
val system2 = ActorSystem("system2")
I'm not sure how you can restart the system (actually, I'm not sure the above would work, but I see no reason for it not to).