I am trying to configure an Akka Actor for Cluster Sharding, one thing that I am not quite sure, is it possible to configure several Stop Messages for an Entity for graceful Shutdown.
for ex, Entity configuration like following will trigger graceful shutdown for both 'onDelete' and 'onExit' or it will do it only for 'onExit'?
sharding
.init(
Entity(Actor1Key) {
context => ....
}
)
.withStopMessage(Actor1.onDelete)
.withStopMessage(Actor1.onExit)
if not do you have any idea how I can achieve this Behaviour?
Thx for answers
I think there may some confusion around what the purpose of the stopMessage is. There should not be a need for multiple stop messages.
The stopMessage sent by sharding after passivation has been requested by the actor, which is done by sending Passivate from the sharded actor itself.
You can let any of the messages that the actor accepts trigger passivation, the shard will send back the stopMessage when it is safe for the actor to actually stop.
The reason you should passivate rather than just Behaviors.stopped the actor is that there may be messages that was en route to the actor (mailbox and I think possibly a buffer in the shard in some circumstances) before the message causing it deciding to stop and you want to process those first. Passivation allows for that to happen by including a roundtrip to the shard actor which is charge of routing messages to the sharded actor.
A bit more details in the docs here: https://doc.akka.io/docs/akka/current/typed/cluster-sharding.html#passivation
What you have specified would only trigger the stop message for Actor1.onExit. The reason is how a stop message is defined for an Entity:
val stopMessage: Optional[M],
So you see that this is a plain optional thus no multiple elements are possible. You can also check how the withStopMessage is implemented here:
def withStopMessage(newStopMessage: M): Entity[M, E] =
copy(stopMessage = Optional.ofNullable(newStopMessage))
So you are basically going to "overwrite" the message any time you call withStopMessage. Unfortunately, I am not aware of any other way of specifying multiple stop messages (besides combining multiple messages in a common trait but I think this is not what you are looking for).
Related
I'm new to the Akka framework and I'm building a group chat application with it. My application may have 10 million actor instances (an actor instance for each group chat) of the same type, only 5% of which are highly active, and 60% of which can be idle (not receiving any messages) for days.
I want to know:
Is there any best practice to identify these idle actors?
What is the best practice to deal with them? Is stopping them enough?
Is there any best practice to identify these idle actors?
An actor's ActorContext has a setReceiveTimeout method that defines an inactivity threshold for the actor: if the actor hasn't received a message in the given amount of time, then an akka.actor.ReceiveTimeout message is sent to the actor. For example:
import akka.actor.{ Actor, ReceiveTimeout }
import scala.concurrent.duration._
class ChatActor extends Actor {
context.setReceiveTimeout(2 hours)
def receive = {
case ReceiveTimeout =>
// do something
// other case clauses
}
}
The above ChatActor will receive a ReceiveTimeout message if it hasn't received a message for two hours. (However, as the documentation states: "the receive timeout might fire and enqueue the ReceiveTimeout message right after another message was enqueued; hence it is not guaranteed that upon reception of the receive timeout there must have been an idle period beforehand as configured via this method.")
What is the best practice to deal with them?
It's a good idea to stop inactive actors; otherwise you could have a memory leak. Here are a few approaches for stopping these actors:
The inactive actor throws an exception, which is handled in a supervisor strategy defined in the actor's parent. In the supervisor strategy, the parent stops the idle actor (e.g., via context stop sender()).
The inactive actor sends its self reference to a "reaper" actor that collects references to idle actors and culls (i.e., stops) these actors on a periodic basis (perhaps using a scheduler).
The inactive actor stops itself (via context stop self).
More information about stopping actors is found here.
Is stopping them enough?
When an actor is stopped, its ActorRef essentially becomes invalid. From the documentation:
After stopping an actor, its postStop hook is called, which may be used e.g. for deregistering this actor from other services. This hook is guaranteed to run after message queuing has been disabled for this actor, i.e. messages sent to a stopped actor will be redirected to the deadLetters of the ActorSystem.
At this point, the underlying actor instance to which the now-stale ActorRef points is eligible for garbage collection. In other words, an actor must be stopped in order for it to be eligible for garbage collection. Therefore, in regard to freeing up memory, stopping the actor is enough. You could also remove the invalid ActorRef itself after the actor has been stopped. Note that removing an ActorRef does not automatically stop the actor:
It is important to note that Actors do not stop automatically when no longer referenced, every Actor that is created must also explicitly be destroyed.
Is there any best practice to identify these idle actors?
The only way is to make each actor to keep the time when it was active last time. Then, to speedup investigation of the longest inactive actor, you can organize an index-like structure, e.g. PriorityQueue. Then a dedicated actor periodically awakes and cleans that structure from actors which are idle longer than some predefined period of time.
What is the best practice to deal with them? Is stopping them enough?
An idle actor does not consume any resources except core memory. If you have plenty of memory, the best practice is to do nothing. If you want to save that memory, store actor in database (after some period of inactivity), and then read it from there by demand.
I have written some actor classes and I find that I have to get a handle into the lifecycle of these entities. For example whenever my actor is initialized I would like a method to be called so that I can setup some listeners on message queues (or open db connections etc).
Is there an equivalent of this? The equivalent I can think of is Spring's InitialisingBean and DisposableBean
This is a typical scenario where you would override methods like preStart(), postStop(), etc. I don't see anything wrong with this.
Of course you have to be aware of the details - for example postStop() is called asynchronously after actor.stop() is invoked while preStart() is called when an Actor is started. This means that potentially slow/blocking things like DB interaction should be kept to a minimum.
You can also use the Actor's constructor for initialization of data.
As Matthew mentioned, supervision plays a big part in Akka - so you can instruct the supervisor to perform specific stuff on events. For example the so-called DeathWatch - you can be notified if one of the actors "you are watching upon" dies:
context.watch(child)
...
def receive = {
case Terminated(`child`) => lastSender ! "finished"
}
An Actor is basically two methods -- a constructor, and onMessage(Object): void.
There's nothing in its lifecycle that naturally provides for "wiring" behavior, which leaves you with a few options.
Use a Supervisor actor to create your other actors. A Supervisor is responsible for watching, starting and restarting Actors on failure -- and therefore it is often valuable to have a Supervisor that understands the state of integrated systems to avoid continously restarting. This Supervisor would create and manage Service objects (possibly via Spring) and pass them to Actors.
Use your preferred Initialization technique at the time of Actor construction. It's tricky but you can certainly combine Spring with Actors. Just be aware that should a Supervisor restart your actor, you'll need to be able to resurrect its desired state from whatever content you placed in the Props object you used to start it in the first place.
Wire everything on-demand. Open connections on demand when an Actor starts (and cache them as necessary). I find I do this fairly often -- and I let the Actor fail when its connections no longer work. The supervisor will restart the Actor, which will recreate all connections.
Something important things to remember:
The intent of Actor model is that Actors don't run continuously -- they only run when there are messages provided to them. If you add a message listener to an Actor, you are essentially adding new threads that can access that actor. This can be a problem if you use supervision -- a restarted actor may leak that thread and this may in turn cause the actor not to be garbage collected. It can also be a problem because it introduces a race condition, and part of the value of actors is avoiding that.
An Actor that does I/O is, from the perspective of the actor system, blocking. If you have too many Actors doing I/O at the same time, you will exhaust your Dispatcher's thread pool and lock up the system.
A given Actor instance can operate on many different threads over its lifetime, but will only operate on one thread at a time. This can be confusing to some messaging systems -- for example, JMS' Spec asserts that a Session not be used on multiple threads, and many JMS interpret this as "can only run on the thread on which it was started." You may see warnings, or even exceptions, resulting from this.
For these reasons, I prefer to use non-actor code to do some of my I/O. For example, I'll have an incoming message listener object whose responsibility is to take JMS messages off a queue, use them to create POJO messages, and send tells to the Actor system. Alternately, I'll use an Actor, but place that actor on a custom Dispatcher that has thread pinning enabled. This assures that that Actor will only run on a specific thread and won't block up the system that other non-I/O actors are using.
I have three actors to handle a CQRS scenario. The ProductWorkerActor handles the command part, ProductQueryWorkerActor handles the query part and ProductStateActor handles the state part.
The way I'm handling the query is by using:
ProductQueryWorkerActor.Ask<ProductState>("give-me-product-state-for-product-1000")
The code from ProductQueryWorkerActor:
if (message == "give-me-product-state-for-product-1000")
{
var actor = Context.ActorSelection("akka://catalogSystem/user/productState/1000");
var psDTO = actor.Ask<ProductStateDTO>(message).Result;
Sender.Tell(ps);
}
Please ignore the path being used to access the product state. It is hardcoded and intentional to make the code read simpler.
Should I be using Ask as I have used in this case to retrieve the state of a product? Is Ask called Futures?
Should I be exposing the state as DTO to the outside work instead of the actor itself?
To change any state of the product, should I handle the message processing in ProductWorkerActor or in ProductStateActor itself? In the second case, the ProductWorkerActor sends a message to ProductStateWorker, the ProductStateWorker processes the message, change the state and send another message to ProductWorkerActor that it passed validation and changed the state.
In case when you're using Event Sourcing with your actors, I advice you to use Akka.Persistence. It handles read/write actors separation and will take a lot of burden from you shoulders.
If not, in my opinion basic problem with your design is that, while you have separate actors for reading/writing to state, state itself is handled in only one actor. Why? One of the points of CQRS is to have a separate models optimized for serving their role (either read or write).
In example: you can have one handler actor (eg. ProductActor) changing it's state based on incoming commands, and bunch of different readonly actors (eg. ProductHistoryActor, ProductListActor), each with it's own state optimized for their role. Readonly actors may subscribe to event stream to listen for incoming messages about handler actor's state changes and updating their own states accordingly, while handler actor after handling a command publishes message about state change using actor system's event stream.
Ad. 1: In my opinion using Ask to communicate between actors is an anti-pattern. In your example, you're using query actor to pass message through to state actor, then blocking current actor until response arrives (which is very bad for performance) just to send message back to sender. Instead of using:
var psDTO = actor.Ask<ProductStateDTO>(message).Result;
Sender.Tell(ps);
you could simply write:
actor.Forward(message);
and let actor send response directly to sender (you query actor doesn't need to participate with sending the response).
Ad. 2: It depends on your case, but remember - you should never pass mutable objects as messages, especially when you use them after sending.
Ad. 3: I think that in your example distinction between ProductWorkerActor and ProductStateWorker is artificial. From what you're showing, they should be a single entity IMO.
EDIT: One important thing I forgot mentioning: the actor creation described below depends on the data - sometimes few processing actors are required, and sometimes many.
One component I'm working on needs to create a number of actors (possibly, round-robin routed ones), that each get a rather large amount of messages to process. Each of those actors belongs to a "processing batch" which has a the same initialization parameters.
When I'm running this on the production machine with many messages, I quickly get a number of actor creation timeouts. I'm creating the actors directly with ActorSystem.actorOf().
What's surprising me though is that all in all there aren't that many actors being created I'd think (8 "processing sinks" with 5 round-robin routed actors would be 40 actors, which doesn't seem very much).
I'm shutting down the actors once they're not needed anymore by having another actor (which counts the amount of successes and failures that it gets via the "processing" actors) send them a PoisonPill so I'd think that they are all shut down correctly.
Am I perhaps doing something wrong here in the way that I am creating those actors, e.g. should I perhaps create them differently? Or would an appropriate strategy be to wait for some of the batches to be done before creating new actors?
Since you did not specify which version you are using I’m assuming that you will be interested in reading this:
http://doc.akka.io/docs/akka/2.0.3/scala/actors.html#Creating_Actors_with_default_constructor (especially the warning)
Besides the technical argument, creating your actors at top level is not a good design, missing out on the fault handling benefits.
I'm using Qt framework which has by default non-blocking I/O to develop an application navigating through several web pages (online stores) and carrying out different actions on these pages. I'm "mapping" specific web page to a state machine which I use to navigate through this page.
This state machine has these transitions;
Connect, LogIn, Query, LogOut, Disconnect
and these states;
Start, Connecting, Connected, LoggingIn, LoggedIn, Querying, QueryDone, LoggingOut, LoggedOut, Disconnecting, Disconnected
Transitions from *ing to *ed states (Connecting->Connected), are due to LoadFinished asynchronous network events received from network object when currently requested url is loaded. Transitions from *ed to *ing states (Connected->LoggingIn) are due to events send by me.
I want to be able to send several events (commands) to this machine (like Connect, LogIn, Query("productA"), Query("productB"), LogOut, LogIn, Query("productC"), LogOut, Disconnect) at once and have it process them. I don't want to block waiting for the machine to finish processing all events I sent to it. The problem is they have to be interleaved with the above mentioned network events informing machine about the url being downloaded. Without interleaving machine can't advance its state (and process my events) because advancing from *ing to *ed occurs only after receiving network type of event.
How can I achieve my design goal?
EDIT
The state machine I'm using has its own event loop and events are not queued in it so could be missed by machine if they come when the machine is busy.
Network I/O events are not posted directly to neither the state machine nor the event queue I'm using. They are posted to my code (handler) and I have to handle them. I can forward them as I wish but please have in mind remark no. 1.
Take a look at my answer to this question where I described my current design in details. The question is if and how can I improve this design by making it
More robust
Simpler
Sounds like you want the state machine to have an event queue. Queue up the events, start processing the first one, and when that completes pull the next event off the queue and start on that. So instead of the state machine being driven by the client code directly, it's driven by the queue.
This means that any logic which involves using the result of one transition in the next one has to be in the machine. For example, if the "login complete" page tells you where to go next. If that's not possible, then the event could perhaps include a callback which the machine can call, to return whatever it needs to know.
Asking this question I already had a working design which I didn't want to write about not to skew answers in any direction :) I'm going to describe in this pseudo answer what the design I have is.
In addition to the state machine I have a queue of events. Instead of posting events directly to the machine I'm placing them in the queue. There is however problem with network events which are asynchronous and come in any moment. If the queue is not empty and a network event comes I can't place it in the queue because the machine will be stuck waiting for it before processing events already in the queue. And the machine will wait forever because this network event is waiting behind all events placed in the queue earlier.
To overcome this problem I have two types of messages; normal and priority ones. Normal ones are those send by me and priority ones are all network ones. When I get network event I don't place it in the queue but instead I send it directly to the machine. This way it can finish its current task and progress to the next state before pulling the next event from the queue of events.
It works designed this way only because there is exactly 1:1 interleave of my events and network events. Because of this when the machine is waiting for a network event it's not busy doing anything (so it's ready to accept it and does not miss it) and vice versa - when the machine waits for my task it's only waiting for my task and not another network one.
I asked this question in hope for some more simple design than what I have now.
Strictly speaking, you can't. Because you only have state "Connecting", you don't know whether you need top login afterwards. You'd have to introduce a state "ConnectingWithIntentToLogin" to represent the result of a "Connect, then Login" event from the Start state.
Naturally there will be a lot of overlap between the "Connecting" and the "ConnectingWithIntentToLogin" states. This is most easily achieved by a state machine architecture that supports state hierarchies.
--- edit ---
Reading your later reactions, it's now clear what your actual problem is.
You do need extra state, obviously, whether that's ingrained in the FSM or outside it in a separate queue. Let's follow the model you prefer, with extra events in a queue. The rick here is that you're wondering how to "interleave" those queued events vis-a-vis the realtime events. You don't - events from the queue are actively extracted when entering specific states. In your case, those would be the "*ed" states like "Connected". Only when the queue is empty would you stay in the "Connected" state.
If you don't want to block, that means you don't care about the network replies. If on the other hand the replies interest you, you have to block waiting for them. Trying to design your FSM otherwise will quickly lead to your automaton's size reaching infinity.
How about moving the state machine to a different thread, i. e. QThread. I would implent a input queue in the state machine so I could send queries non blocking and a output queue to read the results of the queries. You could even call back a slotted function in your main thread via connect(...) if a result of a query arrives, Qt is thread safe in this regard.
This way your state machine could block as long as it needs without blocking your main program.
Sounds like you just want to do a list of blocking I/O in the background.
So have a thread execute:
while( !commands.empty() )
{
command = command.pop_back();
switch( command )
{
Connect:
DoBlockingConnect();
break;
...
}
}
NotifySenderDone();