In my akka application , i am using a main actor as controller which receives commands and delegates it to processor actor .Processor actor upon finishing (which takes around 2 mins to finish each task) , passes the message to controller and then controller actor , sends message to Database actor for persistence. Both processor actor and db database actor are managed using router each with say 5 routees. I am using default dispatcher and all other akka configuration are default only .Now below is the scenario.
The controller actor is receiving around 100 messages which are passed to processor actor , i can see from the log that the processor has finished processing some messages (around 5) and passed on completion message to controller actor. But database actor starts executing after some 5 mins. During this 5 mins , however the processor actor is processing its pending messages. So its not like the application is idle.
When the volume of message is less , the flow from controller -> processor -> controller -> db actor is almost instantaneously and there is hardly any lag.
I dont want this lag after processing , DB execution should happen as soon as processing is finished. But seems threads are busy executing processor task.How can i overcome this situation , ideally i want the turn around time of my task execution to be less , but due to above behavior i am not able to achieve it.
By default, all Akka actors use the same executor which is limited to use 64 threads maximum. From https://doc.akka.io/docs/akka/current/general/configuration-reference.html :
# This will be used if you have set "executor = "default-executor"".
# If an ActorSystem is created with a given ExecutionContext, this
# ExecutionContext will be used as the default executor for all
# dispatchers in the ActorSystem configured with
# executor = "default-executor". Note that "default-executor"
# is the default value for executor, and therefore used if not
# specified otherwise. If no ExecutionContext is given,
# the executor configured in "fallback" will be used.
default-executor {
fallback = "fork-join-executor"
}
and fork-join-executor config:
# This will be used if you have set "executor = "fork-join-executor""
# Underlying thread pool implementation is java.util.concurrent.ForkJoinPool
fork-join-executor {
# Min number of threads to cap factor-based parallelism number to
parallelism-min = 8
# The parallelism factor is used to determine thread pool size using the
# following formula: ceil(available processors * factor). Resulting size
# is then bounded by the parallelism-min and parallelism-max values.
parallelism-factor = 1.0
# Max number of threads to cap factor-based parallelism number to
parallelism-max = 64
# Setting to "FIFO" to use queue like peeking mode which "poll" or "LIFO" to use stack
# like peeking mode which "pop".
task-peeking-mode = "FIFO"
}
The problem could be related to blocking calls in processor actors. Akka assigns separate threads from a pool of 64 to handle these blocking calls in processor actors and waits for one of them to complete message processing to be able to handle messages for other actors. This might cause the time lag that you observe between actors.
A key aspect on which Akka is based is that systems should remain responsive at all times. If you used the same dispatcher/thread pool for the blocking DB operations or processing messages as your main Akka routing infrastructure, it is possible that all the Akka threads could be occupied by the processing actors or DB operations, and your system would effectively be deadlocked until one of the blocking operations is completed. This might not be a problem for a simple system on a single JVM that only performs this task, but when it is scaled, it might cause a lot of problems.
In cases like yours where you cannot avoid blocking, a dedicated dispatcher for blocking operations should be used. This link talks about this aspect(though it is titled for Akka-Http, it can be generalized). You can create two types of dispatchers to handle the two different blocking operations. I also, think that you should throttle your blocking requests as to not overwhelm your system(use dispatchers for throttling). You could also implement buffers within your actors to deal with backpressure situations.
EDIT
The controller mailbox has 100 messages and 5 messages are taken and delegated to processor actors. Each processor actor takes 2 mins of time and sends the response back to the controller and the response gets queued in the controller's mailbox. But before processing these messages, the controller needs to process messages which were added before these messages effectively increasing the service time for processing the messages for controller. The lag is the culmination of this whole process. As soon as the controller got the response message, it delegated to the actor.I think that the processing time increases as the messages increase.
Let me know if it helps!!
Related
We observe that on increasing concurrent http calls to our service, thread count (akka.actor.default-dispatcher) keeps increasing (see screenshot from visualVM). Also after the requests stop, the thread count don’t go down. And most of these remain in PARK state.
Is this proportional increase of threads an expected behaviour? How do we control this and reuse the same actors or kill the actors after request has been served.
I’m running the shopping-cart example from lagom-samples.
akka.actor.default-dispatcher {
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 2
parallelism-factor = 1.0
parallelism-max = 6
}
throughput = 1
}
VisualVM SS of thread analysis for lagom application
Edit: Using thread-pool-executor as akka.actor.default-dispatcher stops serving any requests after multiple (20-30) concurrent requests. Even console goes unresponsive.
default-dispatcher {
type = Dispatcher
executor = default-executor
throughput = 1
default-executor = { fallback = thread-pool-executor }
thread-pool-executor = {
keep-alive-time = 60s
core-pool-size-min = 8
core-pool-size-factor = 3.0
core-pool-size-max = 64
max-pool-size-min = 8
max-pool-size-factor = 3.0
max-pool-size-max = 64
task-queue-size = -1
task-queue-type = linked
allow-core-timeout = on
}
}
In the akka docs introduction it highlights “Millions of actors can be efficiently scheduled on a dozen of threads”. So in that case why would we need to create threads proportional to number of concurrent requests
Are you blocking in your calls? Eg, are you calling Thread.sleep? Or using some synchronous IO? If so, then what you're seeing is entirely expected.
Lagom is an asynchronous framework. All the IO and inter-service communication mechanisms it provides are non blocking. Its thread pools a tuned for non blocking. If you only using non blocking calls, you will see the thread pools behave with very low thread counts, and you won't find things going unresponsive.
But the moment you start blocking, all bets are off. Blocking requires one thread per request.
The default dispatcher that Akka uses is a fork join pool. It is designed for asynchronous use. If you block in a thread in its pool, it will start another thread to ensure other tasks can continue. So, that's why you see the thread pool grow. Don't block, and this won't happen.
The thread pool executor on the other hand uses a fixed number of threads. If you block on this, then you risk deadlocking the entire application. Don't block, and this won't happen.
I have been working on Akka Actor model. I have an usecase where more than 1000 actors will be in active and I have to process those actors. I thought of controlling the thread count through configuration defined in the application.conf.
But no. of dispatcher thread created in my application makes me helpless in tuning the dispatcher configuration. Each time when I restart my application, I see different number of dispatcher threads created (I have checked this via Thread dump each time after starting the application).
Even thread count is not equal to the one which I defined in parallelism-min. Due to this low thread count, my application is processing very slowly.
On checking the no. of core in my machine through the below code:
Runtime.getRuntime().availableProcessors();
It displays 40. But the no. of dispatcher thread count created is less than 300 even I configured parallelism as 500.
Following is my application.conf file:
consumer-dispatcher {
type = "Dispatcher"
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 500
parallelism-factor = 20.0
parallelism-max = 1000
}
shutdown-timeout = 1s
throughput = 1
}
May I know on what basis akka will be creating dispatcher threads internally and how I can increase my dispatcher thread count to increase parallel processing of actors?
X-Post from discuss.lightbend.com
First let me answer the question directly.
A fork-join-executor will be backed by a java.util.concurrent.forkJoinPool pool with its parallelism set to the implied parallelism from the dispatcher config. (parallelism-factor * processors, but no larger than max or less than min). So, in your case, 800.
And while I’m no expert on the implementation of the ForkJoinPool the source for the Java implementation of ForkJoinPool says “All worker thread creation is on-demand, triggered by task submissions, replacement of terminated workers, and/or compensation for blocked workers.” and it has methods like getActiveThreads(), so it’s clear that ForkJoinPooldoesn’t just naively create a giant pool of workers.
In other words, what you are seeing is expected: it’s only going to create threads as they are needed. If you really must have a gigantic pool of worker threads you could create a thread-pool-executor with a fixed-pool-size of 800. This would give you the implementation you are looking for.
But, before you do so, I think you are entirely missing the point of actors and Akka. One of the reasons that people like actors is that they are much more lightweight than threads and can give you a lot more concurrency than a thread. (Also note that concurrency != parallelism as noted in the documentation on concepts.) So trying to create a pool of 800 threads to back 1000 actors is very wasteful. In the akka docs introduction it highlights "Millions of actors can be efficiently scheduled on a dozen of threads".
I can’t tell you exactly how many threads you need without knowing your application (for example if you have blocking behavior) but the defaults (which would give you a parallelism factor of 20) is probably just fine. Benchmark to be certain, but I really don’t think you have a problem with too few threads. (The ForkJoinPool behavior you are observing seems to confirm this.)
I'm new to the Akka framework and I'm building a group chat application with it. My application may have 10 million actor instances (an actor instance for each group chat) of the same type, only 5% of which are highly active, and 60% of which can be idle (not receiving any messages) for days.
I want to know:
Is there any best practice to identify these idle actors?
What is the best practice to deal with them? Is stopping them enough?
Is there any best practice to identify these idle actors?
An actor's ActorContext has a setReceiveTimeout method that defines an inactivity threshold for the actor: if the actor hasn't received a message in the given amount of time, then an akka.actor.ReceiveTimeout message is sent to the actor. For example:
import akka.actor.{ Actor, ReceiveTimeout }
import scala.concurrent.duration._
class ChatActor extends Actor {
context.setReceiveTimeout(2 hours)
def receive = {
case ReceiveTimeout =>
// do something
// other case clauses
}
}
The above ChatActor will receive a ReceiveTimeout message if it hasn't received a message for two hours. (However, as the documentation states: "the receive timeout might fire and enqueue the ReceiveTimeout message right after another message was enqueued; hence it is not guaranteed that upon reception of the receive timeout there must have been an idle period beforehand as configured via this method.")
What is the best practice to deal with them?
It's a good idea to stop inactive actors; otherwise you could have a memory leak. Here are a few approaches for stopping these actors:
The inactive actor throws an exception, which is handled in a supervisor strategy defined in the actor's parent. In the supervisor strategy, the parent stops the idle actor (e.g., via context stop sender()).
The inactive actor sends its self reference to a "reaper" actor that collects references to idle actors and culls (i.e., stops) these actors on a periodic basis (perhaps using a scheduler).
The inactive actor stops itself (via context stop self).
More information about stopping actors is found here.
Is stopping them enough?
When an actor is stopped, its ActorRef essentially becomes invalid. From the documentation:
After stopping an actor, its postStop hook is called, which may be used e.g. for deregistering this actor from other services. This hook is guaranteed to run after message queuing has been disabled for this actor, i.e. messages sent to a stopped actor will be redirected to the deadLetters of the ActorSystem.
At this point, the underlying actor instance to which the now-stale ActorRef points is eligible for garbage collection. In other words, an actor must be stopped in order for it to be eligible for garbage collection. Therefore, in regard to freeing up memory, stopping the actor is enough. You could also remove the invalid ActorRef itself after the actor has been stopped. Note that removing an ActorRef does not automatically stop the actor:
It is important to note that Actors do not stop automatically when no longer referenced, every Actor that is created must also explicitly be destroyed.
Is there any best practice to identify these idle actors?
The only way is to make each actor to keep the time when it was active last time. Then, to speedup investigation of the longest inactive actor, you can organize an index-like structure, e.g. PriorityQueue. Then a dedicated actor periodically awakes and cleans that structure from actors which are idle longer than some predefined period of time.
What is the best practice to deal with them? Is stopping them enough?
An idle actor does not consume any resources except core memory. If you have plenty of memory, the best practice is to do nothing. If you want to save that memory, store actor in database (after some period of inactivity), and then read it from there by demand.
I am using Spring Integration with ActiveMQ. I defined a DefaultMessageListenerContainer with maxConcurrentConsumers = 5. It is referenced in a . After an int-xml:validating-filter and an int-xml:unmarshalling-transformer, I defined a queue channel actionInstructionTransformed. And I have got a poller for this queue channel. When I start my application, in the ActiveMQ console, I can see that a connection is created and inside five sessions.
Now, I have got a #MessageEndpoint with a method annotated
#ServiceActivator(inputChannel = "actionInstructionTransformed", poller = #Poller(value = "customPoller")).
I have got a log statement at the method entrance. Processing of each message is long (several minutes). In my logs, I can see that thread-1 starts the processing and then I can only see thread-1 outputs. Only when thread-1 has finished processing 1 message, I can see thread-2 starts processing the next message, etc. I do NOT have any synchronized block inside my class annotated #MessageEndpoint. I have not managed to get thread-1, thread-2, etc process messages concurrently.
Has anybody experienced something similar?
Look, you say:
After an int-xml:validating-filter and an int-xml:unmarshalling-transformer, I defined a queue channel actionInstructionTransformed.
Now let's go to the QueueChannel and PollingConsumer definitions!
On the other hand, a channel adapter connected to a channel that implements the org.springframework.messaging.PollableChannel interface (e.g. a QueueChannel) will produce an instance of PollingConsumer.
And pay attention that #Poller (PollerMetadata) has taskExecutor option.
By default the TaskScedhuler ask QueueChannel for data periodically according to the trigger configuration. If that is PeriodicTrigger with default options like fixedRate = false, the next poll really happens after the previous one. That's why you see only one Thread.
So, try to configure taskExecutor and your messages from that queue will go in parallel.
The concurrency on the DefaultMessageListenerContainer does not have effect. Because in the end you place all those messages to the QueueChannel. And here a new Threading model starts to work based on the #Poller configuration.
I am working on a multithreaded middleware enviornment. The framework is basically a capturing and streaming framework. So it involves a number of threads.
To give you all a brief idea of the threading architecture:
There are seprate threads for demultiplexer, receiveVideo, DecodeVideo, DisplayVideo etc. Each thread performs its functionlity, for eg:
demultiplexer extracts audio, video packets
receivevideo receives header + payload of video packet & removes payload
DecodeVideo receives payload & decodes payload packet
DisplayVideo receives decoded packets & displays the decoded packets on display
Thus each thread feeds the extracted data to the next thread. The threads share data buffers amongst them and the buffers are synchronised through use of mutexes and semaphores. Similarly, there are other threads for handling ananlogvideo and analogaudio etc.
All the threads are spawned in during initialization but they remain blocked on a semaphore and depending upon the input(analog/digitial) selective semaphores are signalled so that specifc threads get unblocked & move on to do their work. At various stages each thread calls some lower level(driver calls)to get data or write data etc. These calls are blocking and the errors resulting from these calls(driver returning corrupted data, driver stalling) should be handled but are not being handled currently.
I wanted to implement a thread monitoring mechanism where a thread will monitor these worker threads and if an error condition occurs will take some preventive actions. As I understand certain such mechanisms are commonly used like Watchdogs in UI or MMI applications. I am trying to look for something similar.
I am using pthreads and No Boost or STL(its a legacy code, pretty much procedural C++)
Any ideas about specific framework or design patterns or open source projects which do something similar and might help in with ideas for implementing my requirement?
Can you ping the threads - periodically send each one a message on its usual input queue, interleaved with all the other normal stuff, asking it to return its status? When each handler thread gets the message, it loads the message with status stuff - how many messages its processed since the last ping, length of its input/output queue, last time that its driver returned OK, that sort of stats - and queues it back to your Thread Monitoring Mechanism. Your TMM would have to time out the replies in case some thread/s is/are stuck.
You could, maybe, just post one message down the whole chain, each thread adding its own status in different fields. That would mean only one timeout, after which your TMM would have to examine the message to see how far down the chain it got.
There are other things - I like to keep an on-screen dump, on a 1s timer, of the length of queues and depth of buffer pools. If something stuffs, I can usually tell roughly where it is, (eg. a pool is emptying and some queue is growing - the queue comsumer is wasted).
Rgds,
Martin
What about using a signalling system to wake up your monitoring thread when something's gone awry in one of your worker threads. You can emulate the signalling with an ResetEvent of some type.
When an exception occurs in your worker thread, you have some data structure you fill up with the data about the exception and then you can pass that on to your monitoring thread. You wake up the monitoring thread by using the event.
Then the monitoring thread can do what you need it to do.
I'm guessing you don't wish to have your monitoring thread active unless something has gone wrong, right?