WSO2 ESB is not loading all proxies (more than 20). Then we increased following two values in the startup script and it worked:
-Dsnd_t_core=120
-Dsnd_t_max=600
But then we encountered several fatal issues of the WSO2 ESB. Several JMS proxies were blocked and did not consume anymore messages. The worst thing of all: NO ERROR in the carbon.log!
In addition the CPU load on the server went up to 100%.
A restart did not solve the problem, only deactivating scheduled tasks or proxies solved the problem.
We now discovered, that a VFS proxy is creating exactly 120 threads (JConsole). With each transport.PollInterval it creates a new Thread.
Which values do you use for the -Dsnd_t_core and max?
Why is a VFS proxy creating a new thread (see jconsole) a t each PollInterval?
As long as I know, WSO2 ESB thread is based on java.util.concurrent ThreadPool.
In this link you can read about some ThreadPool characteristics like when will it create a new thread, the queue mechanism, and reject task policy.
Which values do you use for the -Dsnd_t_core and max?
-Dsnd_t_core is the minimum number of threads inside a ThreadPool. So WSO2 ESB will automatically create thread as much as you set the -Dsnd_t_core. The default value is 20. WSO2 ESB will create 20 vfs-worker if you don't specify the -Dsnd_t_core.
-Dsnd_t_max is the maximum number of threads inside a ThreadPool. WSO2 ESB will stop create a new thread if the maximum number is reached.
Why is a VFS proxy creating a new thread (see jconsole) a t each PollInterval?
WSO2 ESB will create a new thread in these conditions :
If fewer than corePoolSize threads are running, the Executor always prefers adding a new thread rather than queuing.
If corePoolSize or more threads are running, the Executor always prefers queuing a request rather than adding a new thread.
If a request cannot be queued, a new thread is created unless this would exceed maximumPoolSize, in which case, the task will be
rejected.
So, as long as your queue is full and the maximum number of threads is not reached, WSO2 will create a new thread to handle a process. The PollInterval is set to specify the delay before your service start to poll the message or file from the source folder.
You can set the maxQueue number to unbound (-1) so the queue will never full and new thread will never be created.
I also found something from the JConsole. 1 service/1 proxy service will be handled only by 1 thread. I still trying to figure this out and make 1 service/1 proxy service is handled by 2 or more thread (multithread).
Hope this will help answer your question :)
Related
We have a Reactive REST API using Spring Data Neo4j (SpringBoot v2.7.5) deployed to Kubernetes. When running a stress test to determine the breaking point, once the volume of requests that the service can handle has been breached, the service does not auto-recover, even after the load has dropped to a level at which the service can handle.
After the service has fallen over the Neo4J health indicator shows:
“org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms”
With respect to connection/configuration settings we are using defaults configured by SDN.
Observations:
Up until the point at which the service breaks only a small number of connections are utilised, at the point at which it breaks the connections in use jumps up to the max pool size and the above mentioned error is observed. No matter how much time passes (even well beyond the max connection lifetime) the service is unable to acquire a connection from the pool. Upon manually shutting down and restarting the service/pod the service returns to a healthy state.
As an interim solution we now check the Neo4J health indicator, if the mentioned error is present the liveness state is set to down which triggers Kubernetes to restart the service automatically. However, I’m wondering if there is an underlying issue with the connections in the pool not getting ‘cleaned up’?
You can take a look at this discussion https://github.com/spring-projects/spring-data-neo4j/issues/2632
I had the same issue. The problem is that either Spring Framework or Neo4j reactive transaction manager doesn't close connections in a complex reactive flow e.g. when there are a lot of inner calls/mappings and somewhere inside an exception is thrown.
So as a workaround you can add #Transactional in such places to avoid multiple transactions to be created.
In wso2 ESB 5.0.0, we are playing around with the passthru-http.properties as explained here.
I would like to know if there is a way to get the value of worker_pool_queue_length property in a running ESB. Well, not the value setted in the file, but the actual value of elements stored in the queue any moment. As we are not using the default -1 value. We are exploring using a bound queue.
From here, I have been trying with the following beans from the jmx
org.apache.synapse/Transport/passthru-http-receiver
org.apache.synapse/Transport/passthru-http-sender
And this attribute,
QueueSize, number of messages currently queued. Messages get queued if
all the worker threads in this transport thread pool are busy.
But its value is always 0. And I know for sure it should be higher. As we are firing messages to the ESB using Jmeter. And new worker threads are being created, so there are elements in the queue. No matter how hard we hit the esb, we can see all threads working, connections rejected ... But always a 0.
Also, not lucky with turning into DEBUG the log4.
Any ideas, how could I get the number of queued messages?
WSO2 EI uses the java ThreadPoolExecutor implementation [1] for spawning new threads for processing requests also it handles queuing of messages if sufficient amount of threads are not available. Therefore it seems that you will not be able to get the number of queued messages through the debug logs.
If you are interested in analyzing the number of queued messages what I can suggest is taking a heap dump and analyzing it. It will indicate the amount of queued messages.
[1]-http://tutorials.jenkov.com/java-util-concurrent/threadpoolexecutor.html
I have a Camunda delegate which is supposed to send mails. We want to restrict the parallel executions of tasks so that the mail server is not flooded with too many messages.
Is there any (easy) possibility to restrict that task only to, say, 25 concurrent executions without resorting to Camunda-external synchronization (which is possible but complicated)?
If you make the service task async in the modeler, the camunda jobexecutor will execute the delegate with a managed thread pool, so you will have a maximum of the definied executor threads sending mails. Could already be a solution for your problem.
I want to create a web application were a client calls a REST Webservice. This returns OK-Status for the client (with a link to the result) and creates a new message on an activeMQ Queue. On the listeners side of the activeMQ there should be worker who process the messages.
Iam stucking here with my concept, because i dont really know how to determine the number of workers i need. The workers only have to call web service interfaces, so no high computation power is needed for the worker itself. The most time the worker has to wait for returning results from the called webservice. But one worker can not handle all requests, so if a limit of requests in the queue is exceeded (i dont know the limit yet), another worker should treat the queue.
What is the best practise for doing this job? Should i create one worker per Request and destroying them if the work is done? How to dynamically create workers based on the queue size? Is it better to run these workers all the time or creating them when the queue requiere that?
I think a Topic/Suscriber architecture is not reasonable, because only one worker should care about one request. Lets imagine of 100 Requests per Minute average and 500 requests on high workload.
My intention is to get results fast, so no client have to wait for it answer just because not properly used ressources ...
Thank you
Why don't you figure out the max number of workers you'd realistically be able to support, and then make that number and leave them running forever? I'd use a prefetch of either 0 or 1, to avoid piling up a bunch of messages in one worker's prefetch buffer while the others sit idle. (Prefetch=0 will pull the next message when the current one is finished, whereas prefetch=1 will have a single message sitting "on deck" available to be processed without needing to get it from the network but it means that a consumer might be available to consume a message but can't because it's sitting in another consumer's prefetch buffer waiting for that consumer to be read for it). I'd use prefetch=0 as long as the time to download your messages from the broker isn't unreasonable, since it will spread the workload as evenly as possible.
Then whenever there are messages to be processed, either a worker available to process the next message (so no delay) or all the workers are processing messages (so of course you're going to have to wait because you're at capacity, but as soon as there's a worker available it will take the next message from the queue).
Also, you're right that you want queues (where a message will be consumed by only a single worker) not topics (where a message will be consumed by each worker).
I have simple bpl process with 3 invokes in loop. One of instances hang without any visible reason. So process is in active state but it is not executing any longer. Last logged activity is call to invoke. I search database and find out that both request and response are present in table ode_message and they looks correctly. But output variable from invoke in table ode_xml_data is not filled. There is no logs in bps from time when message arrived. Is any way to find out what happen wrong?
I'm try to use Wso2 BPS 2.1.2