I have a testing in WSO2 ESB dbreport mediator.
When I send a "BIG message" (100 or 1,000 or 10,000 ... 500,000 rows) from Database A to WSO2 ESB.
And WSO2 ESB split the message as rows by Iterate mediators,
then use the DBReport mediator write row by row into Database B (by Data source pool).
When write 100 rows, it spent 5 seconds,
when write 1,000 rows, it spent 188 seconds,
and then write 10,000 rows, it need spent 19163 seconds.
How efficient use of DBReport mediators?
thanks.
DBReport mediator is executed synchronously in WSO2 ESB which means the thread that execute this DBReport mediator is stucked until the Database operation is completed. That means the performance is lower than when the execution happens asynchronously.
Therefore to gain the maximum performance, please use WSO2 Data Services server and use DSS to execute the insert Database operation. That way you can gain the maximum performance.
Related
I want to know about how request processing happens in Wso2 EI 7.1.0, because I am seeing different threads like synapse.threads.core, worker_pool_size_core, snd_t_core, lst_t_core in thread dump.
In my use case I created flow which contains API mediator-->Iterate mediator-->Send mediator--->Aggregate--->Respond. So here request will convert as soap to json while sending it to endpoint and response will convert as json to soap while respond it to client, I am using payload factory mediator for request /response format conversion and I observed more time taking for response conversion, So is their any configuration change required as of now I am using default configurations.
Thanks,
Ajay Babu Maguluri.
PassThroughMessageProcessor threads are used to mediate the message through the mediators. By default, this thread pool has 400 threads and you can tune the pool using the worker_pool_size_core, worker_pool_size_max parameters.
But if you are using clone or iterate mediators inside your mediation, SynapseWorker threads are used to handle the targets of those mediators. By default, this thread pool has 20 threads. You can tune the thread pool using synapse.threads.core and synapse.threads.max parameters
For the scenario
As a user, whenever I try to generate or fetch codes, :
If, while generating codes via PUT callout, the request fails, then the system should identify that the put callout has failed and should not do subsequent GET callout to the codes which were not even created in the first place.
If, while generating codes via PUT callout, the request is successful, the system should wait for a while (30 secs to 1 min) and should not poll the Service API very frequently.
I have written a code thats call the PUT callout than after success of Put , calling the GET callout in future to retrieve the codes
Expected result is -
When PUT callout is sucess , system should wait for 30sec to 1 min to GET callout and retrieve all the data and store it in salesforce using scheduler and batch.
You can't schedule in Salesforce on a second-level cadence. The smallest allowable increment for a Schedulable job is fifteen minutes. Salesforce asynchronous jobs are always executed based on server load and are in a queue; you cannot control the time of their execution to the second.
While some approximation of this pattern could potentially be achieved using a Queueable chain, this pattern is not at all suited to the Salesforce architecture and really should be delegated to a middleware platform.
We are evaluating wso2 ei for a system where the expected system latency is 1,3 seg for 200 TPS (concurrent users).
In our first tests we have an API Proxy that invokes 3 web services, exist a json transformer with payloadfactory and class mediators.
We have performed the tuning recommended by WSO2 EI for production envirment.
The results indicate that we have Latency per transaction between 2 and 3 seconds (average) with 16 TPS, which is very distant from the expected. The times of the services (response time) are excellent.
What is the capacity of WSO2 EI in relation to the response time (Latency per transaction) vs TPS?.
Some recommendations to improve the response times vs TPS?
thank for youy support;
Running WSO2 EI 6.2.0
I have a simple use case (Sequence) for WSO2 EI ESB:
Extract some parameters from the original request
Call an Async REST API
Extract an Execution ID from the Async Call Payload
Poll Loop another Sync API to check Execution Status based on Execution ID
Halt polling when the Sync API says that the request is completed
Extract some parameters from the last Sync Call
Response
My problem lies on the Poll a Sync API until it returns some parameter saying that the previous Async execution is finnished.
Is there any WSO2 EI Sequence mediator for this sort of Poll Loop?
The ESB mediations (sequences) are not really intended to keep the state and wait for anything. I'd believe it is even intention not having any sort "do/while" loop. We had a project requiring many polling steps and we used a process server to do so. So - with pure mediation it is very difficult to accomplish what are you asking for. Even you may check this one http://bsenduran.blogspot.com/2017/08/while-loop-in-wso2-esb.html
I will propose a few things you could do:
write a custom polling mediator (I really do not advice to do so)
use a process server (requires additional no-so-lightweight server)
use messaging with message processor (send a message to a queue, a message processor will poll, call and send the back to the queue or to response)
In all cases - if a client is waiting for a synchronous response, you need to finish the polling before the client times out. IMHO the best option return a message to a client (we are working on it) and avoid polling if possible..
WSO2 ESB is not loading all proxies (more than 20). Then we increased following two values in the startup script and it worked:
-Dsnd_t_core=120
-Dsnd_t_max=600
But then we encountered several fatal issues of the WSO2 ESB. Several JMS proxies were blocked and did not consume anymore messages. The worst thing of all: NO ERROR in the carbon.log!
In addition the CPU load on the server went up to 100%.
A restart did not solve the problem, only deactivating scheduled tasks or proxies solved the problem.
We now discovered, that a VFS proxy is creating exactly 120 threads (JConsole). With each transport.PollInterval it creates a new Thread.
Which values do you use for the -Dsnd_t_core and max?
Why is a VFS proxy creating a new thread (see jconsole) a t each PollInterval?
As long as I know, WSO2 ESB thread is based on java.util.concurrent ThreadPool.
In this link you can read about some ThreadPool characteristics like when will it create a new thread, the queue mechanism, and reject task policy.
Which values do you use for the -Dsnd_t_core and max?
-Dsnd_t_core is the minimum number of threads inside a ThreadPool. So WSO2 ESB will automatically create thread as much as you set the -Dsnd_t_core. The default value is 20. WSO2 ESB will create 20 vfs-worker if you don't specify the -Dsnd_t_core.
-Dsnd_t_max is the maximum number of threads inside a ThreadPool. WSO2 ESB will stop create a new thread if the maximum number is reached.
Why is a VFS proxy creating a new thread (see jconsole) a t each PollInterval?
WSO2 ESB will create a new thread in these conditions :
If fewer than corePoolSize threads are running, the Executor always prefers adding a new thread rather than queuing.
If corePoolSize or more threads are running, the Executor always prefers queuing a request rather than adding a new thread.
If a request cannot be queued, a new thread is created unless this would exceed maximumPoolSize, in which case, the task will be
rejected.
So, as long as your queue is full and the maximum number of threads is not reached, WSO2 will create a new thread to handle a process. The PollInterval is set to specify the delay before your service start to poll the message or file from the source folder.
You can set the maxQueue number to unbound (-1) so the queue will never full and new thread will never be created.
I also found something from the JConsole. 1 service/1 proxy service will be handled only by 1 thread. I still trying to figure this out and make 1 service/1 proxy service is handled by 2 or more thread (multithread).
Hope this will help answer your question :)