I have a CEP flow that has a high throughput with 100+ messages per second.
And I am publishing the result of my processing into a JMS publisher with the following configuration:
Output Event Adapter Type* : JMS
JNDI Initial Context Factory Class: org.apache.activemq.jndi.ActiveMQInitialContextFactory
JNDI Provider URL *: tcp://localhost:61616
Connection Factory JNDI Name *: QueueConnectionFactory
Destination Type *: Queue
Destination *: myqueue
also, in order to try if the problem was not having concurrency i added:
Concurrent Publishers: Allow
to the JMSPublisher
and I am getting the following error:
ERROR {org.wso2.carbon.event.output.adapter.jms.JMSEventAdapter} -
Event dropped at Output Adapter 'kpis' for tenant id '-1234',
Job queue is full, Task java.util.concurrent.FutureTask#5651dd6c
rejected from java.util.concurrent.ThreadPoolExecutor#3c8c7b29
[Running, pool size =1, active threads = 1, queued tasks = 10000,
completed tasks = 176986]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask#5651dd6c rejected from
java.util.concurrent.ThreadPoolExecutor#3c8c7b29[Running, pool size = 1,
active threads = 1, queued tasks = 10000, completed tasks = 176986]
Is there any limitation on the throughput to a JMS activemq?
Also, so far there is no consumer on the queue I am writing all the messages to. Can that have a negative impact on the WSO2 CEP publisher causing that error and degrading performance?
From reading some info online it looks like this might be a direct problem with the pool size!
Is it possible to increase the JMSEventAdapter pool size? if yes, then how?
FULL STACK TRACE:
ERROR {org.wso2.carbon.event.output.ad apter.jms.JMSEventAdapter} - Event dropped at Output Adapter 'kpis' for tenant id '-1234', Job queue is full, Task java.util.concurrent.FutureTask#745cb718 rejected from java.util.concurrent.ThreadPoolExecutor#3a7d9bcf[Running, pool size = 100, active threads = 100, queued tasks = 10000, completed tasks = 5151]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask#745cb718 rejected from java.util.concurrent.ThreadPoolExecutor#3a7d9bcf[Running, pool size = 100, active threads = 100, queued tasks = 10000, completed tasks = 5151]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at org.wso2.carbon.event.output.adapter.jms.JMSEventAdapter.publish(JMSEventAdapter.java:142)
at org.wso2.carbon.event.output.adapter.core.internal.OutputAdapterRuntime.publish(OutputAdapterRuntime.java:62)
at org.wso2.carbon.event.output.adapter.core.internal.CarbonOutputEventAdapterService.publish(CarbonOutputEventAdapterService.java:143)
at org.wso2.carbon.event.publisher.core.internal.EventPublisher.process(EventPublisher.java:414)
at org.wso2.carbon.event.publisher.core.internal.EventPublisher.sendEvent(EventPublisher.java:226)
at org.wso2.carbon.event.publisher.core.internal.EventPublisher.onEvent(EventPublisher.java:294)
at org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvents(EventJunction.java:194)
at org.wso2.carbon.event.processor.core.internal.listener.SiddhiOutputStreamListener.receive(SiddhiOutputStreamListener.java:100)
at org.wso2.siddhi.core.stream.output.StreamCallback.receiveEvents(StreamCallback.java:98)
at org.wso2.siddhi.core.stream.output.StreamCallback.receive(StreamCallback.java:69)
at org.wso2.siddhi.core.stream.StreamJunction.sendEvent(StreamJunction.java:126)
at org.wso2.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:323)
at org.wso2.siddhi.core.query.output.callback.InsertIntoStreamCallback.send(InsertIntoStreamCallback.java:46)
at org.wso2.siddhi.core.query.output.ratelimit.OutputRateLimiter.sendToCallBacks(OutputRateLimiter.java:78)
at org.wso2.siddhi.core.query.output.ratelimit.PassThroughOutputRateLimiter.process(PassThroughOutputRateLimiter.java:40)
at org.wso2.siddhi.core.query.selector.QuerySelector.processNoGroupBy(QuerySelector.java:123)
at org.wso2.siddhi.core.query.selector.QuerySelector.process(QuerySelector.java:86)
at org.wso2.siddhi.core.query.processor.filter.FilterProcessor.process(FilterProcessor.java:56)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.processAndClear(ProcessStreamReceiver.java:154)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.process(ProcessStreamReceiver.java:80)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.receive(ProcessStreamReceiver.java:150)
at org.wso2.siddhi.core.stream.StreamJunction.sendData(StreamJunction.java:214)
at org.wso2.siddhi.core.stream.StreamJunction.access$200(StreamJunction.java:46)
at org.wso2.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:343)
at org.wso2.siddhi.core.stream.input.InputDistributor.send(InputDistributor.java:49)
at org.wso2.siddhi.core.stream.input.InputEntryValve.send(InputEntryValve.java:59)
at org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:51)
at org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39)
at org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:104)
at org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvents(EventJunction.java:183)
at org.wso2.carbon.event.processor.core.internal.listener.SiddhiOutputStreamListener.receive(SiddhiOutputStreamListener.java:100)
at org.wso2.siddhi.core.stream.output.StreamCallback.receiveEvents(StreamCallback.java:98)
at org.wso2.siddhi.core.stream.output.StreamCallback.receive(StreamCallback.java:69)
at org.wso2.siddhi.core.stream.StreamJunction.sendEvent(StreamJunction.java:126)
at org.wso2.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:323)
at org.wso2.siddhi.core.query.output.callback.InsertIntoStreamCallback.send(InsertIntoStreamCallback.java:46)
at org.wso2.siddhi.core.query.output.ratelimit.OutputRateLimiter.sendToCallBacks(OutputRateLimiter.java:78)
at org.wso2.siddhi.core.query.output.ratelimit.PassThroughOutputRateLimiter.process(PassThroughOutputRateLimiter.java:40)
at org.wso2.siddhi.core.query.selector.QuerySelector.processNoGroupBy(QuerySelector.java:123)
at org.wso2.siddhi.core.query.selector.QuerySelector.process(QuerySelector.java:86)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.processAndClear(ProcessStreamReceiver.java:154)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.process(ProcessStreamReceiver.java:80)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.receive(ProcessStreamReceiver.java:102)
at org.wso2.siddhi.core.stream.StreamJunction.sendEvent(StreamJunction.java:126)
at org.wso2.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:323)
at org.wso2.siddhi.core.query.output.callback.InsertIntoStreamCallback.send(InsertIntoStreamCallback.java:46)
at org.wso2.siddhi.core.query.output.ratelimit.OutputRateLimiter.sendToCallBacks(OutputRateLimiter.java:78)
at org.wso2.siddhi.core.query.output.ratelimit.PassThroughOutputRateLimiter.process(PassThroughOutputRateLimiter.java:40)
at org.wso2.siddhi.core.query.selector.QuerySelector.processNoGroupBy(QuerySelector.java:123)
at org.wso2.siddhi.core.query.selector.QuerySelector.process(QuerySelector.java:86)
at org.wso2.siddhi.core.query.input.stream.join.JoinProcessor.process(JoinProcessor.java:110)
at org.wso2.siddhi.core.query.processor.stream.window.LengthWindowProcessor.process(LengthWindowProcessor.java:86)
at org.wso2.siddhi.core.query.processor.stream.window.WindowProcessor.processEventChunk(WindowProcessor.java:57)
at org.wso2.siddhi.core.query.processor.stream.AbstractStreamProcessor.process(AbstractStreamProcessor.java:101)
at org.wso2.siddhi.core.query.input.stream.join.JoinProcessor.process(JoinProcessor.java:118)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.processAndClear(ProcessStreamReceiver.java:154)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.process(ProcessStreamReceiver.java:80)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.receive(ProcessStreamReceiver.java:102)
at org.wso2.siddhi.core.stream.StreamJunction.sendEvent(StreamJunction.java:126)
at org.wso2.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:323)
at org.wso2.siddhi.core.query.output.callback.InsertIntoStreamCallback.send(InsertIntoStreamCallback.java:46)
at org.wso2.siddhi.core.query.output.ratelimit.OutputRateLimiter.sendToCallBacks(OutputRateLimiter.java:78)
at org.wso2.siddhi.core.query.output.ratelimit.PassThroughOutputRateLimiter.process(PassThroughOutputRateLimiter.java:40)
at org.wso2.siddhi.core.query.selector.QuerySelector.processNoGroupBy(QuerySelector.java:123)
at org.wso2.siddhi.core.query.selector.QuerySelector.process(QuerySelector.java:86)
at org.wso2.siddhi.core.query.processor.filter.FilterProcessor.process(FilterProcessor.java:56)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.processAndClear(ProcessStreamReceiver.java:154)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.process(ProcessStreamReceiver.java:80)
at org.wso2.siddhi.core.query.input.ProcessStreamReceiver.receive(ProcessStreamReceiver.java:150)
at org.wso2.siddhi.core.stream.StreamJunction.sendData(StreamJunction.java:214)
at org.wso2.siddhi.core.stream.StreamJunction.access$200(StreamJunction.java:46)
at org.wso2.siddhi.core.stream.StreamJunction$Publisher.send(StreamJunction.java:343)
at org.wso2.siddhi.core.stream.input.InputDistributor.send(InputDistributor.java:49)
at org.wso2.siddhi.core.stream.input.InputEntryValve.send(InputEntryValve.java:59)
at org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:51)
at org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39)
at org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:104)
at org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:146)
at org.wso2.carbon.event.receiver.core.internal.management.InputEventDispatcher.onEvent(InputEventDispatcher.java:27)
at org.wso2.carbon.event.receiver.core.internal.EventReceiver.sendEvent(EventReceiver.java:298)
at org.wso2.carbon.event.receiver.core.internal.EventReceiver.processMappedEvent(EventReceiver.java:222)
at org.wso2.carbon.event.receiver.core.internal.EventReceiver$MappedEventSubscription.onEvent(EventReceiver.java:355)
at org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime.onEvent(InputAdapterRuntime.java:110)
at org.wso2.carbon.event.input.adapter.jms.internal.util.JMSMessageListener.onMessage(JMSMessageListener.java:61)
at org.wso2.carbon.event.input.adapter.jms.internal.util.JMSTaskManager$MessageListenerTask.handleMessage(JMSTaskManager.java:643)
at org.wso2.carbon.event.input.adapter.jms.internal.util.JMSTaskManager$MessageListenerTask.run(JMSTaskManager.java:542)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Due to the high throughput of the execution plans to the publishers and the async mechanism that closes JMS connections, Active MQ JMS connection pool inside the WSO2 CEP engine is unable to keep up with the opening and closure of those connections.
This rapidly exhausts all the available connections. Independently of the maximum # set.
The solution in my case goes through reducing the number of messages sent per unit of time and accumulating results.
Related
In production we are using jetty 9.2.9.v20150224 with the following configuration:
new Server(new QueuedThreadPool(200, 5, 30000, new ArrayBlockingQueue<Runnable>(128)));
maxThreads = 200, minThreads = 5, idleTimout = 30000 ms
I tested our application and it was able to handle 200 requests/sec, but few of our clients complain that sometime even with very less load jetty does not accept any new connection and from the log that's what I found STARTED,5<=18<=200,i=4,q=128. Which as far as I understand shows there are 18 live threads in the thread pool which is way less then 200, but somehow the new connection is still rejected. Following is the snipped from the log:
2021-01-08 00:20:47,813 WARN [qtp1720926658-14444] QueuedThreadPool : QueuedThreadPool[qtp1720926658]#669341c2{STARTED,5<=18<=200,i=4,q=128}[ReservedThreadExecutor#58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint#263c181
2021-01-08 00:20:47,813 WARN [qtp1720926658-14444] QueuedThreadPool : QueuedThreadPool[qtp1720926658]#669341c2{STARTED,5<=18<=200,i=3,q=128}[ReservedThreadExecutor#58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint#431081c5
2021-01-08 00:20:47,813 WARN [qtp1720926658-14445] QueuedThreadPool : QueuedThreadPool[qtp1720926658]#669341c2{STARTED,5<=18<=200,i=2,q=128}[ReservedThreadExecutor#58a5954d{s=3/4,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint#3866d8cd
2021-01-08 00:20:47,813 WARN [qtp1720926658-14442] QueuedThreadPool : QueuedThreadPool[qtp1720926658]#669341c2{STARTED,5<=18<=200,i=6,q=126}[ReservedThreadExecutor#58a5954d{s=1/4,p=2}] rejected CEP:SocketChannelEndPoint#4aa81d65{/10.0.1.29:53143<->/10.0.1.28:7777,OPEN,fill=FI,flush=-,to=16442/30000}{io=1/0,kio=1,kro=1}->HttpConnection#32cdb0b1[p=HttpParser{s=START,0 of -1},g=HttpGenerator#22c521c0{s=START}]=>HttpChannelOverHttp#e02e50f{r=1,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
2021-01-08 00:20:47,829 WARN [qtp1720926658-14442] EatWhatYouKill :
java.util.concurrent.RejectedExecutionException: CEP:SocketChannelEndPoint#4aa81d65{/10.0.1.29:53143<->/10.0.1.28:7777,OPEN,fill=FI,flush=-,to=16442/30000}{io=1/0,kio=1,kro=1}->HttpConnection#32cdb0b1[p=HttpParser{s=START,0 of -1},g=HttpGenerator#22c521c0{s=START}]=>HttpChannelOverHttp#e02e50f{r=1,c=false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:440)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.execute(EatWhatYouKill.java:370)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:305)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
STARTED,5<=18<=200,i=4,q=128 says ...
5 is your configured minimum threads
<=18<= is your remaining threads available in the pool
200 is your configured max threads
i=4 you have 4 idle threads in the pool (eligible for cleanup/removal)
q=128 you have 128 tasks/jobs in the queue (this value is too high, it hints at your QTP configuration being insufficient for your load)
Keep in mind that when you experience the rejected tasks/jobs it can be the queue or the pool that rejects them, we don't know which one in Jetty 9.2.x (this information is now available in the logs on Jetty 9.4.x)
This limited view of the QTP in the Jetty 9.2.x series does not show you even half of what's going on internally.
Example: in Jetty 9.4.x+ you see ...
Configured Minimum
Configured Maximum
idle thread count
active thread count
busy thread count
inactive thread count
reserved thread count
leased thread count
utilized count = threads - idle - reserved - leased
max threads possible = inactive + idle + reserved + leased + utilized
threads total = idle + reserved + leased + utilized
busy threads = leased + utilized
ready threads = idle + reserved
available = max - busy
maxAvailable = max - leased
utilization = utilized / maxAvailable (0.0 means idle, 1.0 means at capacity)
All of which are exposed in JMX as well.
have this error in AmazonDynamoDBLockClient. Im using org.springframework.cloud:spring-cloud-stream-binder-kinesis:2.0.1.RELEASE
#Spring Cloud Stream Kinesis Binder properties
spring.cloud.stream.bindings.cdcInput.group=listener
spring.cloud.stream.bindings.cdcInput.destination=my_stream
spring.cloud.stream.bindings.cdcInput.content-type=application/json
spring.cloud.stream.kinesis.binder.auto-create-stream=false
spring.cloud.stream.kinesis.binder.locks.table=LockRegistry
spring.cloud.stream.kinesis.binder.checkpoint.table=MetadataStore
spring.cloud.stream.kinesis.binder.locks.leaseDuration=10
spring.cloud.stream.kinesis.binder.locks.heartbeat-period=3
and here is my application.properties configs
com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient - Heartbeat thread recieved interrupt, exiting run() (possibly exiting thread)
java.lang.InterruptedException: sleep interrupted
java.lang.Thread.sleep(Native Method)
com.amazonaws.services.dynamodbv2.AmazonDynamoDBLockClient.run(AmazonDynamoDBLockClient.java:1184)
java.lang.Thread.run(Thread.java:748)
[SpringContextShutdownHook] INFO org.springframework.integration.aws.inbound.kinesis.KinesisMessa
geDrivenChannelAdapter - stopped KinesisMessageDrivenChannelAdapter{shardOffsets=[KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='binlog_updates', shard='shardId-000000000000', reset=false}], consumerGroup='cdc-listener'}
[-kinesis-shard-locks-1] ERROR org.springframework.integration.aws.inbound.kinesis.KinesisMessageD
rivenChannelAdapter - Error during unlocking: DynamoDbLock [lockKey=cdc-listener:rds_binlog_updates:shardId-000000000000,lockedAt=2021-01-21#15:54:36.735, lockItem=null]
org.springframework.dao.DataAccessResourceFailureException: Failed to release lock at cdc-listener:binlog_updates:shardI
d-000000000000; nested exception is java.util.concurrent.RejectedExecutionException: Task org.springframework.integration.aws.lock.DynamoDbLockRegistry$DynamoDbLock$ reject
ed from java.util.concurrent.ThreadPoolExecutor#[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
org.springframework.integration.aws.lock.DynamoDbLockRegistry$DynamoDbLock.unlock(DynamoDbLockRegistry.java:526)
org.springframework.integration.aws.inbound.kinesis.KinesisMessageDrivenChannelAdapter$ShardConsumerManager.run(KinesisMessageDrivenChannelAdapter.java:1294)
We are getting this error even if the stream is empty and no data is read by consumer. Service starts application context as regular without any errors, but in 1-2 minutes such an error appears and application falls down.
How can I change the timeout duration for different operations that can fail due to server inaccessibility? (start_session, insert, find, delete, update, ...)
...
auto pool = mongocxx::pool(mongocxx::uri("bad_uri"), pool_options);
auto connection = pool.try_acquire();
auto db = (*(connection.value()))["test_db"];
auto collection = db["test_collection"];
// This does not help
mongocxx::write_concern wc;
wc.timeout(std::chrono::milliseconds(1000));
mongocxx::options::insert insert_options;
insert_options.write_concern(wc);
// takes about 30 seconds to fail
collection.insert_one(from_json(R"({"name": "john doe", "occupation": "_redacted_", "skills" : "a certain set"})"), insert_options);
[Edit]
Here is the exception message:
C++ exception with description "No suitable servers found:
serverSelectionTimeoutMS expired: [connection timeout calling
ismaster on '127.0.0.1:27017']
It would be helpful to see the actual error message from the insert_one() operation, but "takes about 30 seconds to fail" suggests that this may be due to the default server selection timeout. You can configure that via the serverSelectionTimeoutMS connection string option.
If you are connecting to a replica set, I would suggest keeping that timeout a bit above the expected time for a failover to complete. Replica Set Elections states:
The median time before a cluster elects a new primary should not typically exceed 12 seconds
You may find that is shorter in practice. By keeping the server selection timeout above the expected failover time, you'll allow the driver to insulate your application from an error (at the expense of wait time).
If you are not connecting to a replica set, feel free to lower serverSelectionTimeoutMS to a lower value, albeit still greater than the expected latency to your mongod (standalone) or mongos (sharded cluster) node.
Do note that since server selection occurs within a loop, the connectTimeoutMS connection string option won't affect the delay you're seeing. Lower the connection timeout will allow the driver to internally give up when attempting to connect to an inaccessible server, but the server selection will still block for up to serverSelectionTimeoutMS (and likely retry connections to the server during that loop).
I am building a system where a Producer sends a list of tasks to be queued which will be consumed by a number of Consumers.
Assume I have a list of tasks and they can be categorised into Black, Orange and Yellow. All the Black tasks are sent to Queue_0, Orange to Queue_1 and Yellow to Queue_2. And I will assign a worker to each queue(i.e: Consumer_0 to Queue_0, Consumer_1 to Queue_1 and Consumer_2 to Queue_2). If Black lists get larger, I want to add an extra Consumer(i.e: Consumer_3) to Queue_0 to aid Consumer_0.
I went through RabbitMQ tutorials on Worker Queues and Routing. I thought Routing will solve my problem. I launched three terminals, a producer and two consumers which will receive Black tasks. When the producer sends a few black tasks(Black_Task_1, Black_Task_2), both consumers received the two messages (i.e: Consumer_0 receives Black_Task_1 and Black_Task_2, Consumer_3 also receives Black_Task_1 and Black_Task_2) . I want my consumers to share the task, not do the same task. Example, Consumer_0 does Black_Task_1 while Consumer_3 does Black_Task_2. What configurations can I achieve that?
=============================
Update
This is a sample code taken from RabbitMQ, routing tutorial. I modified a little. Note that this code doesn't sent Black, Orange or Yellow queues. But the concept is there.
emit_log_direct.py
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='direct_logs',
type='direct')
severity = sys.argv[1] if len(sys.argv) > 1 else 'info'
message = ' '.join(sys.argv[2:]) or 'Hello World!'
channel.basic_publish(exchange='direct_logs',
routing_key=severity,
body=message)
print " [x] Sent %r:%r" % (severity, message)
connection.close()
receive_logs_direct.py
#!/usr/bin/env python
import pika
import sys
import time
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='direct_logs',
type='direct')
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
severities = sys.argv[1:]
if not severities:
print >> sys.stderr, "Usage: %s [info] [warning] [error]" % \
(sys.argv[0],)
sys.exit(1)
for severity in severities:
channel.queue_bind(exchange='direct_logs',
queue=queue_name,
routing_key=severity)
print ' [*] Waiting for logs. To exit press CTRL+C'
def callback(ch, method, properties, body):
print " [x] %r:%r" % (method.routing_key, body,)
time.sleep(1)
print " [x] Done"
ch.basic_ack(delivery_tag=method.delivery_tag)
channel.basic_qos(prefetch_count=1)
channel.basic_consume(callback,
queue=queue_name)
channel.start_consuming()
Producer
nuttynibbles$ ./4_emit_log_direct.py info "run run info"
[x] Sent 'info':'run run info'
Consumer_0
nuttynibbles$ ./4_receive_logs_direct_customize.py info
[*] Waiting for logs. To exit press CTRL+C
[x] 'info':'run run info'
[x] Done
Consumer_3
nuttynibbles$ ./4_receive_logs_direct_customize.py info
[*] Waiting for logs. To exit press CTRL+C
[x] 'info':'run run info'
[x] Done
I think your basic issue is with this:
If Black lists queue get larger, I want to add an extra Consumer(i.e:
Consumer_3) to Queue_0 to aid Consumer_0.
As soon as you add another consumer to the queue - it will pick up the next available message.
If the first consumer does not acknowledge the message; then multiple workers will be able to work on the same message as it will remain on the queue.
So make sure you are correctly acknowledging the messages:
By default, RabbitMQ will send each message to the next consumer, in
sequence. On average every consumer will get the same number of
messages. This way of distributing messages is called round-robin.
[...]
There aren't any message timeouts; RabbitMQ will redeliver the message
only when the worker connection dies. It's fine even if processing a
message takes a very, very long time.
Depending on the nature of the task, you may be able to split the work between multiple processes by creating a priority queue; which is used by C1 (a consumer) to get additional resources. In this case you'll have to have workers ready and listening on the separate priority queue; thus creating a sub-queue where T1 (a task) is being processed.
However, in other to do this, the initial C1 has to make sure the task is no longer available by acknowledging its receipt.
I think that your problem is that you are creating a new Queue for each consumer. When you call
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
in your consumer, this declares a new queue, tells RabbitMQ to generate a unique name for it, and marks it for exclusive use by the channel in the consumer that is calling it. That means that each consumer will have its own queue.
You then bind each new Queue to the exchange using the severity as a routing key. When a message comes into a direct Exchange, RabbitMQ will route a copy of it to every Queue that is bound with a matching routing key. There is no round-robin across the Queues. Each consumer will get a copy of the message, which is what you are observing.
I believe what you want to do is have each consumer use the same name for the queue, specify the name in the queue_declare, and don't make it exclusive. Then all the consumers will be listening to the same queue. The messages will be delivered to one of the consumers, basically in a round-robin fashion.
The producer (the emit_log.py program) doesn't declare or bind the queue - it doesn't have to, but if the binding isn't established before the message is sent, it will be discarded. If you are using a fixed queue, you can have the producer set it up as well, just be sure to use the same parameters (e.g. queue_name) as the consumer.
Our system processes messages delivered from a messaging system. If no message is received after 10 seconds, an error should be raised (inactivity timeout).
I was thinking of using a ScheduledExecutorService (with 1 Thread). Each time a message is received, I cancel the previous timeout task and submit a new one:
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
Callable timeoutTask = new Callable() {...};
...
synchronized {
timeout.cancel();
timeout = executor.schedule( timeoutTask, 10, TimeUnit.SECONDS);
}
In normal case, we process ~ 1000 / sec. Would this approach scale?
If you share the threadpool and keep the running time of timeoutTask low, it'd be most probably fine. If you'd have one threadpool per ~ 1000 / sec tasks then that would not work.
If you are still worried, you can have a look at HashedWheelTimer from the Netty project (link). This thing is very efficient in scheduling timeouts. Note that you MUST share instance of HashedWheelTimer as well.