We are using kafka.js event client with event hubs and we are seeing performance issues with kafka.js. The event client was originally running on Azure SDK event client for event hubs and we recently upgraded to kafka.js.
The issue is it's taking longer time to checkpoint before committing the event.
Azure SDK used to take 15ms to checkpoint (using external Azure storage account outside of event hubs) vs new kafka.js event client relying on Azure event hubs internal storage for checkpointing but taking more than 1 second to checkpoint. Extremely slow.
We are using eachMessage and checkpointing after each message to ensure data consistency but Azure event hubs throttles 4 calls/second per partition and each checkpoint call takes close to 1 second due to shared internal checkpoint storage used by event hubs.
We tried using eachBatch with autoCommit but it seems to be committing entire batch of messages in case if it fails during batch processing.
Anyone noticed similar issue elsewhere? appreciate any feedback or suggestion to solve this.
Here is the code snippet:
/**
* #method consume()
*/
public async consume() {
await this.client.run({
eachBatchAutoResolve: this.configuration.eachBatchAutoResolve, // default to false so consumer does not auto-resolve the offsets
autoCommitThreshold: this.configuration.autoCommitThreshold,
partitionsConsumedConcurrently: this.configuration.partitionsConsumedConcurrently,
eachBatch: async ({ batch, resolveOffset, heartbeat }) => {
if (this.shutdownContext != null) {
// prevent processing new events during a shutdown.
return;
}
const { topic, partition } = batch;
const partitionId = `${topic}-${partition}`;
if (this.partitionTracker.isPartitionProcessing(partitionId)) {
// prevent concurrent event processing on the same partition.
return;
}
this.partitionTracker.startProcessing(partitionId);
this.genericLogger.debug(
`batch last offset: ${batch.lastOffset()} partitionId: ${partitionId} highWatermark: ${
batch.highWatermark
} offsetLag: ${batch.offsetLag()}`,
);
/* eslint-disable no-await-in-loop, no-restricted-syntax */
for (const message of batch.messages) {
let event: Event<any>;
const partitionKey = message.key ? message.key.toString() : message.key;
const { offset, timestamp } = message;
// build event context
const context: EventContext = {
topic,
partition,
partitionKey,
offset,
timestamp,
};
try {
// transform kafka message to event
event = await this.transform(message);
} catch (error) {
const id = uuid.v4();
this.genericLogger.crit(
`Event failed consumer transformation. Please check the Dead Letter topic for a version of the event:${id}`,
);
const poisonousEvent: Event<any> = {
id,
correlationId: { id, origin: uuid.v4() },
data: message.value,
type: 'ca.event.poisonousmessage.created.v1',
};
await this.dlqProducer.send(
poisonousEvent,
this.configuration.dlqTopic,
this.genericLogger,
);
try {
resolveOffset(message.offset);
await heartbeat();
// await commitOffsetsIfNecessary();
this.partitionTracker.stopProcessing(partitionId);
await this.shutdownCheck(context);
} catch (commitError) {
await this.signalShutdown(
ProcessExceptionType.FAILURE_TO_PROCESS_AND_COMMIT,
commitError,
);
await this.shutdown({
reason: ProcessExceptionType.FAILURE_TO_PROCESS_AND_COMMIT,
error: commitError,
eventContext: context,
});
}
break;
}
const contextLogger = getLoggerFactory(
this.configuration.loggerOptions,
'event-client-consumer',
)(getEventMetaData(event));
const eventMessageContext = `[eventHubName:${topic} partition:${partition} offset:${offset} partitionKey:${partitionKey} timestampUTC:${timestamp}]`;
contextLogger.debug(`received new message on ${eventMessageContext}.`);
try {
// heartbeat
await heartbeat();
} catch (error) {
error.message = `error while manually sending a heartbeat: ${error.message}`;
if (error.message.includes(REBALANCING)) {
contextLogger.warn(error.message);
} else {
contextLogger.error(error.message);
}
break;
}
try {
// process
await this.process(event, context, contextLogger, heartbeat);
resolveOffset(message.offset);
// commit
// await this.commit(
// event,
// context,
// contextLogger,
// message.offset,
// resolveOffset,
// commitOffsetsIfNecessary,
// );
} catch (error) {
if (error.message.includes(REBALANCING)) {
contextLogger.warn(
`rebalancing error during processing or committing: ${error.message}`,
);
throw error;
}
const exception = error as Exception;
switch (exception.type) {
case ProcessExceptionType.COMMIT: {
contextLogger.error(
`[Event Processing Error] failed to commit processed event on ${eventMessageContext}. error: ${error.message}.`,
error,
);
// failed to commit message after retry. sleeping.
await this.sleep(context, error as Error, exception.type, heartbeat);
// await this.signalShutdown(exception.type, error);
// await this.shutdown({ reason: exception.type, error, eventContext: context });
throw error;
}
case ProcessExceptionType.SHUTDOWN_NO_COMMIT: {
// shutdown with no commit. most likely due to non retryable circuit breaker exception.
await this.sleep(context, error as Error, exception.type, heartbeat);
// await this.signalShutdown(exception.type, error);
// await this.shutdown({ reason: exception.type, error, eventContext: context });
throw error;
}
case ProcessExceptionType.SLEEP_NO_COMMIT: {
// return early for sleep cycle.
throw error;
}
default: {
if (this.configuration.continueOnFailedEventProcessing) {
contextLogger.crit(
`[Event Processing Error] Error in user defined code when processing event on ${eventMessageContext} beyond process loop retry limit error: ${error.message}. Commiting event to continue processing new events.`,
error,
);
try {
resolveOffset(message.offset);
await heartbeat();
// await commitOffsetsIfNecessary();
this.partitionTracker.stopProcessing(partitionId);
await this.shutdownCheck(context);
} catch (commitError) {
await this.sleep(context, commitError as Error, exception.type, heartbeat);
// await this.signalShutdown(
// ProcessExceptionType.FAILURE_TO_PROCESS_AND_COMMIT,
// commitError,
// );
// await this.shutdown({
// reason: ProcessExceptionType.FAILURE_TO_PROCESS_AND_COMMIT,
// error: commitError,
// eventContext: context,
// });
}
} else {
contextLogger.crit(
`[Event Processing Error] Error in user defined code when processing event on ${eventMessageContext} beyond process loop retry limit error: ${error.message}. Initiating sleep.`,
error,
);
await this.sleep(context, error as Error, exception.type, heartbeat);
// await this.signalShutdown(
// ProcessExceptionType.FAILURE_TO_PROCESS_AND_COMMIT,
// error,
// );
// await this.shutdown({
// reason: ProcessExceptionType.FAILURE_TO_PROCESS_AND_COMMIT,
// error,
// eventContext: context,
// });
}
}
}
throw error;
}
}
this.genericLogger.debug(
`batch last offset: ${batch.lastOffset()} partitionId: ${partitionId} highWatermark: ${
batch.highWatermark
} offsetLag: ${batch.offsetLag()}`,
);
this.genericLogger.debug(
`resolved batch of ${batch.messages.length} messages partitionId: ${partitionId}`,
);
this.partitionTracker.stopProcessing(partitionId);
await this.shutdownCheck();
},
});
}
What is the best way to call a method continuously after a fixed interval?
I want to design a Poller that can pull messages from AWS SQS automatically after a defined time interval.
Any good suggestions are much appreciated.
There are two polling mechanisms short polling - if you are expecting data more frequently and long polling if less frequently.
You should use something mixed of the above i.e
pull recursively in the timeout of 10ms,
If pull contains any message(successful pull) continue polling with the same speed else
change a timeout to let say 5000ms.
Sample:
//timeout is in ms
timeout = 10;
function pullFromSQS() {
message = sqs.pull();
if (message.length) {
processMessage(message);
timeout = 10;
} else {
timeout = 5000;
}
wait(timeout);
pullFromSQS();
}
you can change the timeout as per your convenience for better optimization (both cost and performance)
You can use SDK provided by AWS to do polling of messages
import java.util.List;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import com.amazonaws.AmazonClientException;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.amazonaws.services.sqs.model.Message;
import com.amazonaws.services.sqs.model.ReceiveMessageRequest;
public class SQSRealtimePoller implements Runnable {
public static final int MAX_MESSAGES = 10;
public static final int DEFAULT_VISIBILITY_TIMEOUT = 15;
//Value greater that 0 makes it long polling, which will reduce SQS cost
public static final int WAIT_TIME = 20;
public static final int PROCESSORS = 2;
ExecutorService executor = Executors.newFixedThreadPool(1);
private String queueUrl;
private AmazonSQS amazonSqs;
ArrayBlockingQueue<Message> messageHoldingQueue = new ArrayBlockingQueue<Message>(
1);
public SQSRealtimePoller(String topic, String queueUrl
) {
this.queueUrl = queueUrl;
this.amazonSqs = getSQSClient();
messageHoldingQueue = new ArrayBlockingQueue<Message>(PROCESSORS);
//process more than 1 messages at a time.
executor = Executors.newFixedThreadPool(PROCESSORS);
}
#Override
public void run() {
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest()
.withQueueUrl(queueUrl)
.withMaxNumberOfMessages(MAX_MESSAGES)
.withVisibilityTimeout(DEFAULT_VISIBILITY_TIMEOUT)
.withWaitTimeSeconds(WAIT_TIME);
while(true){
try {
List<Message> messages = amazonSqs
.receiveMessage(receiveMessageRequest).getMessages();
if (messages == null || messages.size() == 0) {
// If there were no messages during this poll period, SQS
// will return this list as null. Continue polling.
continue;
} else {
for (Message message : messages) {
try {
//will wait here till the queue has free space to add new messages. Read documentation
messageHoldingQueue.put(message);
} catch (InterruptedException e) {
}
Runnable run = new Runnable() {
#Override
public void run() {
try {
Message messageToProcess = messageHoldingQueue
.poll();
//Process your message here
System.out.println(messageToProcess);
//Delete the messages from queue
amazonSqs.deleteMessage(queueUrl,
messageToProcess
.getReceiptHandle());
} catch (Exception e) {
e.printStackTrace();
}
}
};
executor.execute(run);
}}
} catch (Exception e) {
e.printStackTrace();
}
}
}
//Make this singleton
public static AmazonSQS getSQSClient(){
ProfileCredentialsProvider credentialsProvider = new ProfileCredentialsProvider();
try {
credentialsProvider.getCredentials();
} catch (Exception e) {
throw new AmazonClientException(
"Cannot load the credentials from the credential profiles file. " +
"Please make sure that your credentials file is at the correct " +
"location , and is in valid format.",
e);
}
AmazonSQS sqs = AmazonSQSClientBuilder.standard()
.withCredentials(credentialsProvider)
.withRegion(Regions.US_WEST_2)
.build();
return sqs;
}}
I have a data source configured with its connections pool ready to use, and it is exposed to my application via JDNI, but the code my colleagues wrote actually opens and closes a connection for every query. How does WSO2 handle this? Does it really close the connection given by the pool, or it ignores the close and just considers this connection free to be added back to the pool and ready to be used by any other client?
Connection conn = null;
CallableStatement cStmt = null;
try {
Hashtable<String, String> environment = new Hashtable<String, String>();
environment.put("java.naming.factory.initial", "org.wso2.carbon.tomcat.jndi.CarbonJavaURLContextFactory");
Context initContext = new InitialContext(environment);
DataSource ds = (DataSource) initContext.lookup("jdbc/tvaccount");
if (ds != null) {
conn = ds.getConnection();
cStmt = conn.prepareCall("{call getAccountStatusAttr(?)}");
cStmt.setString("pUserLogin", userName);
cStmt.execute();
}
} catch (Exception e) {
log.error("Exception while getting account status: ", e);
} finally {
if (cStmt != null) {
try {
cStmt.close();
} catch (SQLException e) {
}
}
if (conn != null) {
try {
conn.close();
} catch (SQLException e) {
}
}
}
Have you added this java code as a JAR file in WSO2 ESB and then accessed the method by using class mediator? if this is the case then it behaves like a normal java code wherein once the query is executed the connection will be closed.
Trying to send multiple requests at same instant to camel activemq route, one request is serviced and the other request is not serviced and sent back as it is. The Jms messages are set with JMScorrelationId too before sending like below
textMessage.setJMSCorrelationID(UUID.randomUUID().toString());
below is my activemq route
from("activemq:queue:TEST_QUEUE?disableReplyTo=true")
.setExchangePattern(ExchangePattern.InOut)
.process(new Processor() {
public void process(Exchange e) throws Exception {
log.info("Request : "
+ MessageHelper.extractBodyAsString(e.getIn()));
/*Processing Logic*/
}
})
.beanRef("testBean","postDetails")
.inOnly("activemq:queue:TEST_QUEUE");
Multiple (Test for 2 requests) requests sent to the above route concurrently not serviced except one. The servicemix.log shows all recieved requests. But only one is serviced.
Below is the code what is sending request deployed in jboss 6.1 as part of web application.
public Message receive(String message, String queueName) {
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(
"tcp://localhost:61616");
String userName = "smx";
String password = "smx";
Connection connection;
Message response =null;
try {
connection = connectionFactory.createConnection(userName, password);
connection.start();
((ActiveMQConnectionFactory) connectionFactory)
.setDispatchAsync(false);
Session session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
Queue destination = session.createQueue(queueName);
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
TextMessage textMessage = session.createTextMessage(message);
Queue tempQueue = session.createQueue(queueName);
textMessage.setJMSReplyTo(tempQueue);
producer.send(textMessage);
MessageConsumer consumer = session.createConsumer(tempQueue);
response = consumer.receive();
response.acknowledge();
session.close();
connection.close();
} catch (JMSException e) {
e.printStackTrace();
}
return response;
}
Is there some or the other parameter im missing?? please suggest.
Camel will auto send back a reply if the JMS message has a JMSReplyTo header, so your route should just be
from("activemq:queue:TEST_QUEUE")
.process(new Processor() {
public void process(Exchange e) throws Exception {
log.info("Request : "
+ MessageHelper.extractBodyAsString(e.getIn()));
/*Processing Logic*/
}
})
.beanRef("testBean","postDetails");
At the end of the route (eg after calling testBean) then the content of the message body is used as the reply message, that are sent back to the queue named defined in the JMSReplyTo header.
assume the code is correct and webservice timeout occurs.
The problem :
The system crashes and can not display the error message.
How to display error message? So I can provide an alternative to user when there is an error?
1)
I add this Class in the project :
public class MyClass
{
public static async Task LogInSuccess()
{
try
{
-- calling a web service here
}
catch (System.Exception _ex)
{
_strErrorMsg = _ex.InnerException.Message;
throw new Exception("LogInSuccess() " + _strErrorMsg);
}
}
}
--- In the MainPage,
2)
private async void SetUp ()
{
-- code for doing setUp task--
CallWebSvc();
}
3)
private void CallWebSvc()
{
bool ShowError = false;
System.Exception MyException = new Exception();
try
{
-- calling a web service thru the MyClass
System.Threading.Tasks.Task _blnLogInSuccess = MyClass.LogInSuccess();
await _blnLogInSuccess;
if (_blnLogInSuccess.IsCompleted)
{
g_blnLoginStatus = _blnLogInSuccess.Result;
}
}
catch (System.Exception _ex)
{
ShowError = true;
MyException = ex;
}
if (ShowError)
{
var MyMessageBox = new Windows.UI.Popups.MessageDialog("Remote Login Error:" + MyException.Message, "Start Login" );
await MyMessageBox.ShowAsync();
}
}
I assume your CallWebSvc method is async void (as, without async you cannot perform an await) If this is the case, you need to know async void doesn't do the same treatament to exceptions as async task. they aren't catched correctly. If you change your CallWebSvc from async void to async Task, you are going to receive the exception correctly.