What is the best way to call a method continuously after a fixed interval?
I want to design a Poller that can pull messages from AWS SQS automatically after a defined time interval.
Any good suggestions are much appreciated.
There are two polling mechanisms short polling - if you are expecting data more frequently and long polling if less frequently.
You should use something mixed of the above i.e
pull recursively in the timeout of 10ms,
If pull contains any message(successful pull) continue polling with the same speed else
change a timeout to let say 5000ms.
Sample:
//timeout is in ms
timeout = 10;
function pullFromSQS() {
message = sqs.pull();
if (message.length) {
processMessage(message);
timeout = 10;
} else {
timeout = 5000;
}
wait(timeout);
pullFromSQS();
}
you can change the timeout as per your convenience for better optimization (both cost and performance)
You can use SDK provided by AWS to do polling of messages
import java.util.List;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import com.amazonaws.AmazonClientException;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.amazonaws.services.sqs.model.Message;
import com.amazonaws.services.sqs.model.ReceiveMessageRequest;
public class SQSRealtimePoller implements Runnable {
public static final int MAX_MESSAGES = 10;
public static final int DEFAULT_VISIBILITY_TIMEOUT = 15;
//Value greater that 0 makes it long polling, which will reduce SQS cost
public static final int WAIT_TIME = 20;
public static final int PROCESSORS = 2;
ExecutorService executor = Executors.newFixedThreadPool(1);
private String queueUrl;
private AmazonSQS amazonSqs;
ArrayBlockingQueue<Message> messageHoldingQueue = new ArrayBlockingQueue<Message>(
1);
public SQSRealtimePoller(String topic, String queueUrl
) {
this.queueUrl = queueUrl;
this.amazonSqs = getSQSClient();
messageHoldingQueue = new ArrayBlockingQueue<Message>(PROCESSORS);
//process more than 1 messages at a time.
executor = Executors.newFixedThreadPool(PROCESSORS);
}
#Override
public void run() {
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest()
.withQueueUrl(queueUrl)
.withMaxNumberOfMessages(MAX_MESSAGES)
.withVisibilityTimeout(DEFAULT_VISIBILITY_TIMEOUT)
.withWaitTimeSeconds(WAIT_TIME);
while(true){
try {
List<Message> messages = amazonSqs
.receiveMessage(receiveMessageRequest).getMessages();
if (messages == null || messages.size() == 0) {
// If there were no messages during this poll period, SQS
// will return this list as null. Continue polling.
continue;
} else {
for (Message message : messages) {
try {
//will wait here till the queue has free space to add new messages. Read documentation
messageHoldingQueue.put(message);
} catch (InterruptedException e) {
}
Runnable run = new Runnable() {
#Override
public void run() {
try {
Message messageToProcess = messageHoldingQueue
.poll();
//Process your message here
System.out.println(messageToProcess);
//Delete the messages from queue
amazonSqs.deleteMessage(queueUrl,
messageToProcess
.getReceiptHandle());
} catch (Exception e) {
e.printStackTrace();
}
}
};
executor.execute(run);
}}
} catch (Exception e) {
e.printStackTrace();
}
}
}
//Make this singleton
public static AmazonSQS getSQSClient(){
ProfileCredentialsProvider credentialsProvider = new ProfileCredentialsProvider();
try {
credentialsProvider.getCredentials();
} catch (Exception e) {
throw new AmazonClientException(
"Cannot load the credentials from the credential profiles file. " +
"Please make sure that your credentials file is at the correct " +
"location , and is in valid format.",
e);
}
AmazonSQS sqs = AmazonSQSClientBuilder.standard()
.withCredentials(credentialsProvider)
.withRegion(Regions.US_WEST_2)
.build();
return sqs;
}}
Related
Does SWF natively supports manual pause and resume workflow functionality in java framework? If not is there any way to achieve to achieve that semantics?
Edit: I implemented following example, seems to be working with initial testing. Is there anything which can break with this. My workflow is going to be long running (~3-5 hours) with same activity being multiple times with different params.
import com.amazonaws.services.simpleworkflow.flow.annotations.Asynchronous;
import com.amazonaws.services.simpleworkflow.flow.core.Promise;
import com.amazonaws.services.simpleworkflow.flow.core.Settable;
public class GreeterWorkflowImpl implements GreeterWorkflow {
private GreeterActivitiesClient operations = new GreeterActivitiesClientImpl();
Settable<Void> paused = new Settable<>();
public void greet() {
Promise<String> fs = getGreeting(0, operations.getName());
print(fs);
}
#Asynchronous
private Promise<String> getGreeting(int count, Promise<String> name)
{
if (count > 10)
return name;
return getGreeting(count, name, paused);
}
#Asynchronous
private Promise<String> getGreeting(int count, Promise<String> name, Settable<Void> paused) {
Promise<String> returnString = operations.getGreeting(name.get());
return getGreeting(count + 1, returnString);
}
#Asynchronous
private void print(Promise<String> finalString)
{
System.out.println("Final String is " + finalString.get());
}
// #Signal method
#Override
public void pause() {
paused = new Settable<>();
}
// #Signal method
#Override
public void resume() {
paused.set(null);
}
}
In case you get multiple signals for resume, you will be setting the paused settable again (which is already ready) so you might end up with unhandled IllegalStateException
Using the examples provided in https://github.com/aws/aws-sdk-java/tree/master/src/samples/AmazonKinesis I understand how to use the KCL to read KinesisEvents from my local machine.
I am trying to implement the same logic in Lambda Functions.
To set up the KinesisClientLibConfiguration, you need to provide an AWSCredentialsProvider.
Where do I get these AWSCredentials to create the kinesisClientLibConfiguration in a Lambda?
KinesisClientLibConfiguration kinesisClientLibConfiguration = new KinesisClientLibConfiguration(SAMPLE_APPLICATION_NAME, SAMPLE_APPLICATION_STREAM_NAME, credentialsProvider, workerId);
The full code of the handler looks like this:
public class ProcessKinesisRecords implements RequestHandler<KinesisEvent, Void> {
public static final String SAMPLE_APPLICATION_STREAM_NAME = "kinesis-s";
public static final String SAMPLE_APPLICATION_STREAM_REGION = "eu-west-1";
private static final String SAMPLE_APPLICATION_NAME = "SampleKinesisLambdaApplication";
private static final InitialPositionInStream SAMPLE_APPLICATION_INITIAL_POSITION_IN_STREAM =
InitialPositionInStream.LATEST;
private static ProfileCredentialsProvider credentialsProvider;
public Void handleRequest(KinesisEvent event, Context context) {
init();
int exitCode = 0;
try {
String workerId = InetAddress.getLocalHost().getCanonicalHostName() + ":" + UUID.randomUUID();
KinesisClientLibConfiguration kinesisClientLibConfiguration =
new KinesisClientLibConfiguration(SAMPLE_APPLICATION_NAME,
SAMPLE_APPLICATION_STREAM_NAME,
credentialsProvider,
workerId);
kinesisClientLibConfiguration.withInitialPositionInStream(SAMPLE_APPLICATION_INITIAL_POSITION_IN_STREAM);
kinesisClientLibConfiguration.withRegionName(SAMPLE_APPLICATION_STREAM_REGION);
IRecordProcessorFactory recordProcessorFactory = new AmazonKinesisApplicationRecordProcessorFactory();
Worker worker = new Worker(recordProcessorFactory, kinesisClientLibConfiguration);
System.out.printf("Running %s to process stream %s as worker %s...\n",
SAMPLE_APPLICATION_NAME,
SAMPLE_APPLICATION_STREAM_NAME,
workerId);
worker.run();
} catch (Throwable e) {
System.err.println("Caught throwable while processing data.");
e.printStackTrace();
}
System.exit(exitCode);
return null;
}
private static void init() {
// Ensure the JVM will refresh the cached IP values of AWS resources (e.g. service endpoints).
java.security.Security.setProperty("networkaddress.cache.ttl", "60");
credentialsProvider = new ProfileCredentialsProvider();
try {
credentialsProvider.getCredentials();
} catch (Exception e) {
throw new AmazonClientException("Cannot load the credentials", e);
}
}
}
Lambda does provide environment variables for the credentials:
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html#lambda-environment-variables
Access them via the EnvironmentVariableCredentialsProvider:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/EnvironmentVariableCredentialsProvider.java
I have the below Java Class which retrieve messages from JMS queue. This class is invoked in a mule flow. Could you please advice on how I can write a Junit for this class? I have tried to create a standalone broker but I am having trouble
public Object getMessages(final MuleEventContext eventContext)
{
final String consumerID = eventContext.getMessage().getProperty("consumerID", PropertyScope.INVOCATION);
final String messageSelector = "ConsumerID = '" + consumerID + "'";
JmsConnector amqConnector = (JmsConnector) eventContext.getMuleContext().getRegistry().lookupConnector("Active_MQ");
ConnectionFactory factory = amqConnector.getConnectionFactory();
Connection connection = null;
List<String> listOfMessages = null;
try
{
connection = factory.createConnection();
//Consumer Settings
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue(queuename");
MessageConsumer consumer = session.createConsumer(queue, messageSelector);
//Browser Settings
Session sessionBrowser = amqConnector.getSession( false, false );
Queue queueBrowser = sessionBrowser.createQueue( queuename );
QueueBrowser qBrowser = sessionBrowser.createBrowser( queueBrowser, messageSelector );
Enumeration<Message> enumeration = qBrowser.getEnumeration();
connection.start();
listOfMessages = new ArrayList<String>();
while (enumeration.hasMoreElements())
{
enumeration.nextElement();
Message message = consumer.receive();
TextMessage msg = (TextMessage) message;
listOfMessages.add(msg.getText());
}
//Close Browser Settings
qBrowser.close(); sessionBrowser.close();
//Close Consumer Settings
consumer.close(); session.close();
//Close Connection
connection.close();
}
catch ( Exception e )
{
throw new RuntimeException("Unable to retrieve messages from Queue "+ e);
}
}
You can create FunctionalTestCase, and then:
Test the class directly by calling the method, or
Test the whole scenario by calling the flow
public class MessageServiceTest extends FunctionalTestCase {
#Test
public void testJavaClass() throws Exception {
MuleEventContext eventContext = MuleTestUtils.getTestEventContext("", MessageExchangePattern.REQUEST_RESPONSE, muleContext);
MessageService messageService = new MessageService();
assertNotNull(messageService.getMessages(eventContext));
}
#Test
public void testFlow() throws Exception {
MuleEvent event = runFlow("messageserviceFlow");
MuleMessage message = event.getMessage();
assertNotNull(message);
assertNotNull(message.getPayload());
}
protected String getConfigResources() {
return "messageservice.xml";
}
}
I am just starting with Akka and have created a test application. In it I create a bunch of actors who create a scheduler to generate a heartbeat event. Upon another type of event, I cancel the scheduler with heartbeat.cancel();, but I'd like to restart it when another event occurs. If I recreate the scheduler I see that the memory consumption increases continuously.
The question then would be either how do I resume the scheduler or how do I dispose the scheduler properly.
This is the code for that Actor
public class Device extends UntypedActor {
enum CommunicationStatus{
OK,
FAIL,
UNKNOWN
}
private static class Heartbeat {
}
public final String deviceId;
private CommunicationStatus commStatus;
private Cancellable heartBeatScheduler;
public Device(String Id)
{
deviceId = Id;
commStatus = CommunicationStatus.UNKNOWN;
}
#Override
public void preStart() {
getContext().system().eventStream().subscribe(getSelf(), DeviceCommunicationStatusUpdated.class);
startHeartbeat();
}
#Override
public void postStop() {
stopHeartBeat();
}
private void startHeartbeat() {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
log.info("Starting heartbeat");
heartBeatScheduler = getContext().system().scheduler().
schedule(Duration.Zero(),
Duration.create(1, TimeUnit.SECONDS),
getContext().self(),
new Heartbeat(),
getContext().system().dispatcher(),
ActorRef.noSender());
}
private void stopHeartBeat() {
if(!heartBeatScheduler.isCancelled()) {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
log.info("Stopping heartbeat");
heartBeatScheduler.cancel();
}
}
public String getDeviceId() {
return deviceId;
}
public CommunicationStatus getCommunicationStatus(){
return commStatus;
}
#Override
public void onReceive(Object message) throws Exception {
LoggingAdapter log = Logging.getLogger(getContext().system(), this);
if(message instanceof Heartbeat){
log.info("Pum, pum");
}
else if (message instanceof DeviceCommunicationStatusUpdated){
DeviceCommunicationStatusUpdated event = (DeviceCommunicationStatusUpdated) message;
if(event.deviceId == this.deviceId){
log.info("Received communication status update. '{}' is now {}", deviceId, event.state);
this.commStatus =
event.state == DeviceCommunicationStatusUpdated.State.OK ?
CommunicationStatus.OK : CommunicationStatus.FAIL;
if(commStatus == CommunicationStatus.OK && heartBeatScheduler.isCancelled()){
startHeartbeat();
}
else {
stopHeartBeat();
}
}
}
else unhandled(message);
}
}
Finally there is no leak, it's just that I'm new to Java and was impatient with the garbage collection. In any case, I would like to know about the resetting / restarting of a scheduler.
Can i flush all the logs based on time interval using configuration file. Searched a lot. Didn't find any. Short cut is using Timer ourselves and flush all loggers. But wanted to know whether configuraiton file allows it.
The configuration file options are explained in the LogManager documentation. At this time, the only way to do this is via the configuration file is to use the 'config' option to install your custom code to flush all loggers and perform the timer management. If you need to access to the JVM lifecycle, you can create a custom handler that ignores all log records but listens to constructor and close method calls.
public class FlushAllHandler extends Handler {
private final ScheduledExecutorService ses;
private final Future<?> task;
public FlushAllHandler() {
//TODO: Look these up from the LogManager.
super.setLevel(Level.OFF); //Ignore all published records.
ses = Executors.newScheduledThreadPool(1);
long delay = 1L;
TimeUnit unit = TimeUnit.HOURS;
task = ses.scheduleWithFixedDelay(new Task(), delay, delay, unit);
}
#Override
public void publish(LogRecord record) {
//Allow a trigger filter to kick off a flush.
if (isLoggable(record)) {
ses.execute(new Task());
}
}
#Override
public void flush() {
}
#Override
public void close() throws SecurityException {
super.setLevel(Level.OFF);
task.cancel(false);
ses.shutdown();
try {
ses.awaitTermination(30, TimeUnit.SECONDS);
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
}
ses.shutdownNow();
}
private static class Task implements Runnable {
Task() {
}
#Override
public void run() {
final ArrayList<Handler> handlers = new ArrayList<>();
final LogManager manager = LogManager.getLogManager();
synchronized (manager) { //avoid ConcurrentModificationException
final Enumeration<String> e = manager.getLoggerNames();
while (e.hasMoreElements()) {
final Logger l = manager.getLogger(e.nextElement());
if (l != null) {
Collections.addAll(handlers, l.getHandlers());
}
}
}
//Don't hold LogManager lock while flushing handlers.
for (Handler h : handlers) {
h.flush();
}
}
}
}