I know that AWS Lambda can be invoked by CloudWatch scheduler as well as by SQS event, but can they be used together in logical "and" combination?
Basically, what I need is to run my lambda every minute (for example) only when messages available in SQS. Is it even possible with AWS config only?
I need this to be able to utilize some third-party API with hard API limit, that's why I cannot just use SQS event (easy to break the limit) and I don't like the idea to use scheduler only, because it will be useless when queue is empty.
While this is a cool idea, this is unfortunately not possible - event sources in Lambda are always separate from each other. I understand your impulse to save CPU-cycles and API-calls (and money), but I think the only solution that works is your proposed put-it-on-a-timer-and-poll-sqs one.
I was searching the documentation for references on this, but couldn't find any.
Related
I have a service that uses a JSON file on an S3 bucket for its configuration.
I would like to be able to modify this file, but I'm going to run into a concurrency issue as multiple administrators will be able to write in this file at the same time.
I'm going to use an SNS Topic to trigger a lambda that will write the config changes.
For the moment, I'm going to check the queue every minute and then handle the messages, so that I am sure that I don't have multiple instances of lambda running at the same time and writing in the same file.
Is there any way to have an SNS topic to trigger a lambda function for each message, and then wait for this message to be handled and then move on to the next one?
Cheers,
Julien
You can achieve this by setting the max concurrent executions of your Lambda function to 1. See the documentation for more details about managing concurrency for Lambdas.
I have an AWS Lambda function to read from an SQS queue. The lambda logic is basically to read off one message from SQS and then it processes and deletes the message. Code to read the message being something like.
ReceiveMessageRequest messageRequest =
new ReceiveMessageRequest(queueUrl).withWaitTimeSeconds(5).withMaxNumberOfMessages(1);
Now my question is what is the best way to trigger this lambda and how does this lambda scale for instance, if there are let's say 1000 messages in the queue so will there be a 1000 lambdas running together, since in my case one lambda can read only one message off the queue.
Any pointers on best practices around this kind of design.
Right now you best option is probably to setup an AWS Cloudwatch event rule that calls the lambda function on the interval that you need.
Here is a sample app from AWS to do just that:
https://github.com/awslabs/aws-serverless-sqs-event-source
I do believe that AWS will eventually support SQS as a event type for AWS lambda, which should make this even easier, but for now you best choice is probably a version of the code I linked above.
We can now use SQS messages to trigger AWS Lambda Functions. Moreover, no longer required to run a message polling service or create an SQS to SNS mapping.
Further details:
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
AWS added native support in June 2018: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
There are probably a few ways to do this, but I found this guide to be fairly helpful when I tried to implement the same sort of functionality you are describing in Node.js. One downside to this strategy is that you can only poll the queue every 60s.
The basic workflow would look something like this:
Set up a CloudWatch Alarm that gets triggered when the queue has a certain number of messages.
The Cloudwatch alarm then posts to SNS
The SNS message triggers a Lambda scale() function
The scale() function updates a configuration record in a DynamoDB table that sets the number of worker processes needed
You then have a main CloudWatch Schedule that invokes a worker() function every 60s
The worker() function reads configuration from DynamoDB to determine how many concurrent processes are needed, based on the queue size.
Worker() then invokes the appropriate number of process() functions
Process() function consumes messages from SQS, performs your main application logic, and then removes the item from the queue.
You can find an example of what the scaling functions would look like in Node.js here
I have used this solution in a production environment for almost a year without any issues, even with thousands of messages in the queue. If you cut out the scaling portion it is only going to do one message a time.
My use case is as follows: I need to be able to schedule SQS messages in such a way that scheduled messages can be added to a queue on a specific date/time, and also on a recurring basis as needed.
At the implementation level, what I'm basically be looking to do is have some function I can call where I pass in the SQS queue, message, and schedule I want it to run on, without having to build the actual scheduler logic.
I haven't seen anything in AWS itself that seems to allow for that, I also didn't get the impression Lambda functions would do exactly what I need unless I'm missing something.
Is there any other third party cloud service for scheduled processes I should look into, or am I better off in the end just running a scheduling machine at AWS and have some REST API that can add cron jobs/windows scheduled tasks to it that will handle the scheduling of SQS messages?
I could see two slightly different ways of accomplishing this, both based on Cloudwatch scheduled events. The first would be to have Cloudwatch fire off a Lambda. The Lambda would either have the needed parameters or would get them from somewhere else - for example, a DynamoDB table. Otherwise, the rule target allows you to specify a SQS queue - skipping the Lambda. But I'm not sure if that would have the configuration ability you'd want.
Either way, checkout Cloudwatch -> Events -> Create Rule in the AWS console to see your choices.
What's the better option to coordinate tasks between microservices?
For example, if I have a microservice that handles customer information and need to notifies other microservices, is it better to create a workflow (AWS Steps) between microservices or use a SNS?
I think AWS Steps will couple my lambda functions, and SNS not.
AWS Step Functions is a step machine that executes AWS Lambda functions. If your task involves "do this, then this" activities, then Step Functions could be a good option. It includes logic to determine the next step and automatically handles retries. It's the modern version of Amazon Simple Workflow (SWF).
Amazon Simple Notification Service (SNS) can also trigger Lambda functions, but it does not handle the logic nor the retries. It's a good fit for decoupled services, especially for fan-out where multiple subscribers receive the same message from a topic -- for example, for triggering multiple Lambda functions or sending multiple notifications. It's basically a public/subscribe service, of which Lambda is one of the subscriber types.
The choice will depend upon your particular use-case. If you don't want to redesign things to use Step Functions, then send notifications via SNS. If you sometimes send notifications (eg emails) rather than just trigger Lambda functions, use SNS.
Currently, Step Functions is not available in every region, while SNS is everywhere so that might also influence your choice.
It depends on what type of coordination you want.
Synchronous or Asynchronous.
If it is synchronous and if you really want some co-ordination between them, then Amazon Simple Notification Service (SNS) would not help and AWS Step Functions would be the way to go.
But if the requirement is asynchronous, and you just want to notify/invoke the microservices then SNS would be a better fit.
As I can read from your question "need to notify other microservices" I assume it is just about notifying them (as against to co-ordinating them) and each would know what to do further without relying on other microservices. And if that is true then SNS is a good fit.
I am working with PHP technology.
I have my program that will write message to Amazon SQS.
Can anybody tell me how I can use lambda service to get data from SQS and push it into MySQL. Lambda service should get trigger whenever new record gets added to the queue.
Can somebody share the steps or code that will help me to get through with this task?
There isn't any official way to link SQS and Lambda at the moment. Have you looked into using an SNS topic instead of an SQS queue?
Agree with Mark B.
Ways to get events over to lambda.
use SNS http://docs.aws.amazon.com/sns/latest/dg/sns-lambda.html
use SNS->SQS and have the lambda launched by the sns notification just use it to load whatever is in te SQS queue.
use kinesis.
alternatively have lambda run by cron job to read sqs. Depends on needed latency. If you require it be processed immediately then this is not the solution because you would be running the lambda all the time.
Important note for using SQS. You are charged when you query even if no messages are waiting. So do not do fast polls even in your lambdas. Easy to run up a huge bill doing nothing. Also good reason to make sure you set up cloudwatch on the account to monitor usage and charges.