I have an application that uses AWS SQS with Lambda to process the messages pushed on the Queue. The Lambda keeps on polling the Queue, and when a new message appears it process the message.
For this scenario, is it possible to replace the SQS with Kafka on the AWS. In other words, can we use Kafka as a Queue for this use case?
You absolutely can. Have a look at AWS Amazon Managed Streaming for Apache Kafka (Amazon MSK)
. It's a managed service for Apache Kafka.
As for lambda triggers, unfortunately it's not a built in trigger. You can easily replicate the behaviour by using a periodically triggered lambda function that checks if the messsages are visible and then invokes the function that will process the message or processes the message directly. For some direction you can refer this official guide which sets up a similar pipeline, but for AWS MQ.
Related
I have set up a lambda function and an SQS service. I want the events from the SQS service to get consumed by the lambda function.
SQS configuration
All the other configuration is set to match the default.
Lambda trigger
Lambda configuration
The code used is from the sqs-poller template (and the configuration too)
Code configuration
I'm using the following code to send the event. I run the code with the following command
AWS_SDK_LOAD_CONFIG=true AWS_SHARED_CREDENTIALS_FILE=./credentials node sqs.js
That works fine because I'm seeing the messages in the monitoring panel of the SQS service.
Any idea why events are not being consumed by my lambda function?
It would appear that you have two competing concepts in your architecture.
Amazon SQS and AWS Lambda
When an Amazon SQS queue is configured as a trigger to an AWS Lambda function, the AWS service polls the SQS queue looking for messages. When message(s) are found, the Lambda function is invoked, with the messages being passed to the function via the event variable.
The Lambda function can then process those messages, reading the detail of the messages from the event variable.
If the Lambda function completes without error, the AWS service will automatically delete the messages from the SQS queue. If there was an error in the function, the messages will automatically reappear on the SQS queue after the invisibility period has expired.
At no time does the AWS Lambda function actually call the Amazon SQS service to receive or delete messages. Rather, it is given the messages when it is invoked.
SQS Poller
You mention that you are using an sqs-poller class. I'm not sure whether you are referring to Receiving Messages Using the QueuePoller Class in Amazon SQS - AWS SDK for Ruby or #jishimi/sqs-poller - npm.
Nonetheless, polling is a traditional way that worker processes retrieve messages from an SQS queue, and then delete the messages after they are processed. The process is:
They ask check whether there are messages available in the SQS queue
If so, they invoke a worker
When the worker is finished, they delete the message
You should notice that these are the same steps that the AWS Lambda service does when SQS is configured as a trigger for an AWS Lambda function. Therefore, using a polling architecture is incompatible with using SQS as a trigger for an AWS Lambda function.
You should pick one or the other, not both.
I wonder whether there is a way to use only lambda function when I dispatched a job in laravel.
What I'm using is below.
Laravel 5.8(PHP 7.2)
AWS SQS
Supervisord
In Laravel, I dispatch a job with SQS connection and job is in Laravel project.
I searched how I can use SQS as a trigger for lambda function. And I found this document. ( Using AWS Lambda with Amazon SQS
)
If I follow this document, I think I can run job in lambda. But In Laravel project, job will be run again. I want to use only lambda as a job.
How I can run only lambda function as a job?
No it is not possible. Sqs, database or redis are just for keeping the serialized(encoded etc) version of your laravel jobs. Here is the closest you may get;
Forget about sqs queue driver.
Implement your job in aws lambda.
Allow lambda to consume your sqs (policies, triggers etc listed in the documentation)
Make a request from your laravel app via aws php sdk or http request(guzzle, curl) to your sqs and let lambda to consume your sqs.
You may use some async driver to trigger your sqs requests asynchronous.
If you want to use sqs delay queue, The maximum is 15 minutes - here for the doc
Superb news about the Amazon MQ service but now the question arises on how I can trigger a Lambda function (Node.js) on a message on a specific Queue?
I was thinking if I somehow can get a SNS topic posted on a message PUT or some other trigger that can fire a Lambda to consume the message from the Queue...
Any suggestions?
There isn't a native way to do this. Amazon's managed ActiveMQ service is simply a managed deployment of ActiveMQ running in EC2. It has no integration with other services.
You'd need to write a queue consumer and have it running on a server and listening to the queue on ActiveMQ and publishing those messages to SNS or invoking the Lambda function directly via the Lambda API, etc.
(At least for now.)
AWS recently announced
https://aws.amazon.com/about-aws/whats-new/2020/11/aws-lambda-supports-amazon-mq-apache-activemq-event-source/
We can now add Trigger as MQ in lambda. Then Configure
Broker url
Batch size
Queue Name
Here is one approach that AWS describes - https://aws.amazon.com/blogs/compute/invoking-aws-lambda-from-amazon-mq/
Basically, have a cloud watch trigger for the lambda, start polling for MQ messages and process those messages
I have a general AWS question. I have started using AWS sdk, but looks like if I want to receive events asynchronously from AWS(ex: cloudwatch events), lambda functions is the only way. I want to write a simple application that registers a callback to AWS for events, but i couldn't find a way to do that till now, since i don't want to use lambda, i have been polling from my application. Please, let me know if polling is the only option or if there is a better way to resolve it without polling.
From the documentation:
You can configure the following AWS services as targets for CloudWatch Events:
Amazon EC2 instances
AWS Lambda functions
Streams in Amazon Kinesis Streams
Delivery streams in Amazon Kinesis Firehose
Amazon ECS tasks
SSM Run Command
SSM Automation
Step Functions state machines
Pipelines in AWS CodePipeline
Amazon Inspector assessment templates
Amazon SNS topics
Amazon SQS queues
Built-in targets
The default event bus of another AWS account
That's a lot more than just Lambda, so I'm not sure why you state in your question that Lambda is the only option. The options of Amazon EC2 instances and Amazon SNS topics both provide a method for Amazon to "push" the events to your services, instead of requiring your services to poll.
With cloudwatch events, you can set rules and trigger a number of different targets, including SQS queues which you can poll from your EC2 Instances.
Lambda is certainly a popular endpoint, but based on the docs, there are other targets you can send the events to
Already above answers might also be helpful, but one of the possible options to address your problem could be one of this as well.
You can make use of AWS SNS service to subscribe for the events on AWS resources. And the SNS can publish the events to your application end point. Which is nothing but pub/sub model.
Refer this link http://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html
The end-point could be your http or https based application.
I am working with PHP technology.
I have my program that will write message to Amazon SQS.
Can anybody tell me how I can use lambda service to get data from SQS and push it into MySQL. Lambda service should get trigger whenever new record gets added to the queue.
Can somebody share the steps or code that will help me to get through with this task?
There isn't any official way to link SQS and Lambda at the moment. Have you looked into using an SNS topic instead of an SQS queue?
Agree with Mark B.
Ways to get events over to lambda.
use SNS http://docs.aws.amazon.com/sns/latest/dg/sns-lambda.html
use SNS->SQS and have the lambda launched by the sns notification just use it to load whatever is in te SQS queue.
use kinesis.
alternatively have lambda run by cron job to read sqs. Depends on needed latency. If you require it be processed immediately then this is not the solution because you would be running the lambda all the time.
Important note for using SQS. You are charged when you query even if no messages are waiting. So do not do fast polls even in your lambdas. Easy to run up a huge bill doing nothing. Also good reason to make sure you set up cloudwatch on the account to monitor usage and charges.