I wonder whether there is a way to use only lambda function when I dispatched a job in laravel.
What I'm using is below.
Laravel 5.8(PHP 7.2)
AWS SQS
Supervisord
In Laravel, I dispatch a job with SQS connection and job is in Laravel project.
I searched how I can use SQS as a trigger for lambda function. And I found this document. ( Using AWS Lambda with Amazon SQS
)
If I follow this document, I think I can run job in lambda. But In Laravel project, job will be run again. I want to use only lambda as a job.
How I can run only lambda function as a job?
No it is not possible. Sqs, database or redis are just for keeping the serialized(encoded etc) version of your laravel jobs. Here is the closest you may get;
Forget about sqs queue driver.
Implement your job in aws lambda.
Allow lambda to consume your sqs (policies, triggers etc listed in the documentation)
Make a request from your laravel app via aws php sdk or http request(guzzle, curl) to your sqs and let lambda to consume your sqs.
You may use some async driver to trigger your sqs requests asynchronous.
If you want to use sqs delay queue, The maximum is 15 minutes - here for the doc
Related
I have an AWS Python lambda function that connects to a DB, checks data integrity and send alerts to a slack channel(that's already done).
I want to execute that lambda every XX minutes.
What's the best way to do it?
You can build this with AWS EventBridge.
The documentation contains an example for this exact use case:
Tutorial: Schedule AWS Lambda Functions Using EventBridge
I have an application that uses AWS SQS with Lambda to process the messages pushed on the Queue. The Lambda keeps on polling the Queue, and when a new message appears it process the message.
For this scenario, is it possible to replace the SQS with Kafka on the AWS. In other words, can we use Kafka as a Queue for this use case?
You absolutely can. Have a look at AWS Amazon Managed Streaming for Apache Kafka (Amazon MSK)
. It's a managed service for Apache Kafka.
As for lambda triggers, unfortunately it's not a built in trigger. You can easily replicate the behaviour by using a periodically triggered lambda function that checks if the messsages are visible and then invokes the function that will process the message or processes the message directly. For some direction you can refer this official guide which sets up a similar pipeline, but for AWS MQ.
I have a PHP web application that is running on an ec2 server. The app is integrated with another service which involves subscribing to a number of webhooks.
The number of requests the server is receiving from these webhooks has become unmanageable, and I'm looking for a more efficient way to deal with data coming from these webhooks.
My initial thought was to use API gateway and put these requests into an SQS queue and read from this queue in batches.
However, I would like these batches to be read by the ec2 instance because the code used to process the webhooks is code reused throughout my application.
Is this possible or am I forced to use a lambda function with SQS? Is there a better way?
The approach you suggested (API Gateway + SQS) will work just fine. There is no need to use AWS Lambda. You'll want to use the AWS SDK for PHP when writing the application code that receives messages from your SQS queue.
I've used this pattern before and it's a great solution.
. . . am I forced to use a lamda function with SQS?
SQS plus Lambda is basically free. At this time, you get 1M (million) lambda calls and 1M (million) SQS requests per month. Remember that those SQS Requests may contain up to 10 messages and that's a potential 10M messages, all inside the free tier. Your EC2 instance is likely always on. Your lambda function is not. Even if you only use Lambda to push the SQS data to a data store like RDBMS for your EC2 to periodically poll, the operation would be bullet-proof and very inexpensive. With the introduction of SQS you could transition the common EC2 code to Lambda function(s). These now have a run time of 15 minutes.
To cite my sources:
SQS pricing for reference: https://aws.amazon.com/sqs/pricing/
Lambda pricing for reference: https://aws.amazon.com/lambda/pricing/
Superb news about the Amazon MQ service but now the question arises on how I can trigger a Lambda function (Node.js) on a message on a specific Queue?
I was thinking if I somehow can get a SNS topic posted on a message PUT or some other trigger that can fire a Lambda to consume the message from the Queue...
Any suggestions?
There isn't a native way to do this. Amazon's managed ActiveMQ service is simply a managed deployment of ActiveMQ running in EC2. It has no integration with other services.
You'd need to write a queue consumer and have it running on a server and listening to the queue on ActiveMQ and publishing those messages to SNS or invoking the Lambda function directly via the Lambda API, etc.
(At least for now.)
AWS recently announced
https://aws.amazon.com/about-aws/whats-new/2020/11/aws-lambda-supports-amazon-mq-apache-activemq-event-source/
We can now add Trigger as MQ in lambda. Then Configure
Broker url
Batch size
Queue Name
Here is one approach that AWS describes - https://aws.amazon.com/blogs/compute/invoking-aws-lambda-from-amazon-mq/
Basically, have a cloud watch trigger for the lambda, start polling for MQ messages and process those messages
Does AWS lambda provide support for listening to SQS queue? I found some examples which says one can do that but I am not sure if AWS lambda explicity provide support for that. When I create the lambda function, then I found one blueprint for SQS. So,
I linked to it in your other thread - these are the supported event sources. Notice that cloudwatch events are one of the possible event types. You could set up a Lambda to, for example, run every minute and poll an SQS queue. You cannot directly trigger a Lambda off of an SQS queue.
Good news, this feature was released yesterday!
28 JUN 2018: AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources
Read the announcement blog post here: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
AWS Serverless Model supports a new event source as following:
Type: SQS
PropertiesProperties:
QueueQueue: arn:aws:sqs:us-west-2:012345678901:my-queue arn:aws:sqs:us-west-2:0123456789 # NOTE: FIFO SQS Queues are not yet supported
BatchSize: 10
And this is how it can be configured from the AWS Console UI:
You can make your lambda function poll the queue using the SQS API. You could use SNS to trigger the Lambda function.
Update: AWS Lambda can now be triggered from Amazon SQS queues.
Old answer:
Rather than having AWS Lambda poll an Amazon SQS queue, the application that sends the message to SQS queue should instead directly invoke the Lambda function. This could be done in several ways:
Direct invocation via an AWS API call
Sending a message to an Amazon SNS topic, with the Lambda function subscribed to the topic
Calling a function via AWS API Gateway, which can then trigger a Lambda function
The extra step of putting a message into an SQS queue is not necessary.