I have a simple lambda app that is not in production right now, only being used for testing and debugging. The function sends a message to SQS to perform CRUD operations on an external application. I've set this function to be invoked by SQS when it receives a message, so the same function is sending and receiving.
I've just received an email saying I've used over 85% of my free tier SQS requests quota, or over 850,000 requests in just the past 2 weeks. I'm certain these requests are not messages being sent to queue, or received. The number of sends/receives has to be under 1000 for how often I've used this app. I've also verified using SQS monitoring that there are no messages stuck in queue. And the number of sent messages is more or less what I expected, a low number.
Like I said this app is only being used by myself for testing, a few days per week. Where does the 850,000+ requests come from?
Amazon SQS is charged at $0.40 per million API calls. Calls include send, receive and delete, so it is possible that a message might use 3+ API calls.
From AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources | AWS News Blog:
There are no additional charges for this feature, but because the Lambda service is continuously long-polling the SQS queue the account will be charged for those API calls at the standard SQS pricing rates.
Long-polling takes 20 seconds, which makes 4320 polls per day. This equates to 60,480 over two weeks or 129,600 per month. Admittedly, it would be more if messages are flowing, since long polling exits whenever there are messages.
So, either the queue is being used a lot (and you are getting excellent value for your $0.40) or you have something else generating lots of SQS API calls.
If you use the same function for sending to SQS and receive from SQS, it means that:
Lambda send message to SQS -> SQS receive the message -> SQS trigger Lambda -> Lambda send message to SQS
And... It's an infinite loop :)
Related
I am developing an email client that uses SMTP server from my customers. I stock the credentials in an AWS RDS database. Customers are able to create mail campaigns, and email have to be sent asynchronously after the campagin creation.
I want to configure a SQS queue to make messages available with a delay of 2 minutes between each message. The purpose is to send the message to a Lambda function able to send the message through SMTP (using the credentials in the database). Thanks to this delay between messages, I can optimize the delivery of the emails. The order of the delivery is not really important.
Unfortunately, I do not find a way to do that. The timers available in SQS are for the entire Queue, or have to be specific for each message. In my case, if i put 100 messages in the SQS queue, I need the first one to be sent immediately, the second 2 minutes after and so on.
Does someone has solutions with this issue ? If SQS is not the right service to manage this need, is there another one available on the AWS platform ?
We can probably achieve this combination of lambda event source mapping config for sqs and lambda reserved concurrency setting.
Event source Mapping Batch size: 1
This will allow only 1 message to be sent to Lambda at once.
Event source Mapping Batch window: 2 mins
This will allow lambda to be invoked only once every 2 minutes.
Lambda Reserved Concurrency as 1
This will allow only one lambda thread to be called at once.
Thanks #luk2302 for your answer, it is the good one accordingly to the AWS support team. Using the EventBridge service to trigger the lambda at define timing works perfectly.
I have 9000 AWS SNS topics with more than 1M subscribers in each topic. At the moment I am lopping to each topic to send a push message which is consuming lot of my system resources. Is there a way to send message to all the topics at once? what is the best approach to handle the scenario?
It is not possible to subscribe Amazon SNS queues to an Amazon SNS queue, so there is no out-of-the-box method for sending one message to multiple queues.
I would recommend creating an AWS Lambda function that will:
Retrieve a list of all relevant queues (based on tag?)
Loops through and sends a message to each queue
Thus, you would just trigger the Lambda function with one message and it would go to all other queues. It would not "consume system resources", but it is charged based upon run duration. Lambda functions can run for a maximum of 15 minutes, so as long as it sends 10+ messages per minute, it can send to 9000 topics.
Depending upon your use-case, you might also consider using Amazon Pinpoint:
Amazon Pinpoint is an AWS service that you can use to engage with your customers across multiple messaging channels. You can use Amazon Pinpoint to send push notifications, emails, SMS text messages, and voice messages.
Has anyone else solved the following problem?
I have SNS topic filled with events from S3 and there is Lambda function which is subscribed on this topic and when thousand of events are put to this topic, lambda function is throttled because of exceeding the limit of concurrency. I don't want to request a limit increase for concurrent executions but I would decrease concurrent consuming from the topic, but I didn't find information how to do it. Thanks.
A couple of options regarding SNS:
1) SNS Maximum Receive Rate
Set the SNS Maximum Receive Rate. This will throttle the SNS messages sent to a subscriber, but may not be a great option if you have so many messages that they will be discarded before they can be processed. From the documentation:
You can set the maximum number of messages per second that Amazon SNS
sends to a subscribed endpoint by setting the Maximum receive rate
setting. Amazon SNS holds messages that are awaiting delivery for up
to an hour. Messages held for more than an hour are discarded.
If you're only getting thousands of events at a time, setting the Maximum Receive Rate to Lambda's default concurrent execution limit of '100' might be worth a try.
As #kndrk notes, this throttling is currently only available for HTTP/HTTPS subscribers to the SNS topic. To work around this, you can expose your lambda function via AWS API Gateway and subscribe that endpoint to the SNS topic, rather than the lambda function directly.
2) Process from SQS
Subscribe an SQS queue to the SNS topic and process messages from the queue, rather than directly from the sns topic. A single invokation of SQS ReceiveMessage can only handle 10 messages at a time, so that may be easier for you to throttle.
It is also worth noting that you can publish S3 Events directly to AWS Lambda.
I have the following infrastructure:
I have an EC2 instance with a NodeJS+Express process listening on a port for messages (process 1). Every time the process receives a message it sends it to an SQS queue. Then I have another process in the same machine reading the queue using long polling (process 2). When it finds a message in the queue it inserts the data in a MariaDB database sitting on an RDS instance.
(Just to clarify, messages are generated by users, they send a chunk of data which can contain arbitrary information to the endpoint where the process 1 is listening)
Now I want to put the process that reads the SQS (process 2) in a Lambda function so that the process that writes to the queue and the one that reads from the queue are completely independent. The problem is that I don't know if this is possible.
I know that Lambda function are invoked in response to an event, and the events supported at the moment are S3, SNS, SES, DynamoDB, Kinesis, Cognito, CloudWatch and Cloudformation but NOT SQS.
I was thinking in using SNS notifications to invoke the Lambda function so that every time a message is pushed to the queue, an SNS notification is fired and invokes the Lambda function but after playing a bit with it I've realised that is not possible to create an SNS notification from SQS, it's only possible to write SNS notifications to the queue.
Right now I'm a bit stuck because I don't know how to continue. I have the feeling that is not possible to create this infrastructure due to the current limitations in the AWS services. Is there another way to do what I want or am I in a dead-end?
Just to extend my question with some research I've made, this github repo shows how to read an SQS queu from a Lambda function but the lambda function works only if is fired from the command line:
https://github.com/robinjmurphy/sqs-to-lambda
In the readme, the author mentions the following:
Update: Lambda now supports SNS notifications as an event source,
which makes this hack entirely unneccessary for SNS notifcations. You
might still find it useful if you like the idea of using a Lambda
function to process jobs on an SQS queue.
But I think this doesn't solve my problem, an SNS notification can invoke the Lambda function but I don't see how I can create a notification when a message is received in the SQS queue.
Thanks
There are couple of Strategies which can be used to connect the dots, (A)Synchronously or Run-Sleep-Run to keep the data process flow between SNS, SQS, Lambda.
Strategy 1 : Have a Lambda function listen to SNS and process it in real time [Please note that an SQS Queue can subscribe to an SNS Topic - which would may be helpful for logging / auditing / retry handling]
Strategy 2 : Given that you are getting data sourced to SQS Queue. You can try with 2 Lambda Functions [Feeder & Worker].
Feeder would be scheduled lambda function whose job is to take items
from SQS (if any) and push it as an SNS topic (and continue doing it forever)
Worker would be linked to listen the SNS topic which would do the actual data processing
We can now use SQS messages to trigger AWS Lambda Functions. Moreover, no longer required to run a message polling service or create an SQS to SNS mapping.
Further details:
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
AWS SQS is one of the oldest products of Amazon, which only supported polling (long and short) up until June 2018. As mentioned in this answer, AWS SQS now supports the feature of triggering lambda functions on new message arrival in SQS. A complete tutorial for this is provided in this document.
I used to tackle this problem using different mechanisms, and given below are some approaches you can use.
You can develop a simple polling application in Lambda, and use AWS CloudWatch to invoke it every 5 mins or so. You can make this near real-time by using CloudWatch events to invoke lambda with short downtimes. Use this tutorial or this tutorial for this purpose. (This could cost more on Lambdas)
You can consider that SQS is redundant if you don't need to persist the messages nor guarantee the order of delivery. You can use AWS SNS (Simple Notification Service) to directly invoke a lambda function and do whatever the processing required. Use this tutorial for this purpose. This will happen in real-time. But the main drawback is the number of lambdas that can be initiated per region at a given time. Please read this and understand the limitation before following this approach. Nevertheless AWS SNS Guarantees the order of delivery. Also SNS can directly call an HTTP endpoint and store the message in your DB.
I had a similar situation (and now have a working solution deploed). I have addressed it in a following manner:
i.e. publishing events to SNS; which then get fanned-out to Lambda and SQS.
NOTE: This is not applicable to the events that have to be processed in a certain order.
That there are some gotchas (w/ possible solutions) such as:
racing condition: lambda might get invoked before messages is deposited into the queue
distributed nature of SQS queue may lead to returning no messages even though there is a message note1.
The solution to both cases would be to do long-polling of SQS queue; but this does make your lambda bill more expensive.
note1
Short poll is the default behavior where a weighted random set of machines is sampled on a ReceiveMessage call. This means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request.
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html
We had some similar requirements so we ended up building a library and open sourcing it to help with SQS to Lambda async. I'm not sure if this fills your particular set of requirements, but thought it might be worth a look: https://read.iopipe.com/sqs-lambda-teaming-up-92c4096be49c
My system run on an Amazon autoscaling group and one feature allows user to user messaging and I have the following use case to resolve.
A new message is sent in my application between users.
A message to notify the the user by e-mail is dropped into a queue with a 60 second delay. This delay allows time for a realtime chat client (faye/angularjs) to see the message and mark it as viewed.
After the delay the message is picked up, the "read" status is checked and if it has not been read by the client an e-mail is dispatched.
Originally I was going to use a cronjob on each application server poll the message queue however it occurs to me it would be more efficient to use SNS to call some kind of e-mail sending endpoint (perhaps in Lambda).
I can't see any way to have SNS poll SQS however, can anybody suggest how this could be done? Essentially I want SNS with a delay so that I don't spam somebody in a "live" chat with e-mail alerts.
Thanks
Unfortunately this is not yet available out of the box. The missing part is the generation of Amazon SNS notifications on message arrival/visibility by an Amazon SQS queue, be it via push (similar to Amazon S3 notifications, or via poll similar to Amazon Kinesis subscriptions (see The Pull/Push Event Models for more on the difference), which would both allow to directly connect an AWS Lambda function to the resp. SQS delay queue events, see e.g.:
Lambda with SQS
That being said, you can work around this limitations in a few ways, for example:
trigger your Lambda function on schedule (e.g. once per minute), and poll your SQS delay queue from there
scheduled Lambda functions are an eagerly awaited missing Lambda feature in turn, but it is more easily worked around, be it either by a cron job of yours, or Eric Hammond's Unreliable Town Clock (UTC) for example
The AWS Lambda team has delivered many/most similar feature requests over recent month' btw., so I would expect them to offer both SQS event handling and scheduled Lambda functions over the course of the year still.
In early 2019, this problem can be solved in a few different ways:
SQS as an Event Source to Lambda (finally announced 2018-06-28),
similar to the OP's original design.
AWS Step Functions (announced 2016-12-01), using a wait step for
the delay.
DynamoDB Streams with Lambda triggers (announced 2017-02-17),
using TTL expiration on items to fire the Lambda trigger.
As SNS has a topic limit of 100,000 per account, I would recommend using Amazon SES to send the emails (62,000 free emails/month could help with implementation cost decisions).