Step functions - read from SQS? - amazon-web-services

There will be thousands of messages in the SQS.
Is it possible in the Step functions wait until OrderId:123 (json) is in the SQS and then execute the Lambda function when specific Order Id is received?
Edit:
Step Functions to call the Lambda function at regular intervals until it manages to retrieve a message with a particular attribute. OrderId attribute will be in the body message. For example:
{
"OrderId": 1235,
"Items": [{"Id":1, "Name": "Item 1"}]
}

No, it is not possible to selectively retrieve a message from Amazon SQS.
Your application can request to receive 1-10 messages from SQS, but cannot request specific messages.
See: Finding certain messages in SQS
You could use send the messages to Amazon SNS instead and then Filter Messages with Amazon SNS using attributes, and also subscribe the SQS queue to the SNS topic to send a copy of the message to SQS, but this is starting to get too complicated.
You should probably re-architect the solution to have incoming orders trigger the next activity, rather than searching for a given order response.

Related

AWS Step Functions connect and visualize sns + sqs + lambda

I have a situation in which I have a message published to sns topic, an sqs queue subscribed to receive all messages sent to the topic, then a lambda that polls the sqs for messages, the performs some processing (in this case saving to dynamodb). Then there is another lambda that gets triggered off the dynamodb change stream.
For a particular event (identified by a unique id) it would be nice to able to visualize where the event is in this process (i.e. whether it is in the first queue, whether it is in a dlq, etc). I know i can use x-ray or structured logs combined with queries to find this information, but I was wondering if I could use step functions to do this instead.
I know I can send messages to sns with step functions, and I know I can have the next step after that being receive messages from sqs and invoke lambda. However, this seems to just call "receive message" once on the queue, and doesn't connect to the specific event (which would be the point of the visualization).
Does anyone know if this is possible with step functions, or would I need to build my own ui or rely on x-ray or cloudwatch logs insights queries?
Thanks

Polling an AWS SQS queue for messages with certain attributes

I have set up a standard queue with AWS SQS and I want to poll this queue for messages containing a
specific attribute, preferably using the boto3 library in python. I know that boto3 has a
method recieve_message() which polls messages from the queue. However, I want to only get those messages which contain a specific attribute. A naive approach is to iterate through the receive_message() output and check if a message in receive_message() contains the attribute, but I was wondering if there is another solution to this problem.
You can't filter certain messages with SQS solely, however, you can do that with SNS.
You can publish the messages to an SNS topic. The message filter feature of SNS enables endpoints subscribed to an SNS topic to receive only the subset of topic messages it is interested in. So you can ensure only the relevant messages with specific attributes are enqueued to the consumer's queue.
Refer to Filter Messages Published to Topics and SNS subscription filtering policies.
Selective polling is not supported in SQS ReceiveMessage API. An alternative is to let your EC2 send messages containing different attributes into different SQS queues.

How to know once all the SNS subscribers received the message?

Let's say I have set up a AWS SNS with 3 subscribers. I'd like to know when all of the subscribers received/processed the message in order to mark that message as processed by all 3, and to generate some metrics.
Is there a way to do this?
You can log delivery status for SNS topics to CloudWatch, but only for certain types of messages (AWS has no reliable way of knowing if some messages were received or not, such as with SMS or email).
The types of messages you can log are:
HTTP
Lambda
SQS
Custom Application (must be configured to tell AWS that the message is received)
To set up logging in SNS:
In the SNS console, click "Edit Topic"
Expand "delivery status logging"
Then you can configure which protocols to log and the necessary permissions to do so.
Once you're logging to CloudWatch, you can draw metrics from there.
If you need to be notified when the subscribers have received the messages, you could set up a subscription filter within cloudwatch to send the relevant log events to a lambda function, in which you would implement custom logic to notify you appropriately.
I mean successful processing by the consumer
Usually your consumers would have to indicate this somehow. This is use-case specific, therefore its difficult to speculate on exact solutions.
But just to give an example, a popular patter is Request-response messaging pattern. In here, your your consumers would use a SQS queue to publish outcome of the message processing. The producer(s) would pull the queue to get these messages, subsequently, knowing which messages were correctly process and which not.

Sending multiple messages to SNS on a single call

I have a scenario where every time lambda runs, it will send 1000 messages to a SNS topic.
I can loop through the list of messages and call the publish method 1000 times, one message at a time but I was wondering if there is a way to send multiple messages in one call.
In that case I can batch the messages and call publish let say 10 with 100 messages on each execution.
I would really appreciate if someone can help on this.
SNS has just introduced the PublishBatch API, which you can use to send up to 10 messages to an SNS topic in a single API request. More information on the feature launch can be found here: https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-sns-supports-publishing-batches-messages-single-api-request/
With SNS publish you could publish message one by one, AWS does not provide a way to publish message in bulk or batch to SNS.
Below is the possible solution you could try:
You could use send-message-batch API and send bulk messages to SQS. That SQS will be subscribed by a SNS. Below is an image to create subscription to SQS:
You can't achieve it directly via SNS maybe you can have a SQS queue at the front and back of SNS to post and receive messages as batch.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html

Send Cloudwatch logs matching a pattern to SQS queue

I would like to send all Cloudwatch logs where the message of the console.log (appearing in my Cloudwatch logs) matches a certain pattern( for example including the word "postToSlack", or having a certain json field like "slack:true"...)
But I'm stuck at the very beginning of my attempts: I am first trying to implement the most basic task: send ALL cloudwatch logs written when my lambdas are executed (via console.logs placed inside the lambda functions) message to SQS (why? because I first try to make the simplest thing before complexifying with filtering which log to send and which log not to send).
So I created a Cloudwatch Rules > Event > Event Pattern like here below:
{
"source": [
"aws.logs"
]
}
and as a Target, I selected SQS and then a queue I have created.
But when I trigger for example my lambdas, they do appear in Cloudwatch logs, so I would have expected the log content to be "sent" to the queue but nothing is visible on SQs when I poll/check the content of the queue.
Is there something I am misunderstanding about cloudwatch Rules ?
CONTEXT EXPLANATION
I have lambdas that every hour trigger massively (at my scale:) with like maybe 300 to 500 executions of lambdas in a 1 or 2 minutes period.
I want to monitor on Slack all their console.logs (i am logging real error.stack javascript messages as well as purely informative messages like the result of the lambda output "Report Card of the lambda: company=Apple, location=cupertino...").
I could just use a http call to Slack on each lambda but Slack for incoming hooks has a limit of about 1 request per second, after that you get 429 errors if you try to send more than 1 incoming webhook per second... So I thought I'd need to use a queue so that I don't have 300+ lambdas writing to Slack at the same second, but instead controlling the flow from AWS to Slack in a centralized queue called slackQueue.
My idea is to send certain logs (see further down) from Cloudwatchto the SQS slackQueue, and then use this SQS queue as a lambda trigger and sending with this lambda batches of 10 messages (the maximum allowed by AWS; for me 1 message= 1 console.log) concatenated into one big string or array (whatever) to send it to my Slack channel (btw, you can concatenate and send in one call up to 100 slack messages based on Slack limits, so if i could process 100 messages=console.log and concatenate I would but the current batch size limit is 10 for AWS I think ), this way, ensuring I am not sending more than 1 "request" per second to Slack (this request having the content of 10 console.logs).
When I say above "certain logs", it means, I actually I don't want ALL logs to be sent to the queue (because I don't want them on Slack): indeed I don't want the purely "debugging" messages like a console.log("entered function foo"). which are useful during development but have nothing to do on Slack.
As regards some comments: I don't want to use , to my understanding (not expert of AWS) cloudwatch alarms, or metrics filters because they're quite pricy (I'd have those triggered hundreds of times every hour) and don't actually fit my need: I don't want only to read on Slack only when critical problem or a "problem" occurs (like CPU> xxx ...) but really send a regular filtered flow of "almost" all my logs to Slack to read the logs inside Slack instead of inside AWS as Slack is the tool opened all day long, that it's being used for logs/messages coming from other sources than AWS as a centralized place, and that pretty Slack attachment messages formatting is better digested by us. Of course the final lambda (the one sending the messages to slack) would do a bit of formatting to add the italic/bold/etc., and markdown required by slack to have nicely formatted "Slack attachements" but that's not the most complex issue here :)
#Mathieu, I guess you've misunderstood the CloudWatch Events with CloudWatch logs slightly.
What you need is a real time processing of the log data generated by your lambda functions, filter the logs based on a pattern and then store those filtered logs to your Slack for analysis.
But configuring a CloudWatch Event with SQS is similar like a SQS trigger to Lambda. Here, cloudWatch will trigger (send message to) the SQS queue. The content of the message is not your logs but either the default or custom message that you've created.
Solution #1:
Use Subscription filter to filter out the logs as per requirement and subscribe to AWS Kinesis/AWS Lambda/Amazon Kinesis Data Firehouse.
Using the filtered stream (Kinesis), trigger your lambda to push that data to Slack.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
Solution #2:
Push your cloudWatch logs to S3.
Create a notification event in S3 on 'ObjectCreated' event and use that to trigger a Lambda function.
In your Lambda function, write the logic to read the logs from S3 (equivalent to reading a file), filter them and push the filtered logs to Slack.