Send Cloudwatch logs matching a pattern to SQS queue - amazon-web-services

Question

I would like to send all Cloudwatch logs where the message of the console.log (appearing in my Cloudwatch logs) matches a certain pattern( for example including the word "postToSlack", or having a certain json field like "slack:true"...)
But I'm stuck at the very beginning of my attempts: I am first trying to implement the most basic task: send ALL cloudwatch logs written when my lambdas are executed (via console.logs placed inside the lambda functions) message to SQS (why? because I first try to make the simplest thing before complexifying with filtering which log to send and which log not to send).
So I created a Cloudwatch Rules > Event > Event Pattern like here below:
{
"source": [
"aws.logs"
]
}
and as a Target, I selected SQS and then a queue I have created.
But when I trigger for example my lambdas, they do appear in Cloudwatch logs, so I would have expected the log content to be "sent" to the queue but nothing is visible on SQs when I poll/check the content of the queue.
Is there something I am misunderstanding about cloudwatch Rules ?
CONTEXT EXPLANATION
I have lambdas that every hour trigger massively (at my scale:) with like maybe 300 to 500 executions of lambdas in a 1 or 2 minutes period.
I want to monitor on Slack all their console.logs (i am logging real error.stack javascript messages as well as purely informative messages like the result of the lambda output "Report Card of the lambda: company=Apple, location=cupertino...").
I could just use a http call to Slack on each lambda but Slack for incoming hooks has a limit of about 1 request per second, after that you get 429 errors if you try to send more than 1 incoming webhook per second... So I thought I'd need to use a queue so that I don't have 300+ lambdas writing to Slack at the same second, but instead controlling the flow from AWS to Slack in a centralized queue called slackQueue.
My idea is to send certain logs (see further down) from Cloudwatchto the SQS slackQueue, and then use this SQS queue as a lambda trigger and sending with this lambda batches of 10 messages (the maximum allowed by AWS; for me 1 message= 1 console.log) concatenated into one big string or array (whatever) to send it to my Slack channel (btw, you can concatenate and send in one call up to 100 slack messages based on Slack limits, so if i could process 100 messages=console.log and concatenate I would but the current batch size limit is 10 for AWS I think ), this way, ensuring I am not sending more than 1 "request" per second to Slack (this request having the content of 10 console.logs).
When I say above "certain logs", it means, I actually I don't want ALL logs to be sent to the queue (because I don't want them on Slack): indeed I don't want the purely "debugging" messages like a console.log("entered function foo"). which are useful during development but have nothing to do on Slack.
As regards some comments: I don't want to use , to my understanding (not expert of AWS) cloudwatch alarms, or metrics filters because they're quite pricy (I'd have those triggered hundreds of times every hour) and don't actually fit my need: I don't want only to read on Slack only when critical problem or a "problem" occurs (like CPU> xxx ...) but really send a regular filtered flow of "almost" all my logs to Slack to read the logs inside Slack instead of inside AWS as Slack is the tool opened all day long, that it's being used for logs/messages coming from other sources than AWS as a centralized place, and that pretty Slack attachment messages formatting is better digested by us. Of course the final lambda (the one sending the messages to slack) would do a bit of formatting to add the italic/bold/etc., and markdown required by slack to have nicely formatted "Slack attachements" but that's not the most complex issue here :)

Answer 1

#Mathieu, I guess you've misunderstood the CloudWatch Events with CloudWatch logs slightly.
What you need is a real time processing of the log data generated by your lambda functions, filter the logs based on a pattern and then store those filtered logs to your Slack for analysis.
But configuring a CloudWatch Event with SQS is similar like a SQS trigger to Lambda. Here, cloudWatch will trigger (send message to) the SQS queue. The content of the message is not your logs but either the default or custom message that you've created.
Solution #1:
Use Subscription filter to filter out the logs as per requirement and subscribe to AWS Kinesis/AWS Lambda/Amazon Kinesis Data Firehouse.
Using the filtered stream (Kinesis), trigger your lambda to push that data to Slack.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Subscriptions.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
Solution #2:
Push your cloudWatch logs to S3.
Create a notification event in S3 on 'ObjectCreated' event and use that to trigger a Lambda function.
In your Lambda function, write the logic to read the logs from S3 (equivalent to reading a file), filter them and push the filtered logs to Slack.