AWS SNS, sending "continuous" notifications to slack - amazon-web-services

Question

On a high level I wrote a lambda that notifies slack when there's an error or not.
From an aws tool-chain perspective, the tech design looks like this:
Acceptance Criteria (in BDD style)
Scenario: As an engineer I want to get notified if my lambda PASSED or FAILED whenever it executes
Given I have a lambda function that runs on a schedule (9am everyday)
Given I have a metric filter that looks for the string "error" in the logs
And I created an alarm that does the following:
# +------------------------+--------------+
# | ALARM |
# +------------------------+--------------+
# | Statistic | Sum |
# | Period | 5 minutes |
# | Threshold type | Static |
# | Alarm condition | >= threshold |
# | Threshold value | 1 |
# | Datapoints to Alarm | 1 of 1 |
# | missing data treatment | ignore |
# | Alarm State | in Alarm |
# +------------------------+--------------+
And I created another alarm that does the following:
# +------------------------+--------------+
# | OK |
# +------------------------+--------------+
# | Statistic | Sum |
# | Period | 5 minutes |
# | Threshold type | Static |
# | Alarm condition | <= threshold |
# | Threshold value | 1 |
# | Datapoints to Alarm | 1 of 1 |
# | missing data treatment | good |
# | Alarm State | OK |
# +------------------------+--------------+
Then EVERY TIME time my function executes without "error" Then I should get "OK"
Then EVERY TIME time my function executes with "error" then I should get "ALARM"
The actual behavior is it will send out a notification only ONCE, and it will only send again when the alarm type changes i.e.
ALARM -> OK
OK -> ALARM
I don't seem to get notifications for this pattern
ALARM -> ALRM
OK -> OK
Ideally I want to receive a notification every time function executes

Answer 1

There’s no need to use a CloudWatch alarm. If you want one message every time the Lambda executes, you should just publish the SNS message as the last thing inside your Lambda function.
try {
// existing code goes here...
snsClient.publish("my-chatbot-topic", "Some success message");
} catch (Exception e) {
snsClient.publish("my-chatbot-topic", "Some error message");
// rethrow the exception so that the lambda still fails for this
throw e;
}

Answer 2

As per AWS documentations:
Alarms invoke actions for sustained state changes only. CloudWatch
alarms don't invoke actions simply because they are in a particular
state, the state must have changed and been maintained for a specified
number of periods.
One solution is to stream the CW logs to a lambda function that sends the SNS messages.
With a fast search I found this code that does exactly this (I didn't try myself): https://github.com/codemonauts/aws-cloudwatch-stream-filter-sns-gateway