AWS Cloudwatch Alarm on Error fails on Insufficient Data - amazon-web-services

Question

I try to create an alarm if my a json log message is of error log level. My filter works fine but when I create my alarm, it always fails on insufficient data. Seemingly, because there are no errors.
Any ideas?

Answer 1

The way around this was to define two metrics with the same name but with inverse filters. The filter that matches error level log messages must return a metric value of 1, while the second filter should match all messages, or at least one message within the time period and return a metric value of 0. The presence of the 0 value avoids the insufficient data error.
When the alarm is created from the metric, both filters are combined. If a sum statistic is applied and a alarm rule of >0 applied, the alarm will trigger only when error messages arrive and not run into insufficient data.
Here is an example using the boto3 client:
import boto3
client = boto3.client('logs')
logGroupName = 'myLogGroup'
# create this SNS topic with your email subscription...
env['aws_sns_arn_error_email'] = 'arn:aws:sns:eu-west-1:1234567:log_error'
env['sys_type'] = 'production'
metricsNamespace = 'LogMetrics'
metricName = 'ErrorCount' + "_%(sys_type)s" % env
print colors.cyan('Put metric $(metricName)s' % env)
cloudwatch_client = boto3.client('cloudwatch')
response = cloudwatch_client.put_metric_data(
Namespace=metricsNamespace,
MetricData=[
{
'MetricName': metricName,
'Unit': 'Count',
'Value': 1
},
]
)
logs_client = boto3.client('logs')
print colors.cyan('Put metric filter $.levelname-ERROR')
logs_client.put_metric_filter(
logGroupName=env.log_group_name_ea,
filterName='levelname-ERROR',
filterPattern='{ $.levelname = "ERROR" }',
metricTransformations=[
{
'metricNamespace': metricsNamespace,
'metricValue': '0',
'metricName': metricName,
}]
)
print colors.cyan('Put metric filter catchAll')
logs_client.put_metric_filter(
logGroupName=env.log_group_name_ea,
filterName="catchAll",
filterPattern='',
metricTransformations=[
{
'metricNamespace': metricsNamespace,
'metricValue': '1',
'metricName': metricName,
}]
)
print colors.cyan('Put metric alarm, email on error')
response = cloudwatch_client.put_metric_alarm(
AlarmName='email on error',
AlarmDescription='email on error',
ActionsEnabled=True,
AlarmActions=[
env.aws_sns_arn_error_email,
],
MetricName=metricName,
Namespace=metricsNamespace,
Statistic='Sum',
Period=300,
Unit='Count',
EvaluationPeriods=1,
Threshold=0,
ComparisonOperator='GreaterThanThreshold'
)