AWS Cloudwatch Alarms - One alarm for multiple dimension values possible? - amazon-web-services

In my AWS organization we have multiple (5+) accounts and a Cloudtrail trail for management events with Cloudwatch log group for all the accounts and regions in the org.
For the log group there is a series of generic metric filters setup, each with a Cloudwatch alarm. These are all tied to one or more SNS topics and associated subscriptions and all out of the box.
For example:
aws logs put-metric-filter --log-group-name CLOUDWATCH-LOG-GROUP --filter-name FILTER-NAME --metric-transformations metricName=METRIC-NAME,metricNamespace=NAMESPACE,metricValue=1 --filter-pattern '{ ($.eventName = ConsoleLogin) && ($.errorMessage = "Failed authentication") }'
Creates a metric filter for any failed logins on console for any of the accounts that sets the metric value to 1 when a match for the filter pattern is found in the log.
Paired with
aws cloudwatch put-metric-alarm --alarm-name METRIC-NAME-alarm --metric-name METRIC-NAME --statistic Sum --period 300 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --evaluation-periods 1 --namespace NAMESPACE --alarm-actions SNS-TOPIC-ARN
This creates a simple alarm that fires an event to the SNS topic.
The major drawback here however is if my org account ID is #1234 and my worker account ID is #4321 and the failed login happens at the worker account, the Alarm that is sent refers only to the org account ID and I need to login and look to figure out where this happened.
Now, I can add a dimension to the metric-transformation to include something to help me differentiate (for example recipientAccountId). That would look something like
aws logs put-metric-filter --log-group-name CLOUDWATCH-LOG-GROUP --filter-name FILTER-NAME --metric-transformations metricName=METRIC-NAME,metricNamespace=NAMESPACE,metricValue=1,dimensions={"Account"=($.recipientAccountId)} --filter-pattern '{ ($.eventName = ConsoleLogin) && ($.errorMessage = "Failed authentication") }'
But, in order for the Cloudwatch alarm to trigger, it appears that the dimension must also be included there, it isn't possible to have one Alarm for all the possible Value that exist. Which then seems to require one Alarm per recipientAccountId (ACCOUNT-ID in the below).
Ex:
aws cloudwatch put-metric-alarm --alarm-name METRIC-NAME-alarm --metric-name METRIC-NAME --statistic Sum --period 300 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --evaluation-periods 1 --dimensions Name=Account,Value=ACCOUNT-ID --namespace NAMESPACE --alarm-actions SNS-TOPIC-ARN
Is this correct?
All of this is really just to include some metadata with the alarm to let me know which Account triggered, is there a better solution?

Related

AWS - CloudWatch Alarm - how to get metric count from AWS CloudWatch Console?

I am struggling to match CloudWatch metric count (as shown in CloudWatch Console ) with count shown in Billing (Estimate for current month ). Attached here are both screen shots.
1) According to below billing screen shot, I should have 10+9.597 = 19.597 metric Alarms
2) According to CloudWatch Console, I have only 3 Alarms. Infact I don't remember of creating more.
I have couple of In Alarms, but in current month ( billing period July ), have only one, see screen shot below.
Couple of In Alarms are shown in red, but they are for last month, which is already billed.
Let me know please if you need more information or any specific screen shot from aws console.
4) Output of aws query
aws cloudwatch describe-alarms --query 'MetricAlarms[*].[AlarmName]' --region us-east-2 > metric-alarams
Music-ReadCapacityUnitsLimit-BasicAlarm
Music-WriteCapacityUnitsLimit-BasicAlarm
TargetTracking-table/TextNote-AlarmHigh-09765769-6e5d-6cab83249c9d
TargetTracking-table/TextNote-AlarmHigh-82c98240-0435-101ab605b404
TargetTracking-table/TextNote-AlarmLow-a0552914-7d04-bd0d74cb9d9a
TargetTracking-table/TextNote-AlarmLow-d4b5d3ff-9b62--b6fafd379abe
TargetTracking-table/TextNote-ProvisionedCapacityHigh-1fc9e0fc--8fc1-c5830689655d
TargetTracking-table/TextNote-ProvisionedCapacityHigh-e2f0ac8b--8826-fd764296f4e8
TargetTracking-table/TextNote-ProvisionedCapacityLow-3e182ade--a070-3d1a515b01a5
TargetTracking-table/TextNote-ProvisionedCapacityLow-e8f2afd9--8ccf-d7dad436cedb
TargetTracking-table/TextNote/index/textNoteSecondaryIndex-AlarmHigh-7693771a-92ee-8cd83a388fec
TargetTracking-table/TextNote/index/textNoteSecondaryIndex-AlarmHigh-b761bab7-a8e6-8386252be6b2
TargetTracking-table/TextNote/index/textNoteSecondaryIndex-AlarmLow-2bc4ee0c-95c6-31721866055d
TargetTracking-table/TextNote/index/textNoteSecondaryIndex-AlarmLow-8b591a75-be8f-ff3209a4b54e
TargetTracking-table/TextNote/index/textNoteSecondaryIndex-ProvisionedCapacityHigh-a369b9dc-8d8f-40d2bb7966cb
TargetTracking-table/TextNote/index/textNoteSecondaryIndex-ProvisionedCapacityHigh-d65c9c16-9313-aed4e691d811
TargetTracking-table/TextNote/index/textNoteSecondaryIndex-ProvisionedCapacityLow-3bd977f5-9acb-b6608ff14d91
TargetTracking-table/TextNote/index/textNoteSecondaryIndex-ProvisionedCapacityLow-f852b0c7-b066-5ac2734d9a65
TargetTracking-table/texthash-AlarmHigh-26e45329-b495-85f3eda0f92e
TargetTracking-table/texthash-AlarmHigh-7b2169a1-d914-50d8b09341d8
TargetTracking-table/texthash-AlarmLow-844f04e2-8e2d-b38bb95e8f1b
TargetTracking-table/texthash-AlarmLow-f7ae2480-7cb8-0bf1adffece6
TargetTracking-table/texthash-ProvisionedCapacityHigh-ad8c3e30-9861-feb73bb2b88f
TargetTracking-table/texthash-ProvisionedCapacityHigh-dc6e4a74-beab-1e55e10f25f6
TargetTracking-table/texthash-ProvisionedCapacityLow-7f34588a-872e-26413a88f905
TargetTracking-table/texthash-ProvisionedCapacityLow-c8bbf607-962b-c7ecd956a6f2
awsec2-i-0fc458fad8fc7fac2-LessThanOrEqualToThreshold-CPUCreditBalance
To view the alarm names of all the billable CW alarms in the us-east-2 region, use the following AWS CLI command:
aws cloudwatch describe-alarms --query 'MetricAlarms[*].[AlarmName]' --region us-east-2
You can delete them using:
aws cloudwatch delete-alarms --region us-east-2 --alarm-names ...

Adding Custom Metric to CloudFormation template

I need to add an alarm the triggers a rollback in CloudFormation. I've settled on adding a custom metric that can be assigned manually. I'm looking at the documentation here. This looks like it would work but I need it to be added to the template. Is this possible? The way I'm thinking initially the value would be 0 failures like below
aws cloudwatch put-metric-data --metric-name Failures --namespace MyNamespace --value 0
when I want to trigger a rollback I would set it to 1
aws cloudwatch put-metric-data --metric-name Failures --namespace MyNamespace --value 1
But I need this to be in the template so that I can have access to the correct/dynamic value of the namespace when defining the alarm.
You can use cloudformation simple template to add metrics
resource "AWS::Logs::MetricFilter" specifies a metric filter that
describes how CloudWatch Logs extracts information from logs and
transforms it into Amazon CloudWatch metrics. If you have multiple
metric filters that are associated with a log group, all the filters
are applied to the log streams in that group.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-metricfilter.html
sample
404MetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName:
Ref: "myLogGroup"
FilterPattern: "[ip, identity, user_id, timestamp, request, status_code = 404, size]"
MetricTransformations:
-
MetricValue: "1"
MetricNamespace: "WebServer/404s"
MetricName: "404Count"
unless you want to do this via a cloudformation custom resource
you can use this as well Running bash commands in AWS CloudFormation templates
With the above you can run your command like
Resources:
CommandRunner:
Type: AWSUtility::CloudFormation::CommandRunner
Properties:
Command: 'aws cloudwatch put-metric-data --metric-name Failures --namespace MyNamespace --value 1'

Execute Lambda function based on threshold

I have created a Lambda function which will be triggered via a subscription to a CloudWatch Log Pattern and the function will in-turn pass the logs to a web-hook (Refer https://gist.github.com/tomfa/f4e090cbaff0189eba17c0fc301c63db).
Now, I need this lambda function to EXECUTE only if the the function is called "x" times in "y" minutes.
Is it possible to disable/enable a lambda through SNS. Another idea is to
1. Create CloudWatch Events on State Change
2. Subscribe this to a SNS which will
enables the lambda, if state goes from OK to ALARM
disables the lambda, if state goes back to OK
You can use CloudWatch Events to send a message to an Amazon SNS topic on a schedule. make sure you are in correct region as as CloudWatch Events is not available in every region.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html
How to configure Cloudwatch :
AWS Lambda Scheduled Tasks
run scheduled task in AWS without cron
AWS Lambda Scheduled Tasks
Use CloudWatch and get metrics about the lambda invocation and error and you can find successful call and error , threshold count. now you can use AWS SDK
https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-data.html
export.handler = function(event, context, callback) {
apiCall().then(resp => callback(null, resp).catch(err => callback(err));
}
You could create a custom CloudWatch Metric based of your search filter of the CloudWatch Logs
Examples of this can be found in the Amazon CloudWatch Logs User Guide
Count Log Events
aws logs put-metric-filter \
--log-group-name MyApp/access.log \
--filter-name EventCount \
--filter-pattern "" \
--metric-transformations \
metricName=MyAppEventCount,metricNamespace=MyNamespace,metricValue=1,defaultValue=0
Count Occurrences
aws logs put-metric-filter \
--log-group-name MyApp/message.log \
--filter-name MyAppErrorCount \
--filter-pattern 'Error' \
--metric-transformations \
metricName=ErrorCount,metricNamespace=MyNamespace,metricValue=1,defaultValue=0
Then you can go in and create a CloudWatch Alarm that will fire based on x of these events being logged in y time span. The CloudWatch Alarm can send a message to an SNS topic that triggers your Lambda function

how to properly use aws rds describe-events

I'm trying to view all events related to all of my RDS instance for audit purposes.
I am using this
aws rds describe-events --source-type db-instance
But I am getting an empty result
{
"Events": []
}
Even if I include a --source-identifier, it still yields an empty result
by default it retrieves result for the last an hour, if there are no events to display then it might be empty.
so include flag --duration 3600 (in minutes) to get one day events .. if you go higher then you will get the results ( max 14 days)
Just experienced the same: adding --start-time and --end-time fixed it for me. For some reason --duration did not.
So something like this should work:
aws rds describe-events --start-time 2020-07-24T00:00Z --end-time 2020-07-24T18:00Z

Can I send a CloudWatch Metrics namespace to S3?

I want to send my custom cloudWatch metrics namespace filtered events to an s3 bucket to monitor specific events. I have data coming in from cloudTrail and am filtering the logs I want through metrics in a custom namespace.
I've looked into creating a cloudWatch subscription which will let me filter by specific options like eventName. I'm not sure if this is the only option though.
This is the AWS filtering subscription setup and I've done a similar filtering in cloudWatch by specifying {$.userIdentity.type = Root} with other filters set. So I imagine if I need to go the route of doing a subscription I would do multiple --filter-pattern {$.userIdentity.type = Root} / lines
aws logs put-subscription-filter \
--log-group-name "CloudTrail" \
--filter-name "RecipientStream" \
--filter-pattern "{$.userIdentity.type = Root}" \
--destination-arn "arn:aws:logs:region:999999999999:destination:testDestination"
I have a cloudWatch log that is pulling my cloudTrail logs and filtering them through metrics. I would like to sends those results to an s3 bucket I have in place to store the filtered logs from my metrics name space. Is this possible or do I need to use a subscription configuration?