I am running a kube cluster in AWS/EKS. All the alarms are managed in AWS CloudWatch. While that could change in the futur, this a requirement I have to deal with today.
I also have alerts in Prometheus. I wish to "export" them to CloudWatch. What would be the best solution for this? I see only two possibilities so far:
I create a lambda in AWS, which query the ALARM{} metrics to Prometheus, then export the result in CW. I then create an additional alarm in CW monitoring the state of the Prometheus alarme.
I create a webhook in alert manager calling an API gateway in AWS, which would turn on/off the alarm in CW.
Any other suggestions ?
Related
I am trying to send an SNS notification when a backup in the backup vault fails twice consecutively. Is there a CloudWatch alarm or any other way to do this in CloudFormation?
You can use CloudWatch metrics for this purpose and then setup alarms based on the thresholds that you need.
You can find the list of Metrics that are emitted to CloudWatch in this document: https://docs.aws.amazon.com/aws-backup/latest/devguide/cloudwatch.html
For instance you can setup an alarm on NumberOfBackupJobsFailed metric.
Was studying about Amazon web services and fundamentals when came across these 2 concepts:
Amazon CloudWatch
Amazon CloudWatch Events
Even while going through the official documents on AWS, I couldn't find a difference between the two even when Amazon mentions that they are different. Excerpt is:
CloudWatch provides you with data and actionable insights to monitor
your applications, respond to system-wide performance changes,
optimize resource utilization, and get a unified view of operational
health. CloudWatch collects monitoring and operational data in the
form of logs, metrics, and events, providing you with a unified view
of AWS resources, applications, and services that run on AWS and
on-premises servers. You can use CloudWatch to detect anomalous behavior in your environments, set alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to keep your applications
running smoothly.
Documentation of AWS CloudWatch
Amazon CloudWatch Events delivers a near real-time stream of system
events that describe changes in Amazon Web Services (AWS) resources.
Using simple rules that you can quickly set up, you can match events
and route them to one or more target functions or streams. CloudWatch
Events becomes aware of operational changes as they occur. CloudWatch
Events responds to these operational changes and takes corrective
action as necessary, by sending messages to respond to the
environment, activating functions, making changes, and capturing
state information.
Documentation of AWS CloudWatch Events
CloudWatch
CloudWatch is a monitoring service for your AWS resources. You can log your log files. By default the resources created within AWS logs in CloudWatch(CW). You can monitor the performance of resources too for example you can monitor how is the CPU utilisation of your EC2 instances. You can set Alarms for your resources
threshold and get an SNS alert on that. For example you can create an Alarm for your DynamoDB if Write capacity is exceeding. You can set an alarm for your billing too. So basically CW is used as a Monitoring solution.
CloudWatch Events
CW Events is also the part of CloudWatch. CloudWatch Events is helpful when you want to schedule something. Say you to want run your lambda every other day, you can create a Rule for that or you want to trigger your lambda by Event Pattern. There are bunch of services supported by CloudWatch Events, you can use anyone of them as your target not just Lambda. Event Buses is used to send your events to multiple accounts also. For example if you have a CICD account and every month you bake new AMI there, to notify all accounts you can use Event Buses, after getting the event from Event Buses other accounts can trigger some important tasks.
I have a view on a PostgreSQL RDS instance that lists any ongoing deadlocks. Ideally, there are no deadlocks in the database, causing the view to show nothing, but on rare occasions, there are.
How would I setup an alarm in Cloudwatch to query this view and raise an alarm if any records return?
I found the cool script on Github specifically for this:
A Serverless MySQL RDS Data Collection script to push Custom Metrics to CloudWatch on AWS
Basically, there are 2 main possibilities to publish any custom metrics on CloudWatch:
Via API
You can run it on a schedule on EC2 instance (AWS example) or as a lambda function (great manual with code examples)
With CloudWatch agent
Here is the pretty example for Monitor your Microsoft SQL Server using custom metrics with Amazon CloudWatch and AWS Systems Manager.
After all, you should set up CloudWatch alarms with Metric Math and relevant thresholds.
It is not possible to configure Amazon CloudWatch to look inside an Amazon RDS database.
You will need some code running somewhere that regularly runs a query on the database and sends a custom metric to Amazon CloudWatch.
For example, you could trigger an AWS Lambda function, or use cron on an Amazon EC2 instance to trigger a script.
With CloudWatch you can monitor applications running on AWS. Is it also possible to monitor an external service?
For example, I have a REST API and I want to get notified once that API is not accessible anymore. Does AWS offer you a monitoring tool for that purpose?
Not Cloudwatch just by itself, but you can use a combination of Cloudwatch and Lambdas to do what you're asking. You can use cloudwatch events to run lambdas on a schedule, something like once every 5 mins.
CloudwatchEvents -> HealthCheck Lambda -> Cloudwatch Custom Metrics
Your lambda can then ping the API you're monitoring the health of, and either send its status to cloudwatch as a custom metric; or potentially if your lambda throws an error when the API fails, the lambda error metric which is already in cloudwatch becomes your API failure metric
Once the metric exists in cloudwatch, either as a custom metric or the lambda metric by proxy, you're able to do usual cloudwatch things like alarms and notifications.
Now there is a simple way to monitor external resources - CloudWatch Synthetics. Just create a canary to regularly monitor a website, API or even validate a multi-step UI flow.
Read more in the docs: CloudWatch > Using Synthetic Monitoring
Amazon CloudWatch supports custom metrics generated by your applications and services that you do not run on AWS. In this way, CloudWatch can be an integrated storage and aggregation point, allowing you to monitor all of the metrics that you collect, and track on a single platform.
There might be more than one way to reach your goal by using the AWS CLI, an API/SDK, or the CloudWatch collectd plugin etc. I'd recommend you take a look at these links for more details: link-1, link-2, link-3, link-4
I would like to create a cloudwatch alarm for the sns metric NumberOfMessagesPublished on a new sns topic. If I attempt to do this before I trigger the topic (and there is no data for it) then the option to create an alarm does not exist in the web console.
How can I do this with the web console? Is using a cli tool the only option?
I noticed that too -- you cannot select a metric until the data has been sent to CloudWatch.
I suspect that CloudWatch actually doesn't have any visibility into metrics until they are sent from the originating service into CloudWatch. Thus, it can't populate the console because it doesn't know what data will be sent. That would be a nice, loosely-coupled design.
My experiments show that it is possible create an alarm via the AWS Command-Line Interface (CLI) on metrics that do not exist -- even within the EC2 namespace. That's the way to go!
I suggest you to use Cloudformation. It is the best way to create resources stack in AWS and overall you can easily experiment with it and delete everything when you are done. Here there is a sample application that uses Cloudformation and Cloudwatch https://github.com/awslabs/cloudwatch-dashboards-cloudformation-sample
The manual solution I went with was to create the sns topic and then use the Publish to topic option in the sns menu to send out a test message to my subscribers. This creates a set of notifications without the need to set off the cloudwatch rule. After this the option appears in the cloudwatch alarm creation menu.
The other CLI options would be better for automation.
Yes, it is possible. The AWS Official Documentation answers your question. Just to quote the relevant lines:
You can create an alarm for a custom metric before you've created that custom metric. For the alarm to be valid, you must include all of the dimensions for the custom metric in addition to the metric namespace and metric name in the alarm definition.