I am using AWS CDK library to create alarm and metric. Both component have been created fine and once deploy cloudformation template using cdk deploy command then components are visible in AWS env.
But sometime things are not executed as per exceptions therefor need to test locally.
Is there any way to test CloudWatch alarm locally ?
Any help would be appreciated.
One way is to write a test where you set the alarm state in a language supported by AWS SDK. i.e Python below
response = client.set_alarm_state(
AlarmName='string',
StateValue='OK'|'ALARM'|'INSUFFICIENT_DATA',
StateReason='string',
StateReasonData='string'
)
or AWS CLI
aws cloudwatch set-alarm-state --alarm-name "{YOUR_ALARM}" --state-reason "Testing alarm" --state-value ALARM
Related
I am running a kube cluster in AWS/EKS. All the alarms are managed in AWS CloudWatch. While that could change in the futur, this a requirement I have to deal with today.
I also have alerts in Prometheus. I wish to "export" them to CloudWatch. What would be the best solution for this? I see only two possibilities so far:
I create a lambda in AWS, which query the ALARM{} metrics to Prometheus, then export the result in CW. I then create an additional alarm in CW monitoring the state of the Prometheus alarme.
I create a webhook in alert manager calling an API gateway in AWS, which would turn on/off the alarm in CW.
Any other suggestions ?
so im trying to run Terraform through CodePipeline. I need to manage a fleet of clusters. It seems CodePipeline is one of the good ways to trigger certain pipelines on some conditions.
I have a very simple requirement - i want to see the terraform execution in real time. i want to expose the CodePipeline run in a way that i can stream this. Is this where EventBridge is used. I tried to look at an EventBridge example here - https://medium.com/hackernoon/monitoring-ci-cd-pipelines-with-amazon-eventbridge-32177e2f2c3e - but it doesnt seem to be streaming run output in real time.
Which event or hook to should i attach to? And is CodePipeline even the right thing to use here ?
Which event or hook to should I attach to?
You're looking at the wrong AWS service. EventBridge is not for streaming log output. It is for discrete events, not a stream.
Your CodePipeline would be using a CodeBuild task to execute Terraform. Your CodeBuild task will be configured to log to AWS CloudWatch Logs. You can view the CloudWatch Logs output in the AWS CloudWatch web console, with the option to poll for new log output.
You can also do the same in a command line console with the aws logs tail command, documented here.
To do the same thing in your own code you would have to write your code to poll the CloudWatch Logs API in an loop.
And is CodePipeline even the right thing to use here?
Yes absolutely
I've got a training job running on Sagemaker. I would like to retrieve instance metrics like MemoryUtilization etc by CLI or boto3 client.
Obviously I can see them in the console. However, I cannot see them in the CLI/API. For example, when running:
aws cloudwatch list-metrics --namespace "AWS/SageMaker"
I can see only metrics regarding endpoint invocation but not any training job related metrics.
Any idea?
Thanks!
I have a view on a PostgreSQL RDS instance that lists any ongoing deadlocks. Ideally, there are no deadlocks in the database, causing the view to show nothing, but on rare occasions, there are.
How would I setup an alarm in Cloudwatch to query this view and raise an alarm if any records return?
I found the cool script on Github specifically for this:
A Serverless MySQL RDS Data Collection script to push Custom Metrics to CloudWatch on AWS
Basically, there are 2 main possibilities to publish any custom metrics on CloudWatch:
Via API
You can run it on a schedule on EC2 instance (AWS example) or as a lambda function (great manual with code examples)
With CloudWatch agent
Here is the pretty example for Monitor your Microsoft SQL Server using custom metrics with Amazon CloudWatch and AWS Systems Manager.
After all, you should set up CloudWatch alarms with Metric Math and relevant thresholds.
It is not possible to configure Amazon CloudWatch to look inside an Amazon RDS database.
You will need some code running somewhere that regularly runs a query on the database and sends a custom metric to Amazon CloudWatch.
For example, you could trigger an AWS Lambda function, or use cron on an Amazon EC2 instance to trigger a script.
My case is the following. I want to launch a cluster during working hours and terminate it after 18:00 and weekends. The clusters will be used for a datascience project. Years ago we would use a boring crontab for this, but these days i prefer to do this with a lambda function.
In boto3 i can launch a cluster (thanks to Jose Quinteiro) and this post describes it very well How to launch and configure an EMR cluster using boto
How can i terminate a cluster in boto3 in the same lambda function as where i start it?
Using AWS CloudWatch event/rule and AWS Lambda function to check for Idle EMR clusters, you complete your goal. You achieve visibility on the AWS Console level and can easily enable and disable it.
Keeping in mind the need for this, I have developed a small framework to achieve that using the 2nd solution mentioned above. This framework is an AWS based solution using AWS CloudWatch and AWS Lambda using a Python script that is using Boto3 to terminate AWS EMR clusters that have been idle for a specified period of time.
You specify the maximum idle time threshold and AWS CloudWatch event/rule triggers an AWS Lambda function that queries all AWS EMR clusters in WAITING state and for each, compares the current time with AWS EMR cluster's ready time in case of no EMR steps added so far or compares the current time with AWS EMR cluster's last step's end time. If the threshold has been compromised, the AWS EMR will be terminated after removing termination protection if enabled. If not, it will skip that AWS EMR cluster.
AWS CloudWatch event/rule will decide how often AWS Lambda function should check for idle AWS EMR clusters.
You can disable the AWS CloudWatch event/rule at any time to disable this framework in a single click without deleting its AWS CloudFormation stack.
AWS Lambda function is using Python 3.7 as its runtime environment.
In your case, while creating the stack, you can specify your required Cron expression and maximum idle EMR cluster threshold in minutes to achieve this.
You can get the code and use it from GitHub here: https://github.com/abdullahkhawer/auto-terminate-idle-emr
Any contributions, improvements and suggestions to this solution will be highly appreciated. :)
You can terminate the cluster using boto3 by using
emr_client = boto3.client('emr')
emr_client.terminate_job_flows(JobFlowIds=[#replace it with cluster Id you want it to close ])
You could create a scheduled event in cloudwatch that triggers the lambda you are using.
Scheduled events use Cron expressions so you will be able to apply the same logic. Once your function is triggered you will need to determine that it is a shutdown trigger from the event input.