Using Grafana's CloudWatch data source and a little InfluxDB magic, I can pull many metrics from my live environment; like CPU utilisation, memory utilisation, host count, thread count, ect etc.
These metrics will make more sense if I can spot the moments of live deployments on that graph.ELB Health Host Count metric kinda helps but does not show deployments, rather shows auto scale activities.
I can't find any metrics in AWS CloudWatch adapter for CodeDeploy. Dooes anybody has a way of doing this?
(My Env: Sprint Boot app on Docker containers deployed on AWS Fargate using CodeDeploy)
You can push datapoints into a CloudWatch metric using the "put-metric-data" aws cli call [1]. You can call this command from the AppSpec file hooks like BeforeInstall and AfterInstall. Make sure the EC2 instance role has the requisite permissions.
[1] https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/put-metric-data.html
[2] https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-example.html#appspec-file-example-server
Related
Looking into adding autoscaling of a portion of our application using AWS simple message queuing which would launch EC2 on-demand or spot instances based on queue backlog.
One question I had, is how do you deal with collecting logs from autoscaled instances? New instances are spun up based on an image, but then they are shut down when complete. Currently, if there is an issue with one of our services, which causes it to crash, we have a system to automatically restart the service, but the logs and core dump files are there to review. If we switch to an autoscaling system, where new instance are spun up, how do you get logs and core dump files when there is a failure? Particularly if the instance is spun down.
Good practice is to ship these logs and aggregate them somewhere else, and there are many services such as DataDog and Rapid7 which will do this for you at a cost.
AWS however provides CloudWatch logs, which gives you a central place to store and view logs. It also allows you then to give users access to logs on the AWS console without them having to ssh onto a server.
Shipping your logs to CloudWatch logs requires the installation of the CloudWatch agent on your server and specifying in the config which logs to ship.
You could install the CloudWatch agent once and create an AMI of that server to use in your autoscaling group, or install and configure the CloudWatch agent in userdata for every time a server is spun up.
All the information you need to get started can be found here:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
I'm trying to automate the turning on and off process of Redis Cluster in aws. I saw the following link for reference (https://forums.aws.amazon.com/thread.jspa?threadID=149772). Is there a way to do it via cloudwatch ?
I am very new to aws platform.
Check the documentation regarding scale in/out
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/redis-cluster-resharding-online.html It also has commands to reshard a cluster manually.
Check CloudWatch metrics from the Redis cluster. https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.HostLevel.html and https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.Redis.html Choose the metrics that will trigger autoscaling
You can trigger an AWS Lambda on some event for a metric https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
From the Lambda you cal call aws cli to reshard the cluster as described in 1. Example: https://alestic.com/2016/11/aws-lambda-awscli/
If you need to turn off the cluster completely, instead of the resharding commands just use https://docs.aws.amazon.com/cli/latest/reference/elasticache/delete-cache-cluster.html
How can I see the used resources per ECS container? Now im able to see the resources from the whole cluster but I want to see it per container.
I know I can see it via docker stats when I SSH to my cluster, but I prefer Cloudwatch for it.
ECS doesn't provide per container metrics on CloudWatch.
You could install the CloudWatch Agent in your containers via the Docker file. You can configure CloudWatch agent to publish a bunch of host level metrics.
However, note that all of these metrics would be Custom Metrics and would cost a lot.
I have an AWS account, and have several EC2 servers and an ElasticSearch domain set up to take the syslogs from these servers. However, in Cloudwatch and when investigating a specific server instance in the EC2 control panel, I see specific metrics and graphs for things like CPU, memory load, storage use, etc. Is there some way I can pipe this information into my ElasticSearch as well?
Set up Logstash and use this plugin https://github.com/EagerELK/logstash-input-cloudwatch
Or go the other way and use AWS Logs agent to put your syslogs into Cloudwatch and stop using ElasticSearch
I'm running a wordpress off of AWS, and I can't figure out how to monitor if the volume is running out of capacity. There are many options for monitoring other things, but I just want to know when we run low on space.
There are no built in metrics for disk space - Because Amazon is looking at this 'from the outside', they don't know what you've done with an EBS volume: it could be part of a raid set, formatted in some exotic format etc.
Amazon has a system called CloudWatch that can be used to monitor AWS systems and resources. Luckily, CloudWatch supports custom metrics: you can use the CloudWatch APIs to add any data you want (and then create alerts off them).
Amazon provides some sample scripts that will publish (among other things) disk space utilisation to Cloudwatch. Do be aware that there is a small charge associated with using custom metrics - $0.50 per metric per month.
Selected answer will no longer work because:
The CloudWatch monitoring scripts are deprecated. We recommend that you use the CloudWatch agent to collect metrics and logs. - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html
I found this guide helpful at setting up custom cloudwatch metric such as disk monitoring:
https://marbot.io/blog/monitoring-ec2-disk-usage.html
The only thing that the above is missing is setting up IAM role and attaching it to the EC2, guide for this is here (do this first, then follow the blog post): https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent.html
Per recommendation, summary steps below:
Set up IAM role per link here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent.html
Attach it to EC2 you want to monitor
ssh into your EC2
cd /tmp
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm (to download the agent rpm)
sudo rpm -U amazon-cloudwatch-agent.rpm (to install)
sudo vi /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json (to edit config)
paste the config (in the link, no changes necessary)
sudo systemctl restart amazon-cloudwatch-agent (restart agent)
tail -f /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log (see if its working)
Navigate to cloudwatch console and set up alarm based on custom metric that will show up as CWAgent (thats the name specified in the config, more detail in the link on exact way to set up alarm)
I agree with Frederick Cheung's answer; however, I found this utility for Windows instances: http://www.eleven41.com/labs/cloudwatch-monitor-for-windows/. It runs as a service and will send the disk space and memory usage metrics to CloudWatch for you. From there, you'd just need to set up the alarm in CloudWatch.
I agree with Christopher Hinkle's agreement with Frederick Cheung's answer :)
Instead of using a utility where you still have to setup all the alerts yourself, try using Blue Matador. It will set up all the alerts for you, on all the mount points, on all the servers.
That said, you should also watch a lot more metrics than just the disk space. There's a guide called How to Monitor Amazon EBS with CloudWatch that goes over all the metrics and how you should approach monitoring them (if you don't use Blue Matador to automate them).