Just wondering if the AWS cloudwatch runs on the same VPC where i have all my applications are running?
Is there any chance that AWS cloudwatch might go down and we may loose the monitoring capability?
Do we need to have a monitoring mechanism to check the Cloudwatch health?
Thanks
AWS Cloudwatch isn't run on your instances. Its infrastructure is fully managed by Amazon and independent from your VPC. You can see it as a SaaS (Software as a Service).
So you don't have to worry about that. For more informations, please see: https://aws.amazon.com/cloudwatch/
Cloudwatch collects data from the host OS, where your VMs are actually running.
If the physical server had a significant issue both cloudwatch and your VM would go down but in that case the VM would get started automatically on another physical server. In such a case, recovery would be usually quite quickly.
You don't need to check Cloudwatch at all because AWS handles that but you could add alerts for things such as CPU usage on your VMs.
Because Cloudwatch doesn't run on your machines it can't know some things such as memory usage, disk space usage or others so if you need more advanced monitoring capabilities you might consider running something like collectd inside your virtual machine.
Just wondering if the AWS cloudwatch runs on the same VPC where i have all my applications are running?
If you chose to install CloudWatch Agent on your EC2 then only it runs in your EC2 and thus in the VPC your EC2 is provisioned.
CloudWatch service that publishes/maintain logs, metrics, alarms etc is managed by AWS and runs outside your VPC.
CloudWatch has a SLA of 99.9%
https://aws.amazon.com/cloudwatch/sla/
Is there any chance that AWS cloudwatch might go down and we may loose the monitoring capability?
CloudWatch like any other service can have outages and it did have some in the past but I have never seen any data getting lost, only temporarily not being available or slow to retrieve during the outage.
Do we need to have a monitoring mechanism to check the Cloudwatch health?
SLA is already 99.9% for CloudWatch Service so chances of catching a blip is very rare on your own monitoring mechanism.
If you are using CloudWatch Agent then consider checking health of agent to make sure it is in running state (you can use AWS System Manager Run command).
Related
I am planning to migrate our app to AWS Fargate and so want to set up logging for the same as well and store all the logs in cloudwatch. I could see we have two options in Fargate - either use default awslogs log driver or use AWS Firelens to gather logs. I read the AWS documentation but unfortunately still not able to figure out which option to use and when precisely. Also, can someone advise on the cost side as well- which option costs how much between using awslogs driver vs aws firelens to send logs to cloudwatch in the same account? [I am looking for easy and efficient and cost effective option]
Is it fair to say in general,we use AWS Firelnes when you want to send logs data to non-AWS tools like elastic stack or datadog etc vs use awslogs driver when sending logs to cloudwatch
Can someone please advise?
Using Fargate launch type and want to use CloudWatch: You have to use awslog driver in your task definition. You can find more information about the CloudWatch pricing here. CloudWatch has a free tier and anything after the free tier cap (metrics, dashboards, alarms, logs, events, etc) has a different pricing calculation. For example, first 10K metrics cost 0.3$ in most regions, but the next 240K will cost 0.1$ while events are priced at 1$ for 1 million
Using Fargate and don't want to use CloudWatch: Use AWS FireLens to push container logs to third party logstorage system. The cost of 3rd party log storage system would come in play here. Data Dog/AppDynamics and others usually offer membership packages (Free/Premium/Enterprise etc). Unlike CloudWatch, each package will give you different capabilities. For example, on the free tier in DataDog you do not have alerts. Also non-AWS native monitoring tools are priced per host/CPU core for a specific amount of hours.
Firelens would also make sense if you want to ship to CloudWatch but want to do upfront filtering versus sending everything to CloudWatch
If you simply want all the log output from your ECS Fargate tasks to go to AWS CloudWatch Logs, then use the awslogs driver. This basically works "out of the box" with no further configuration needed on your part. This is the easiest solution. The only additional cost will be the cost of CloudWatch Logs, detailed in the "Logs" tab here.
If you want to send logs to some other logging service, like Splunk, then use the Firelens driver, and provide a Firelens configuration file that tells Firelens where to send your logs. There is no added cost for using the Firelens driver, but of course there is the added cost of whatever target services you configure Firelens to send your logs to.
I am a bit confused in monitoring of EC2 with and without Cloudwatch agent. As far as I know, Cloudwatch agent does not get installed by default on EC2 linux but some basic system metrics like CPU usage can still be monitored and shown in Cloudwatch.
My questions
If I need to monitor memory usage which is not being monitored by default in EC2 now, should I just setup Cloudwatch agent and memory usage can be published to CloudWatch metrics?
What about I don't setup Cloudwatch agent but just enable detailed monitoring? Can memory usage be monitored by just enabling detailed monitoring without cloudwatch agent?
If I need to monitor memory usage which is not being monitored by
default in EC2 now, should I just setup Cloudwatch agent and memory
usage can be published to CloudWatch metrics?
Yes, this is the correct way to monitor OS level metrics on your EC2 instances.
What about I don't setup Cloudwatch agent but just enable detailed
monitoring? Can memory usage be monitored by just enabling detailed
monitoring without cloudwatch agent?
Detailed monitoring just changes the monitoring interval from 5 minutes to 1 minute, it doesn't enable additional metrics. CloudWatch can't reach into the EC2 operating system to see things like memory usage, so you have to install the CloudWatch agent on the server to monitor memory usage.
CloudWatch can be used to perform system monitoring(AWS resources) & application monitoring on AWS resources.
Does CloudWatch rely on SNMP mib(of AWS instance) to monitor the metrics?
No. CloudWatch collects all these metrics from the hypervisor on the physical host that is running your instance. It does not communicate with any processes running inside your instance, SNMP or otherwise.
No definitely not. CloudWatch standard EC2metrics are hypervisor-level and not reported by the instance. If you put the instance in a Security Group that blocks ALL outbound network traffic, the metrics still update. That also explains why CloudWatch metrics don’t include any OS-level stuff like memory utilization, free disk space, etc.
You can install a CloudWatch agent on the instance that pushes additional metrics (like memory utilization and free disk space), but that communicates over HTTPS like all the other AWS APIs. The CloudWatch API is well documented here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/Welcome.html
Furthermore, SNMP is typically a LAN protocol. So if CloudWatch was using SNMP it would have an IP address in your VPC. CloudWatch does not have a VPC endpoint by default.
No, CloudWatch doesn't support SNMP monitoring. But you can use some external tools, which will read SNMP metrics and it will put metrics to the CloudWatch.
For example, you can use customized collectd with the SNMP plugin.
No, CloudWatch can´t get SNMP directly, but you can use other solutions to "collect" traditional SNMP data into CloudWatch.
Follow reference:
https://aws.amazon.com/blogs/mt/snmp-monitoring-using-amazon-cloudwatch-and-elastic-logstash/
I am wanting to deploy a Django webapp with a PostgreSQL database to AWS Elastic Beanstalk using this tutorial, but I am so confused about pricing. It says it uses services in the AWS Free Tier, but those seem to be limited to a certain number of hours a month, so how do I make sure I don't go above that threshold? And how do I make sure I'm only using free services? They even require a card on file, so it seems really hard to make sure I don't get charged.
You can do the following configuration to make sure you use AWS Elastic Beankstalk for one year free.
Use only Micro instances for the WebServer and RDS instance.
Limit the scaling of the WebServer maximum to 1 or use Standalone deployment without autoscaling.
When selecting storage, use less than 30GB for EBS and don't enable Provision Throughput.
Apart from these, there are usage base costs for Network, EBS IOPS & etc which includes a free quota and the cost is not considerable when it comes to light use cases.
The AWS Free Tier allows AWS accounts to use a certain amount of services for no charge. Any usage beyond the free tier limits will result in a charge on your credit card.
The Free Tier is intended to provide a trial of AWS services. It is not intended for production use, nor is there any guaranteed way to stay within the free limits. It is up to you to monitor your usage.
There is no such thing as a totally free AWS account.
I have found "Cost Management Preferences" -> "Receive Free Tier Usage Alerts" setting in Billing preferences menu. Hopefully this will be enough for a small personal projects with low usage. I would guess it is not enough for large projects since this is only a notification.
In short, you can absolutely make sure that your app stays free, just not from within the AWS interface. You'll have to use your own usage monitoring to ensure you stay within the free limits as others state.
As Ashan said, this is a pretty silly approach since fees are nominal and the alternative is a loss of service, however, AWS does offer APIs to help you do this through CloudWatch.
CloudWatch exposes pretty much all of the billable metrics on a service-by-service basis, for example here are the metrics for EC2, and here are the metrics for S3. After starting your services through beanstalk, just look up all the services you're using via the billing page of the AWS console, look up the CloudWatch APIs for each, then check them.
At least for EC2, there are even customizable alarms and actions, including shutting down the instance. See the Monitoring tab at the bottom of the EC2 console. Not sure, but you might have to manually throw status updates to their status system for some of the other metrics. If so, it's not that difficult. You'd set up an access key for some IAM identity so you can check CloudWatch stuff from command line. Then, you'd write a watchdog script to run on that instance using AWSCLI to regularly ping CloudWatch and call your shutdown code or modify your status if you're over some percentage of your quota.
Suppose I have an EC2 instance, which I understood is a VM instance. So if I enable CloudWatch for this EC2 instance, should this monitoring capabilitiy offered by CloudWatch added into my EC2 instance, or it is just running in the hypervisor like XEN?
Thanks.
CloudWatch monitoring is always enabled by default for every EC2 instance at 5-min granularity. What you can enable is detailed monitoring which means you get 1-min observation granularity and aggregate metrics. Default monitoring at 5-min level is free, but detailed monitoring costs money.
Out-of-the-box CloudWatch metrics are measured at hypervisor level and you do not need to do anything to turn them on. See more info on what metrics are available here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch.html
Things like memory utilization and disk space can't be measured at hypervisor level so CloudWatch distributes a simple package with scripts that can be installed on the instance (Linux or Windows.) Those scripts report the data as custom metrics which also costs money. See http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts.html
It is monitored at the hypervisor layer. Aamazon generally will not look into the instance at VM layer so they can't monitor some feature such as memroy usage at the VM.