I'm trying to monitor the change in traffic on our application between days. We currently have a metric plotting the sum of 1 day between sequential days, however this comes with a 24 hr delay as we need to wait for the current day's traffic.
I'm trying to figure out a way to produce a graph that sums data over a 1hr period, and subtracts the same datapoint from 24 hrs in the past.
I can't figure out a way to do this with metric math. Anyone have any suggestions? Preferably within AWS, thanks!
You can set up an export of the CloudWatch logs into S3. Then you can use Athena to write SQL queries of arbitrary complexity, which you can then visualize using QuickSight.
It is not going to be as near-realtime as CloudWatch: there may be about a few hours' lag in S3 export. For a real-time solution, you may consider streaming logs into a Kinesis Datastream and then create a custom Lambda to process the logs and push them into a QuickSight dataset.
Related
Is there a quick way to check how many data (volume wise, GBs, TBs etc) did my specific DMS task transfered for example within last month?
I can't find any note in the documentation on that, I could probably try with boto3 but want to double check first. Thanks for help!
Even with Boto3, you can check the API - DescribeReplicationTasks but likely, there is no information about your data transfer.
Reference: https://docs.aws.amazon.com/dms/latest/APIReference/API_DescribeReplicationTasks.html
If you have only 1 data replication task that is associated with only 1 replication instance, you can check that replication instance's network metric via CloudWatch metric. From CloudWatch metrics, AWS DMS namespace, there will be several network metrics such as NetworkTransitThroughput or NetworkReceiveThroughput. You can choose one and try as below:
Statistic: Sum
Period: 30 days (or up to you)
And you have a 30DAYS_THROUGHPUT.
As I know that AWS CloudWatch store metric datapoint for 15 months with variable data point sampling like
01 sec DP for 3 hours
60 sec DP for 15 Days
300 sec DP for 63 Days and so on
But there is no data available before 15 months also the available data is not helpful due to large data sample size
In my case, I am looking for some metric of ALB like
Max Active Connection per minute,
Max Request count per minute(RPM) &
New connection count
before 15 months, somehow I was able to identify max RPM via parsing ALB access logs.
I am looking for some solution which is easy to use and storing data in time serise DB (ex: influx DB)
Use&configure Telegraf Amazon CloudWatch Statistics Input Plugin and save CloudWatch metrics into InfluxDB.
I have a use case where I'm expecting some write volume to my Dynamodb table everyday around 3pm from a kinesis stream, the streaming takes about 20 mins.
So in order to make sure the kinesis successfully put data to my Dynamodb table, I want to use CloudWatch to monitor such behavior on my Dynamodb and configure a alarm saying, if everyday around 3pm to 4pm, there's no write traffic to my Dynamodb table, alarm.
Does Cloudwatch support such use case?
I have tried to set up an alarm saying I'm expecting write volume to my table on a daily basis, if not, alarm. However, I want to shorten this period since I know if there's no write volume around 3pm, something must be wrong and actions need to be taken. I don't want to wait till the day is over.
I know you can use math on metric, but after reading the document: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html, I don't think my use case is supported, unless I missed something?
Appreciate any suggestions here! Thanks!
I have a Streaming Data use case where for every record inserted into a Kinesis Stream, I would like to calculate a value based on the last 24 hours worth of records.
Now, Kinesis Analytics seems to fit this bill nicely. I can use a WINDOW with a RANGE INTERVAL '1' DAY PRECEDING to get my aggregation. My Kinesis Stream is set to persist records for 26 hours.
My conflict comes from the Best Practices documentation.
Specifically:
In your SQL statement, we recommend that you do not specify time-based window that is longer than one hour for the following reasons:
If an application needs to be restarted, either because you updated the application or for Amazon Kinesis Data Analytics internal reasons, all data included in the window must be read again from the streaming data source. This will take time before Amazon Kinesis Data Analytics can emit output for that window.
Amazon Kinesis Data Analytics must maintain everything related to the application's state, including relevant data, for the duration. This will consume significant Amazon Kinesis Data Analytics processing units.
I want to be sure I understand the consequences of this.
Say I proceed with my plans and implement this using Kinesis Analytics. Then for whatever reason my Kinesis Analytics Application has to be restarted. For context, my application deals with approximately 1 million records a day and each record is approximately 550 bytes.
If my Kinesis Analytics Application restarts, what will be I be dealing with since I would be ignoring this recommendation:
we recommend that you do not specify time-based window that is longer than one hour
I have also considered skipping Kinesis Analytics and just feeding my rows into RDS (postgres - Single Deployment) and triggering a lambda to run my calculations or doing something with Redis (still exploring those consequences).
I can't seem to find this info anywhere.
Specifically, I'm looking at metrics like NumberOfMessagesPublished and NumberOfNotificationsDelivered. How far back does CloudWatch retain this data?
2 weeks according to Monitoring Your Instances Using CloudWatch
These statistics are recorded for a period of two weeks, so that you
can access historical information and gain a better perspective on how
your web application or service is performing.
If you want to archive metrics beyond 2 weeks
If you want to archive metrics beyond 2 weeks you can do so by calling
mon-get-stats command from the command line and storing the results in
Amazon S3 or Amazon SimpleDB.