I just started using AWS services. I want to receive notifications if any service usage exceeds limit. After searching for the options I found that same can be achieved suing AWS Cloudwatch alarm and AWS Limit Monitor using AWS CloudFormation. My question is, will i be charged if i use these services to receive notifications?
Yes, you can setup all kinds of notifications to keep a handle on what you are being billed, but that doesn't stop you from actually getting billed if you exceed your limits.
For example I have alerts to notify me when I reach 25%, 50%, 75% and 100% of my typical monthly spend - so I roughly should get one notification each week - but a lot can happen between when you get sent the notification, and when you take action - especially if, for example, someone got access to your account and started crypto-mining on some big ec2 instances.
Related
Is there any data limitation on aws cloudwatch logs to send the logs , because in my case I am getting the logs data 6 million records per 3 days from my application. So is aws cloudwatch logs will able to handle that much data?
Check out the aws quotas page. Not sure what you mean by "60lac" but the limits on CloudWatch are more than adequate for the majority of use cases.
There is no published limit on the overall data volume held. There'll be a practical limit somewhere but it won't be hit by a single AWS customer. If you're using the putLogEvents API you could be constrained by the limit of 5 requests per second per log stream, in which case consider using more streams or larger batches of events (up to 1MB).
I have a AWS Lambda function using an AWS SQS trigger to pull messages, process them with an AWS Comprehend endpoint, and put the output in AWS S3. The AWS Comprehend endpoint has a rate limit which goes up and down throughout the day based off something I can control. The fastest way to process my data, which also optimizes the costs I am paying for the AWS Comprehend endpoint to be up, is to set concurrency high enough that I get throttling errors returned from the api. This however comes with the caveat, that I am paying for more AWS Lambda invocations, the flip side being, that to optimize the costs I am paying for AWS Lambda, I want 0 throttling errors.
Is it possible to set up autoscaling for the concurrency limit of the lambda such that it will increase if it isn't getting any throttling errors, but decrease if it is getting too many?
Very interesting use case.
Let me start by pointing out something that I found out the hard way in an almost 4 hour long call with AWS Tech Support after being puzzled for a couple days.
With SQS acting as a trigger for AWS Lambda, the concurrency cannot go beyond 1K. Even if the concurrency of Lambda is set at a higher limit.
There is now a detailed post on this over at Knowledge Center.
With that out of the way and assuming you are under 1K limit at any given point in time and so only have to use one SQS queue, here is what I feel can be explored:
Either use an existing cloudwatch metric (via Comprehend) or publish a new metric that is indicative of the load that you can handle at any given point in time. you can then use this to set an appropriate concurrency limit for the lambda function. This would ensure that even if you have SQS queue flooded with messages to be processed, lambda picks them up at the rate at which it can actually be processed.
Please Note: This comes out of my own philosophy of being proactive vs being reactive. I would not wait for something to fail to trigger other processes eg invocation errors in this case to adjust concurrency. System failures should be rare and actually raise alarm (if not panic!) rather than being normal that occurs a couple of times a day !
To build up on that, if possible I would suggest that you approach this the other way around i.e. scale Comprehend processing limit and AWS Lambda concurrency based on the messages in the SQS queue (backlog) or a combination of this backlog and the time of the day etc. This way, if every part of your pipeline is a function of the amount of backlog in the Queue, you can be rest assured that you are not spending more than you have at any given point in time.
More importantly, you always have capacity in place should the need arise or something out of normal happens.
I am using an ec2 linux instance on aws, I am trying to stay inside the Free Usage Tier, but today I have noticed a charge for DataTransfer cost in Billing Management.
I don't know what the limits for DataTransfer are but I'm using aws for the host of a telegram bot
I would like to stay within the limits of the Free Usage Tier, is there a way to receive notifications just before exceeding the limit for data transfers or simply setting a limit not to be exceeded?
You can set up billing alerts, at any dollar amount you want, but keep in mind that if your service is getting slammed with requests, your account will still keep accumulating charges even after the alert is sent out - so depending on how long it takes you to respond to the alert (and turn off services for example), and how fast services are being consumed, you may still end up with an unexpected bill.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html
I'm trying to send SMS via Amazon SNS, but since yesterday they are failing, I see a delivery rate of 0% in the AWS Console. As I understand, I would have to pay 29$ upfront to report a failing service.
Is there a possibility to get via CloudWatch the reason for the failed SMS service, or is there another method to complain to Amazon?
Since yesterday, the messages have completely stopped coming so you might have reached the SNS SMS spend limit for your account. By default it is 1 USD for a AWS account in a AWS region. You can check Delivery Status logs which might say 'No Quota Left for Account'.You should get your limit increased with limit increase case for which you do not have to pay 29 dollars.
But in future if you want to prevent this i.e. if you want to get notified earlier before you reach the actual SMS Spend Limit for your account so that limit can be increased before hand. In this way you would not face SMS delivery issues due to limit being reached. You can create a CloudWatch Alarm on your SMSMonthToDateSpentUSD metric.
Please have a look at this video: https://www.youtube.com/watch?v=5-HdLf_lizI
I'm researching the AWS CloudWatch SDK for Java and I see there's a limit of 5,000 alarms per account per region for PutMetricAlarm:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html
My situation is such that the number of alarms could potentially surpass this limit (i.e. transaction fails for a particular product). I wouldn't need to configure thresholds for a predetermined set of alarms. Rather, the alarm would be fired off ad hoc programmatically at the time failure is detected, with different failure possibilities that could reach well over 5,000.
Does CloudWatch support this scenario, either through PutMetricAlarm or otherwise?
You could:
Increase your account limits (contact the AWS support)
Use SetAlarmState API