I have an app, built upon multiple AWS services, that provides a document storage service for my users.
Rather than track my users usage based on the way they use my app, and then multiplying their usage with the cost of each service they consume (doing the calculation myself), I was wondering if there was a way to do this automatically (having aws track my users at a granular per user level and compute per user costs automatically)?
For example, when a user consumes some AWS service, is there an option to provide an identifieR to AWS, so AWS tracks usage and computes costs of individual ids itself? That way it would be much simpler to just ask AWS how much my users are consuming and charge them appropriately.
It appears that you are wanting to determine the costs of your service so that you can pass-on these costs to your customers.
I would recommend that you re-think your pricing strategy. Rather than charging as "cost plus some profit", focus your pricing with these rules in mind:
Charge for things that are of value to your customers (that they want to do)
Charge for things that you want them to do less (what you don't want them to do)
Make everything else free
Think about how this applies to other services that charge money:
Water utilities: Charge for water, but charge people for consuming too much water
Netflix: Charge for providing shows, but charge extra for consuming more bandwidth (4K, multiple accounts)
Cell phone: Charge for service, but charge extra for consuming more data
Each of these organizations suffer extra costs for providing more water, bandwidth and data, so they would prefer people either don't consume them much, or pay extra for consuming it.
If your application provides benefits to customer for storing documents, then they will pay for your service. However, you will suffer extra costs for storing more documents so you should charge extra for consuming more storage. Everything else, if possible, should be free.
Therefore, don't calculate how much each user costs you for running the Amazon EC2 instances, Data Transfer bandwidth, domain names, database storage, support staff, programmers, management and your time. Instead, concentrate on the value you are giving to your customers and they should be willing to pay if the benefit is greater than the cost. Find the element that really costs you more if they over-consume and charge extra for that. For example, if somebody stores more documents, which consumes more space in Amazon S3 and costs more Data Transfer, then charge them for having more documents. Don't charge them based on some technical aspect like the size of EC2 instance you are using.
To answer your specific question, you could use tagging to identify resources that relate to a specific user, such as objects stored in Amazon S3. However, you'll probably find that most of your costs are shared costs (eg you can't split-up an EC2 instance hosting a web app or a database used by all customers). Your application database is probably already keeping track of their usage so tagging wouldn't necessarily provide much additional insight unless specific resources are being consumed by specific users.
See: AWS Tagging Strategies
Related
I was hoping to get other's perspective on something that we've done and beginning to realize it was not the best idea.
Here's some information about our "environment":
Account A: We have an AWS account that acts as a data lake (we upload transaction data to S3 daily)
Account B: We have another AWS account that our business partners use to access the data in Account A
A few months back, we enabled Intelligent Tiering in S3 where objects are moved to Archive and Deep Archive in 90 and 180 days, respectively. We're now seeing the downfall of this decision. OUr business partners are unable to query data (in account A) from 3 months ago in Athena (account B). Oof.
I guess we did not understand the purpose of intelligent tiering and had hoped that Athena would be able to move tiered objects back into standard s3 when someone queries the data (as in instant retrieval).
There's definitely some use cases that we missed in vetting intelligent tiering.
I am curious how are others leveraging intelligent tiering? Are you only tiering objects that your business partners do not need as "instant retrieval"?
If you goal is to reduce Storage costs, it is worth investigating and understanding the various Storage Classes offered by Amazon S3.
They generally fall into three categories:
Instantly available: This is the 'standard' class
Instantly available, lower storage cost but higher retrieval cost: This is the 'Infrequent Access' classes. They can be cheaper for data that is only accessed once per month or less. If they are accessed more often, then the Request charges outweigh the savings in Storage costs.
Archived: This is typically the Glacier classes. Avoid them if you want to use Amazon Athena.
See that table on: Comparing the Amazon S3 storage classes
For your use-case, you might consider keeping data in Standard by default (since it is heavily accessed), and then move data older than 90 days to S3 One Zone - Infrequent Access. It will still be accessible, but will have a lower storage cost if rarely used.
I would also recommend converting your data to Snappy-compressed Parquet format (preferably partitioned), which will reduce the amount of storage required and will allow Athena to selectively pick which objects it needs to access. It will will also make Athena run faster and reduce the cost of Athena queries.
See: Top 10 Performance Tuning Tips for Amazon Athena | AWS Big Data Blog
I have newly created an API service that is going to be deployed as a pilot to a customer. It has been built with AWS API Gateway, AWS Lambda, and AWS S3. With a SaaS pricing model, what's the best way for me to monitor this customer's usage and cost? At the moment, I have made a unique API Gateway, Lambda function, and S3 bucket specific to this customer. Is there a good way to create a dashboard that allows me (and perhaps the customer) to detail this monitoring?
Additional question, what's the best way to streamline this process when expanding to multiple different customers? Each customer would have a unique API token — what's the better approach than the naive way of making unique AWS resources per customer?
I am new (a college student), but any insights/resources would help me a long way. Thanks.
Full disclosure: I work for Lumigo, a company that does exactly that.
Regarding your question,
As #gusto2 said, there are many tools that you can use, and the best tool depends on your specific requirements.
The main difference between the tools is the level of configuration that you need to apply.
cloudwatch default metrics - The first tool that you should use. This is an out-of-the-box solution that provides you many metrics on the services, such as: duration, number of invocations and errors, memory. You can configure metrics over different timeslots and aggregators (P99, average, max, etc.)
This tool is great for basic monitoring.
Its limitation is its greatest strength - it provides monitoring which is common to all the services, thus nothing tailored-fit to serverless applications. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html
cloudwatch custom metrics - The other side of the scale - getting much more precise metrics, which allows you to upload any metric data and monitor it: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html
This is a great tool if you know exactly what you want to monitor, and you already familiar with your architecture limitations and pain points.
And, of course, you can configure alarms over this data:
Lumigo - 3rd party company (again, as a disclosure, this is my workplace). Provides out-of-the-box monitoring, specifically created for serverless applications, such as an abnormal number of invocations, costs, etc.. This tool also provides troubleshooting capabilities to enable deeper observability.
Of course, there are more 3rd party tools that you can find online. All are great- just find the one that suits your requirement the best.
Is there a good way to create a dashboard
There a are multiple ways and options depending in your scaling, amount of data and requirements. So you could start small and simple, but check if any option is feasible or not.
You can start with the CloudWatch. You can monitor basic metrics, create dashboards and even share with other accounts.
naive way of making unique AWS resources per customer
For the start I would consider creating custom cloudwatch metrics with the customer id as a metric and put the metrics from the Lambda functions.
Looks simple, but you should do the math and a PoC about the number of requested datapoints and the dashboards to prevent a nasty surprise on the billing.
Another option is sending metrics/events to DynamoDB, using atomic functions you could directly build some basic aggregations (kind of naïve stream processing).
When scaling to a lot of events, clients, maybe you will need some serious api analytics, but that may be a different topic.
Is there any cost difference or any other type of difference(latency, performance, data corruption etc.) for regions?? I am making primarily a IoT service stack.
Each region has its own prices for services, this will generally vary marginally with some regions (such as us-east-1, eu-west-1) generally costing less overall.
Generally data transfer itself is the same from the source to destination (defined at the region level), however CloudFront does charge depending on the nearest edge location to the user.
Latency will definitely impact your end users so it is worth investing in a region closer to them so that this can be minimised. You can get a rough guideline from CloudPing.
You should check each services pricing page to get more of a breakdown of the overall costs you'd pay in a region. However, generally you need to decide between performance and cost so you should evaluate between the both of these.
Also bare in mind that new service releases and features are rolled out to specific regions at a time, if you want to use the latest features you may want to decide on a region that is known to get them earlier.
In Short Yes.. Each region in AWS has different costs associated with same services. There are some exceptions as well but generally it s the case. e.g. IOT Core service prices in US East region is $0.08 (per million minutes of connection) vs $0.092 (per million minutes of connection) in ASIA Pacific region. This is 15% increase but if you are going to use let's say 100 million minutes of connections then the difference is just around a Dollar!!!
However if any of your device in Asia region wants to send some data to US Region then there will definitely be latency & performance issues. So it is always better to use nearby region. You can always go to the pricing page of AWS Service you are going to use and check Costs.
I'm responsible for two AWS accounts where a web service is run in two different environments. I'm now trying to have a look at cutting costs, and I'm a bit confused as to how to make sense of the CloudTrail costs, i.e., break it down into different categories etc, and also if there are some common pitfalls when it comes to high CloudTrail costs.
For example, my company's security department is running some monitoring SW in my accounts - both internal like GuardDuty and external, and I think these tools are responsible for a lot of this cost. I'd like to be able to chart down exactly which of these costs are attributed to tools that I am in no control of (and consequently which I may be able to reduce), and which are due to infrastructure I'm responsible of. Right now, CloudTrail costs are the single highest item, and I think it seems odd that we're paying more for it than what our combined cost is for EC2 + Lambda + DynamoDB + S3.
When looking at the CloudTrail event stream itself, most events don't hold that much information useful for me to understand why this event is sent or to group them in a bar chart by "monitoring tool" vs "normal operation". There's basically an event source and a user name. At a glance, a large portion of events seem to be CreateLogStream and UpdateInstanceInformation, which are likely due to normal operation of some of the services?
If it turns out simple events such as CreateLogStream are costing thousands of dollars, how should I attack this problem? Is it possible to manually disable certain CloudTrail events? Is there a best practice as to what events are important and which are not? I mean, I never use CloudTrails for anything during normal operation, but it feels nice to have them in case something nasty happens.
It is unclear to me whether you're asking about the costs to operate the CloudTrail service, or about using CloudTrail to track your operational costs.
If the former, I'd be very surprised if you are running up significant costs for CloudTrail. You get one trail per account for free, and additional trails cost $2.00 per 100,000 events (pricing page). The only way that I can see for that to become a significant cost factor is to have an extremely large number -- as in dozens or hundreds -- of trails per account. Which is unlikely to be the case.
In the latter case, CloudTrail is the wrong tool for the job. Instead, you should use Cost Explorer, which will let you group your costs by multiple factors, including service and usage type.
If you are part of an organization (which it seems you are), and use consolidated billing (in which the organization owner pays all bills), then you'll need to be granted permission to see your cost data.
I'm hosting a static website in Amazon S3 with CloudFront. Is there a way to set a limit for how many reads (for example per month) will be allowed for my Amazon S3 bucket in order to make sure I don't go above my allocated budget?
If you are concerned about going over a budget, I would recommend Creating a Billing Alarm to Monitor Your Estimated AWS Charges.
AWS is designed for large-scale organizations that care more about providing a reliable service to customers than staying within a particular budget. For example, if their allocated budget was fully consumed, they would not want to stop providing services to their customers. They might, however, want to tweak their infrastructure to reduce costs in future, such as changing the Price Class for a CloudFront Distribution or using AWS WAF to prevent bots from consuming too much traffic.
Your static website will be rather low-cost. The biggest factor will likely be Data Transfer rather than charges for Requests. Changing the Price Class should assist with this. However, the only true way to stop accumulating Data Transfer charges is to stop serving content.
You could activate CloudTrail data read events for the bucket, create a CloudWatch Event Rule to trigger an AWS Lambda Function that increments the number of reads per object in an Amazon DynamoDB table and restrict access to the objects once a certain number of reads has been reached.
What you're asking for is a very typical question in AWS. Unfortunately with near infinite scale, comes near infinite spend.
While you can put a WAF, that is actually meant for security rather than scale restrictions. From a cost-perspective, I'd be more worried about the bandwidth charges than I would be able S3 requests cost.
Plus once you put things like Cloudfront or Lambda, it gets hard to limit all this down.
The best way to limit, is to put Billing Alerts on your account -- and you can tier them, so you get a $10, $20, $100 alerts, up until the point you're uncomfortable with. And then either manually disable the website -- or setup a lambda function to disable it for you.