Costs of enabling versioning in Amazon S3 - amazon-web-services

I have a question about the costs of versioning in Amazon S3 that don't seem to be present in the guide. There is a cost for every PUT/POST, but for versioned objects(especially when you keep older versions in alternative storage such as glacier) does each PUT/POST cost 2x the PUT/POST cost, one for the new version then one to move the old version to glacier?

You can refer to FAQ page: https://aws.amazon.com/s3/faqs/?nc1=h_ls
Q: How am I charged for using Versioning?
Normal Amazon S3 rates apply for every version of an object stored or
requested. For example, let’s look at the following scenario to
illustrate storage costs when utilizing Versioning (let’s assume the
current month is 31 days long):
1) Day 1 of the month: You perform a PUT of 4 GB (4,294,967,296 bytes)
on your bucket. 2) Day 16 of the month: You perform a PUT of 5 GB
(5,368,709,120 bytes) within the same bucket using the same key as the
original PUT on Day 1.
When analyzing the storage costs of the above operations, please note
that the 4 GB object from Day 1 is not deleted from the bucket when
the 5 GB object is written on Day 15. Instead, the 4 GB object is
preserved as an older version and the 5 GB object becomes the most
recently written version of the object within your bucket. At the end
of the month:
Total Byte-Hour usage [4,294,967,296 bytes x 31 days x (24 hours /
day)] + [5,368,709,120 bytes x 16 days x (24 hours / day)] =
5,257,039,970,304 Byte-Hours.
Conversion to Total GB-Months 5,257,039,970,304 Byte-Hours x (1 GB /
1,073,741,824 bytes) x (1 month / 744 hours) = 6.581 GB-Month
The fee is calculated based on the current rates for your region on
the Amazon S3 Pricing page.

Related

Google Cloud Run charges more CPU and RAM seconds than actually used

I have a Cloud Run Job running every 10min, on 1 vCPU and 2 GiB of RAM, with a 300sec timeout. There is no retry policy. The job always runs in less than 30 seconds, the average being around 20 seconds.
Based on these info, I was expecting to be using approximately:
Minutes in a month: 31 * 24 * 60 ~= 45,000
Runs per month: 45,000 / 10 = 4500
Runtime per run: 20 seconds
Total expected vCPU-second usage: 4500 * 20 = 90,000
Instead, the billing report indicates a number more than twice as high. Is there anything I am missing on? Where all this extra usage could come from?
Thanks for your help.
TLDR: you get charged a minimum of 1 minute for each Job run.
I realized that my actual usage was 179,176 vCPU-second for 3015 runs, corresponding to approximately a minute per run.
This sounded like a minimum time that would be billed for each run. It was confirmed by looking more in depth at the pricing page for Cloud Run.
It was NOT mentioned in the pricing table, but at the very bottom of the page: "Billable container instance time for Cloud Run jobs". I got fooled by the free tier, hopefully this will help others :)

Do CDN such as cloudfront have any limitation to maximum volume of cache that can be stored in a distribution?

I am trying to understand the pricing of CDN in AWS, GCP and Azure.
One thing I am not able to figure out is, if there is any limit to total amount of cache which can be stored in a single distribution, and is there any added cost as the total volume of cache keeps on increasing.
To give my prospective:-
Usecase 1) Using CDN for delivering 1_000_000 files of size 100KB in a location daily, with retention time of 1day.
Total Volume of data present in cache (1year later): 1_00_000 * 100KB * 1 = 100GB
Total Bandwidth consumed (1year later): 1_000_000 * 100KB * 365 = 3650GB
Usercase 2) Using CDN for delivering 100 files of size 100MB in a location daily, with retention time of 1years.
Total Volume of data present in cache (1year later): 100 * 100MB * 365 = 3650GB
Total Bandwidth consumed (1year later): 100 * 100MB * 365 = 3650GB
(Note: Let's assume in the use case above all the files counted are always unique.)
So speaking of cloudfront, it will mainly charging me for the bandwidth, which is same for both the use cases. However in Usecase 2 the resources spent for storing the cache is a lot higher.
My question is am I missing something in the pricing, or CDN providers don't care about storage costs?
Regarding CloudFront. Cache size is virtually unlimited and you pay for it with transfer fees. The fact that you set "retention time of 1years" does not mean your files will stay in cache for that long. If AWS deems your files as in-frequently used, it will purge them well before they get 1 year old. From docs:
If a file in an edge location isn't frequently requested, CloudFront might evict the file—remove the file before its expiration date—to make room for files that have been requested more recently.

Firehose to S3 Vs Direct PUT to S3

I have a requirement to PUT about 20 records per second to an S3 bucket.
That comes to roughly 20 * 60 * 60 * 24 * 30 = 51,840,000 PUTs per month.
I do not need any transformations but I would certainly want the PUTs to be GZIPped and partitioned by year/date/month/hour.
Option 1 - Just do PUTObjects on S3
Price comes to about ~$260 a month
I would have to do the GZIP/Partitions etc on the client side
Option 2 - Introduce a Firehose and wire it to S3
And let's say I buffer only once in 10 minutes then that is about 6 * 24 * 30 = 4,320 PUTs. Price of S3 comes down to $21. With each record about 20 KB Firehose pricing is about 1000GB * 0.029 comes to about $30. So total pricing is $51. Costs for data transfer / storage etc are same in both approaches I believe.
Firehose provides GZip/Partitions/buffering for me OOTB
It appears like Option 2 is the best for my use case. Am I missing something here?
Thanks for looking!

EBS Snapshots full backups

Is it possible to take a full backup (snapshot) of a volume again after the incremental backup?
E.g. Day 1 : Full backup
Day 2-6 : Incremental backups
Day 7 : Full backup again.
The reason : Client wants to keep their RTO low as it will take more time to restore from an incremental backup. Any solutions?
I dont think its possible to take a full backup after an incremental backup.
You can back up the data on your Amazon EBS volumes to Amazon S3 by taking point-in-time snapshots. Snapshots are incremental backups, which means that only the blocks on the device that have changed after your most recent snapshot are saved
For Example:
In State 1, the volume has 10 GiB of data. Because Snap A is the first snapshot taken of the volume, the entire 10 GiB of data must be copied.
In State 2, the volume still contains 10 GiB of data, but 4 GiB have changed. Snap B needs to copy and store only the 4 GiB that changed after Snap A was taken. The other 6 GiB of unchanged data, which are already copied and stored in Snap A, are referenced by Snap B rather than (again) copied. This is indicated by the dashed arrow.
In State 3, 2 GiB of data have been added to the volume, for a total of 12 GiB. Snap C needs to copy the 2 GiB that were added after Snap B was taken. As shown by the dashed arrows, Snap C also references 4 GiB of data stored in Snap B, and 6 GiB of data stored in Snap A.
The total storage required for the three snapshots is 16 GiB.

How much it cost to use Amazon S3 for Video Streaming backend?

We are developing a video streaming site
For that we want to use Amazon S3 storage
But I can't understand the pricing structure . The price calculator is also confusing
Please tell how much it will cost for below calculation
100 Videos uploaded via S3 Api .. Each having size of 500 MB.(Region LONDON)
=> So 100 x 500 = ~ 5GB storage used
each and every video requested (GET) 1000 times
=> 100 * 1000 = 100,000 GET requests
each and every video viewed 1000 times
=> So (500 x 1000) x 100 = ~ 5 TB bandwidth used
Now please say how much it will cost ? deduce the pricing step by step
AWS has an official pricing tool which is helpful for estimating a service cost.
The estimate for your S3 cost is: $448.69 / month
You can see the full workings, and update the calculation here.
100 Videos uploaded at 500mb each:
100 * $0.23 / 2 = $1.15
Each video requested 1,000 times:
(100 * 1,000) / 1,000 * $0.0004 = $0.04
Each video viewed 1,000 times:
S3 doesn’t charge for a video being viewed only a video being requested I believe.
So your total is $1.19
Here’s a link to s3 pricing for storage & request:
https://aws.amazon.com/s3/pricing/