Scenario is that I want to store my images uploaded by user through my application in amazon s3 and later retrieve them on demand.
So I was going through how much cost it will be there to store images in s3 and I across term "Data transfer cost",
I am bit confused like suppose I uploaded and image in s3 and make it public and when even I access that image suppose in a Brower tab through public s3 link, do I have to pay data transfer cost for that? or accessing and image from it's public link is free?
Same is if I download my image from s3 using my front end application, then also I have to pay data transfer cost?
Second part of my question :
If data transfer cost apply can I avoid my data transfer cost by first accessing my image from NodeJS running in ec2 instance then sending it back from my NodeJS application running in ec2 instance ?
Yes, you pay in both cases, that is exactly what the data transfer cost means.
Additionally you pay for the storage and for the number of requests.
You can e.g. go through https://calculator.aws/#/createCalculator to get a rough estimate of what cost you may incur.
No, you cannot avoid the cost. If you start loading them into the ec2 machine that cost is free (assuming same region etc.) but then transferring the data from the ec2 instances does cost again.
Related
I have 8 TB of on premise data at present. I need to transfer it to AWS S3. Going forward every month 800gb of data will be required to update. What will be the cost of the different approaches?
Run a python script in ec2 instance.
Use AWS Lambda for the transfer.
Use AWS DMS to transfer the data.
I'm sorry that I wont do the calculations for you,
but i hope with this tool you can do it yourself :)
https://calculator.aws/#/
According to
https://aws.amazon.com/s3/pricing/
Data Transfer IN To Amazon S3 From Internet
All data transfer in $0.00 per GB
Hope you will find your answer !
While data is inside SQL, you need to move that out of it first. If your SQL is AWS's managed RDS, that's easy task, just backup to s3. Yet if it's something you manage by hand, figure out to move data to s3. Btw, you can not only use s3, but disk services too.
You do not need EC2 instance to make data transfer unless you need some compute on that data.
Then to move 8Tb there are couple of options. Cost is tricky thing while downtime of slower transfer may mean losses, maybe security risk is another cost to think about, developer's time etc. etc. so it really depends on your situation
Option A would be to use AWS File Gateway and mount locally network drive with enough space and just sync from local to that drive. https://aws.amazon.com/storagegateway/file/ Maybe this would be the easiest way, while File Gateway will take care of failed connections, retries etc. You can mount local network drive to your OS which sends data to S3 bucket.
Option B would be just send over the public network. Which may be not possible if connection is slow or insecure by your requirements.
Option C which is usually not used for single time transfer - private link to AWS. This would provide more security and probably speed.
Option D would be to use snow family products. Smallest AWS Snowcone has exactly 8Tb of capacity, so if you really under 8Tb, maybe it would be more cost effective way to transfer. If you actually have a bit more than 8Tb, you need AWS Snowball, which can handle much more then 8Tb but it's <80Tb, which is enough in your case. Fun note, for up to 100PB data transfer there is Snowmobile.
AWS S3 has a standard public bucket and folder (Asia Pacific region) which hosts ~30 GB of images/media. On another hand, the website and app access these images by using a direct S3 object URL. Unknowingly we run into high data transfer cost and its significantly unproportionate:
Amazon Simple Storage Service: USD 30
AWS Data Transfer: USD 110
I have also read that if EC2 and S3 is in the same region cost will be significantly lower, but problem is S3 objects are accessible from anywhere in the world from client machine directly and no EC2 is involved in between.
Can someone please suggest how can data transfer costs be reduced?
The Data Transfer charge is directly related to the amount of information that goes from AWS to the Internet. Depending on your region, it is typically charged at 9c/GB.
If you are concerned about the Data Transfer charge, there are a few things you could do:
Activate Amazon S3 Server Access Logging, which will create a log file for each web request. You can then see how many requests are coming and possibly detect strange access behaviour (eg bots, search engines, abuse).
You could try reducing the size of files that are typically accessed, such as making images smaller. Take a look at the Access Logs and determine which objects are being accessed the most, and are therefore causing the most costs.
Use less large files on your website (eg videos). Again, look at the Access Logs to determine where the data is being used.
A cost of $110 suggests about 1.2TB of data being transferred.
I was going through the AWS s3 calculator for Standard Storage. I didnt understood the meaning of Data Transfer calculation. How it is calculated?
Data Transfer:
Inter-Region Data Transfer Out: 10GB/Month
Data Transfer Out:10GB/Month
Data Transfer In:10GB/Month
Data Transfer Out to CloudFront:
Let say my use case is i want to upload a file, store it in S3 s long as i want it, i will download it whenever i want, i will modify the file and these operation i will do from across the world from mobile, laptop through my app/website.
You pay for all traffic going out of AWS (Data Transfer Out). You also pay for inter-region traffic.
Let's say you have a S3 bucket in Virginia and an EC2 in Oregon. This EC2 downloads a 100MB file from that bucket. You'll pay 100MB of inter-region.
If you download the file from your PC, you pay 100MB of transfer out. You always pay for what goes out or to another region. If you partially download a file, you'll be charged that exact amount. That's basically it.
Background
We have developed an e-commerce application where I want to use CDN to improve the speed of the app and also to reduce the load on the host.
The application is hosted on an EC2 server and now we are going to use Cloud Front.
Questions
After reading a lot of articles and documents, I have created a distribution for my sample site. After doing all the experience I have come to know the following things. I want to be sure if am right about these points or not.
When we create a distribution it takes all the accessible data from the given origin path. We don't need to copy/ sync our files to cloud front.
We just have to change the path of our application according to this distribution CNAME (if cname is given).
There is no difference between placing the images/js/CSS files on S3 or on our own host. Cloud Front will just take them by itself.
The application will have thousands of pictures of the products, should we place them on S3 or its ok if they are on the host itself? Please share any good article to understand the difference of both the techniques.
Because if S3 is significantly better then I'll have to make a program to sync all such data on S3.
Thanks for the help.
Some reasons to store the images on Amazon S3 rather than your own host (and then serve them via Amazon CloudFront):
Less load on your servers
Even though content is cached in Amazon CloudFront, your servers will still be hit with requests for the first access of each object from every edge location (each edge location maintains its own cache), repeated every time that the object expires. (Refreshes will generate a HEAD request, and will only re-download content that has changed or been flushed from the cache.)
More durable storage
Amazon S3 keeps copies of your data across multiple Availability Zones within the same Region. You could also replicate data between your servers to improve durability but then you would need to manage the replication and pay for storage on every server.
Lower storage cost
Storing data on Amazon S3 is lower cost than storing it on Amazon EBS volumes. If you are planning on keeping your data in both locations, then obviously using S3 is more expensive but you should also consider storing it only on S3, which makes it lower cost, more durable and less for you to backup on your server.
Reasons to NOT use S3:
More moving parts -- maintaining code to move files to S3
Not as convenient as using a local file system
Having to merge log files from S3 and your own servers to gather usage information
I am planning to store objects in S3 standard storage, each object could be of 100MB in size so monthly it could go upto 1TB. I will use a single region to store these objects in S3.
I want to create a mobile app to store and fetch this objects using post/get apis.
And then show them in my app.
S3 uses different pricing sections, I understand storage and requests (post/get) pricing.
My question is around data transfer in/out pricing, in my case above will I be billed for data transfer in/out? if no why not?
Yes, you will be billed because you mobile app will connect from internet. Even connected from within AWS there are fees associated with your number of requests and data transferred (inside region or outside region).
You can use the AWS Calc to get an estimate for the cost associated: https://calculator.s3.amazonaws.com/index.html
All traffic FROM mobile phones to S3 or EC2 is free.
All traffic TO mobile phones from S3/CloudFront is billed according to a selected region. Take a look at https://aws.amazon.com/s3/pricing/.
Keep in mind that incoming traffic (to S3) is free only if you're NOT using S3 Transfer Acceleration.