For a web app with a moderate amount of static media content which will be deployed on an Amazon EC2 instance, does it make sense to include the static content inside the war file or to host it on a separate S3 instance.
What will be the differences in terms of performance?
What will be the differences in terms of service costs?
What will be the differences in terms of complexity?
Absolutely. For 4+ reasons:
Amazon S3 is almost management-free, so no hassles on provisioning, scaling, etc.
You will reduce EC2 server load
The storage is cheaper in S3 than in EC2 EBS volumes, as in S3 you only pay for what you consume, in EC2 you pay for the whole EBS provisioned storage (so there is some free space which you are paying for)
You could eventually add a CloudFront distribution to approach the static content to your users wherever they are (http://aws.amazon.com/cloudfront)
probably more ...
In terms of costs:
the data transfer from S3 to internet would be the same as you would pay on EC2
you will probably reduce the cost of the storage
you will have an additional cost for the number of requests made to your S3 files (http://aws.amazon.com/s3/#pricing)
on high traffic loads, you will also probably need less EC2 instances / resources (this is obviously not a fact, as it depends 100% on your app)
You will also have an overhead of complexity when releasing a new version of the app, because besides deploying it into the EC2 instances, you will also have to upload the new static file versions to S3. But you could automate this with a pretty simple script.
Related
Is it more efficient to store icons and small decorative images (eg arrow.jpg or Submit.jpg etc) on S3 vs on the hosting server's local disk ?
images/arrow.jpg (relative url to the hosting server)
vs
https://aws.com/images/arrow.jpg
I am not concerned about the storage/space but the number of hits to the server because every page may contain 50 to 100 such icons and small images. I am not using CDN
When you say hosting server, you are ultimately going to store your volume on EBS, EFS ,( not instance store because it is not persistent).
There are many factors to distinguish based on efficiency but ultimately it will boil down to COST.
S3 is way cheaper, highly available, and more durable than EBS.
EBS is AZ bound but s3 is not. if you want to access to ebs in another az you will need to copy and snapshot and restore to another AZ.
with s3 you can utilize intelligent teiring with lifecycle policies to move your files automatically between different storage tiers for cost optimization.
however, if you want to fast access and low latency then EBS and EFS is a better option without a doubt.
s3 gives you unlimited storage but individual element size is limited to 5TB
For performance go for ebs , for cost go for s3.
NOTE : if you want to access your data on ebs or efs you would need to spin up an ec2 instance which also add cost
FAq for comparison
s3 v/s ebs v/s efs
This post on stack is also a good read AWS EFS vs EBS vs S3 (differences & when to use?)
Basically, I'm trying to figure out what design to use. I'm collecting 1TB of data per month using an EC2 instance mounted to EBS. I created another Elastic Beanstalk instance serving as the website, and I wanted to figure out if it's better to access this EC2 instance's data through EFS or S3. Also, the amount of data that the elastic beanstalk webpage would access maybe be 10 - 50GB occasionally from a web application.
Basically, it depends upon the type of data you want to store.
EFS - Amazon EFS is automatically scalable - that means that your running applications won't have any problems if the workload suddenly becomes higher - the storage will scale itself automatically. If the workload decreases - the storage will scale down, so you won't pay anything for the storage you don't use. Good for shareable applications and workloads , Faster than S3
S3 - Amazon S3 also allows hosting static website content. provides simple object storage, useful for hosting website images and videos, data analytics, and both mobile and web applications. Object storage manages data as objects, meaning all data types are stored in their native formats.
So I would suggest, as you are collecting 1TB of data and webpage would access 10 - 50GB occasionally, so S3 will make your process (API's) slow and its good the amount of disk space you use, have to pay for that only.
And as you are talking about 1Tb, if data goes beyond that, the disk will be scalable and the application will be highly available.
We are currently runs in-house hardware that we would like to potentially move to AWS. Our main application uses MySQL on a Linux machine (200GB Disk, 32GB RAM, 4 Cores) serving content to customers through a hardware load balancer (around 1 million unique users per month).
We also use a 500 GB CDN hosted by a third party that we would like to move to AWS potentially. What AWS services would you recommend we look at to achieve comparable functionality and would you have a rough monthly cost estimate?
The main reason we would like to move to AWS would be for cost reduction in hardware and better backup strategies.
Thanks!
1.You can host your application using two or more EC2 instances and you can use elastic load balancer to distribute load amongst these EC2 instances.
2.You could use amazon aurora MySQL(server less) which offers you pay as you go service which will allow to get maximum benefits minimising your cost.It is the very best option for your MySQL database as your users are very high and so as the load on the database.
3.For CDN you could go for aws cloudfront and s3. it offers higher availability to your application and it also has less costing.You just need to make some proper configuration and you are ready to go.
AWS is the best cloud option for you as it provides service for your each problem so can use services according to your use and make most of it.
It also provides very good costing options whcih makes your tasks easy.
Please go through aws docs and costing before you choose aws.
Comparable functionality would be to use AWS RDS to replace your MySQL database, one or more EC2 instances to run your application, and then AWS Load Balancer to distribute the load amongst those EC2 web instances. A combination of S3 and Cloudfront to use as a CDN.
Cost is going to depend on how many ec2 instances you use, and the size and options you use for RDS database(s) plus storage and bandwidth - impossible for me to estimate for you
But you can do your own estimates here: https://awstcocalculator.com/
Which one is better for storing pictures and videos uploaded by user ?
Amazon s3 or Filesystem EC2 ?
While opinion-based questions are discouraged on StackOverflow, and answers always depend upon the particular situation, it is highly likely that Amazon S3 is your better choice.
You didn't say whether only wish to store the data, or whether you also wish to serve the data out to users. I'll assume both.
Benefits of using Amazon S3 to store static assets such as pictures and videos:
S3 is pay-as-you-go (only pay for the storage consumed, with different options depending upon how often/fast you wish to retrieve the objects)
S3 is highly available: You don't need to run any servers
S3 is highly durable: Your data is duplicated across three data centres, so it is more resilient to failure
S3 is highly scalable: It can handle massive volumes of requests. If you served content from Amazon EC2, you'd have to scale-out to meet requests
S3 has in-built security at the object, bucket and user level.
Basically, Amazon S3 is a fully-managed storage service that can serve static assets out to the Internet.
If you were to store data on an Amazon EC2 instance, and serve the content from the EC2 instance:
You would need to pre-provision storage using Amazon EBS volumes (and you pay for the entire volume even if it isn't all used)
You would need to Snapshot the EBS volumes to improve durability (EBS Snapshots are stored in Amazon S3, replicated between data centres)
You would need to scale your EC2 instances (make them bigger, or add more) to handle the workload
You would need to replicate data between instances if you are running multiple EC2 instances to meet request volumes
You would need to install and configure the software on the EC2 instance(s) to manage security, content serving, monitoring, etc.
The only benefit of storing this static data directly on an Amazon EC2 instance rather than Amazon S3 is that it is immediately accessible to software running on the instance. This makes the code simpler and access faster.
There is also the option of using Amazon Elastic File System (EFS), which is NAS-like storage. You can mount an EFS volume simultaneously on multiple EC2 instances. Data is replicated between multiple Availability Zones. It is charged on a pay-as-you-go basis. However, it is only the storage layer - you'd still need to use Amazon EC2 instance(s) to serve the content to the Internet.
I have a website which gets backup from different social media services and then stores the data on server and then that is displayed on my website. content includes, videos, images, and text data.
Currently i am using an EC2 instance with RDS and EBS. Data is stored in EBS Volumes, But as the amount of the data is big enough more than 1 TB and that is increasing. Every time my EBS volume gets filled i attach another volume.
Then i added S3 to my Setup. Cron jobs runs and stores data on S3 and the EC2 instance displays data from the S3. I am using PHP SDK for this purpose.
The problem which i am facing is that the S3 is very slow in my current setup.
Please suggest whether my setup is good or i need some change in my setup and the other way how can i speedup S3. or i should opt some other way to my setup.
EC2 instance is large reserved instance running CentOS.
I have listened some about the S3fs that mount S3 bucket to Ec2 as a volume. Is this a good choice, as when i mounted S3 Bucket to Ec2 instance the transfer rate was very slow.
I am new to the AWS. My users does not access files directly from S3, but they access through my website which is running on EC2 Instance.
RDS is a good choice for storing metadata such as tags, comments and other relevant information about your multimedia files. S3 is good for storing static content such as Video, Audio and Pictures. I think your approach with RDS and S3 is good enough.
EBS backed instances are good for persistence. If you store your metadata on RDS and static content on S3, the only reason why you should use EBS backed EC2 instances is that you have some configuration files which are unversioned right now. If that's not the case, assuming that your configuration is checked into version control and can be pulled on-demand for a fresh instance every time, then you might want to ditch EBS volumes in favor of ephemeral storage. That may give you some performance boost, nothing significant though.
Regarding your concern with S3's latency, yes, S3 is slow. While all your writes may happen directly to S3, I would highly recommend that you set up Amazon CloudFront for your S3 buckets and let your website consume multimedia content from the CloudFront. CloudFront is a Content Delivery Network (CDN) which works with disk volumes (EBS backed or ephemeral) as well as with S3. Setting it up would take not more than a few minutes. CloudFront also supports streaming media files over RTMP. You may need a library like GPAC for hinting multimedia files to make them streamable if not being done already. You might then want to consider creating one distribution for Video/Audio files for streaming and another distribution for Images, Javascript, Stylesheets and other text files.
Hope this helps.
For faster getting and uploading files from Amazon S3 I use batch() found here.
Also you can use cloudfront for faster getting files. I think 9gag uses cloudfront also..