Is it more efficient to store icons and small decorative images (eg arrow.jpg or Submit.jpg etc) on S3 vs on the hosting server's local disk ?
images/arrow.jpg (relative url to the hosting server)
vs
https://aws.com/images/arrow.jpg
I am not concerned about the storage/space but the number of hits to the server because every page may contain 50 to 100 such icons and small images. I am not using CDN
When you say hosting server, you are ultimately going to store your volume on EBS, EFS ,( not instance store because it is not persistent).
There are many factors to distinguish based on efficiency but ultimately it will boil down to COST.
S3 is way cheaper, highly available, and more durable than EBS.
EBS is AZ bound but s3 is not. if you want to access to ebs in another az you will need to copy and snapshot and restore to another AZ.
with s3 you can utilize intelligent teiring with lifecycle policies to move your files automatically between different storage tiers for cost optimization.
however, if you want to fast access and low latency then EBS and EFS is a better option without a doubt.
s3 gives you unlimited storage but individual element size is limited to 5TB
For performance go for ebs , for cost go for s3.
NOTE : if you want to access your data on ebs or efs you would need to spin up an ec2 instance which also add cost
FAq for comparison
s3 v/s ebs v/s efs
This post on stack is also a good read AWS EFS vs EBS vs S3 (differences & when to use?)
Related
I am new to AWS and trying to understand it.
Presently i have a small Django App.
I am planning to have an EC2 instance of type t3.small (2 cpus and 2 GB ram)
I will have the following EBS storages:
1) EBS root system: 10GB (mainly used for OS and other config files)
2) Attached EBS: 20GB (mainly used to store all the code and images and videos etc)
I will keep taking snapshot of both the EBS volumes daily.
I want my images and videos only accessible from the EC2 instance.
So Is this approach a good approach. or i have to use S3/
I'll go for storing in S3 for many reasons :
Duarbility : S3 is 99,99999999 (11x9). Data is replicated in all AZ of the selected region. You can also easily setup a Cross Region Replication for better data durability and security.
Price: You pay only for the disk space you're using it. You don't have to specify a volume size like with EBS.
Scalibility : You add data as you go and the bucket will scale to infinite.
If your application become successful and you have more traffic you should go to a horizontal scaling using AutoScaling and LoadBalancer and that means multiple EC2 instances. In that case it will be so easier to access Data on S3 instead on an EBS attached to an instance.
Which one is better for storing pictures and videos uploaded by user ?
Amazon s3 or Filesystem EC2 ?
While opinion-based questions are discouraged on StackOverflow, and answers always depend upon the particular situation, it is highly likely that Amazon S3 is your better choice.
You didn't say whether only wish to store the data, or whether you also wish to serve the data out to users. I'll assume both.
Benefits of using Amazon S3 to store static assets such as pictures and videos:
S3 is pay-as-you-go (only pay for the storage consumed, with different options depending upon how often/fast you wish to retrieve the objects)
S3 is highly available: You don't need to run any servers
S3 is highly durable: Your data is duplicated across three data centres, so it is more resilient to failure
S3 is highly scalable: It can handle massive volumes of requests. If you served content from Amazon EC2, you'd have to scale-out to meet requests
S3 has in-built security at the object, bucket and user level.
Basically, Amazon S3 is a fully-managed storage service that can serve static assets out to the Internet.
If you were to store data on an Amazon EC2 instance, and serve the content from the EC2 instance:
You would need to pre-provision storage using Amazon EBS volumes (and you pay for the entire volume even if it isn't all used)
You would need to Snapshot the EBS volumes to improve durability (EBS Snapshots are stored in Amazon S3, replicated between data centres)
You would need to scale your EC2 instances (make them bigger, or add more) to handle the workload
You would need to replicate data between instances if you are running multiple EC2 instances to meet request volumes
You would need to install and configure the software on the EC2 instance(s) to manage security, content serving, monitoring, etc.
The only benefit of storing this static data directly on an Amazon EC2 instance rather than Amazon S3 is that it is immediately accessible to software running on the instance. This makes the code simpler and access faster.
There is also the option of using Amazon Elastic File System (EFS), which is NAS-like storage. You can mount an EFS volume simultaneously on multiple EC2 instances. Data is replicated between multiple Availability Zones. It is charged on a pay-as-you-go basis. However, it is only the storage layer - you'd still need to use Amazon EC2 instance(s) to serve the content to the Internet.
I am new to cloud environment. I do understand the definition and storage types EBS and S3. I wanted to understand the application of EBS as compared to S3.
I do understand EBS looks like a device for heavy though put operations. I cannot find any application where this can be used in comparison to S3. I could think of putting server logs on EBS on magnetic storage, as one EBS can be attached to one instance.
S3 you can use the scaling property to add some heavy data and expand in real time. We can deploy our slef managed dbs on this service.
Please correct me if I am wrong. Please help me understand what is best suited for what and application of them in comparison with one another.
As you stated, they are primarily different types of storage:
Amazon Elastic Block Store (EBS) is a persistent disk-storage service, which provides storage volumes to a virtual machine (similar to VMDK files in VMWare)
Amazon Simple Storage Service (S3) is an object store system that stores files as objects and optionally makes them available across the Internet.
So, how do people choose which to use? It's quite simple... If they need a volume mounted on an Amazon EC2 instance, they need to use Amazon EBS. It gives them a C:, D: drive, etc in Windows and a mountable volume in Linux. Computers traditionally expect to have locally-attached disk storage. Put simply: If the operating system or an application running on an Amazon EC2 instance wants to store data locally, it will use EBS.
EBS Volumes are actually stored on two physical devices in case of failure, but an EBS volume appears as a single volume. The volume size must be selected when the volume is created. The volume exists in a single Availability Zone and can only be attached to EC2 instances in the same Availability Zone. EBS Volumes persist even when the attached EC2 instance is Stopped; when the instance is Started again, the disk remains attached and all data has been presrved.
Amazon S3, however, is something quite different. It is a storage service that allows files to be uploaded/downloaded (PutObject, GetObject) and files are replicated across at least three data centers. Files can optionally be accessed via the Internet via HTTP/HTTPS without requiring a web server. There are no limits on the amount of data that can be stored. Access can be granted per-object, per-bucket via a Bucket Policy, or via IAM Users and Groups.
Amazon S3 is a good option when data needs to be shared (both locally and across the Internet), retained for long periods, backed-up (or even for storing backups) and made accessible to other systems. However, applications need to specifically coded to use Amazon S3 and many traditional application expect to store data on a local drive rather than on a separate storage service.
While Amazon S3 has many benefits, there are still situations where Amazon EBS is a better storage choice:
When using applications that expect to store data locally
For storing temporary files
When applications want to partially update files, because the smallest storage unit in S3 is a file and updating a part of a file requires re-uploading the whole file
For very High-IO situations, such as databases (EBS Provisioned IOPS can provide volumes up to 20,000 IOPS)
For creating volume snapshots as backups
For creating Amazon Machine Images (AMIs) that can be used to boot EC2 instances
Bottom line: They are primarily different types of storage and each have their own usage sweet-spot, just like a Database is a good form of storage depending upon the situation.
What are the performance implications of using S3 vs. EBS?
Say my app needs to access files fast. Will EBS be much faster than S3? I would like to use S3 because I don't want a space limitations, but am worried about performance.
In general EBS will perform better than S3 (EBS has 4 variations - gp2, io1, st1, sc1). With io1 you get to specify IOPS capabilities (S3 has no user specified performance control). gp2 has IO bursts (up to 3000 IOPS) and IO credits (5.4 Million) which may be good for some applications. Check out http://www.slideshare.net/AmazonWebServices/deep-dive-maximizing-ec2-and-ebs-performance
Hope it is understood that both use different access methods. S3 is an object store, EBS is a block storage.
(Please also note that EBS is also network attached, not directly attached to EC as far as I know)
You should consider the AWS Elastic File System as an alternative to S3 and EBS. Its speed will likely fall between S3 and EBS but it does provide ways to control performance and it is not limited in terms of size. Amazon EFS Performance details: http://docs.aws.amazon.com/efs/latest/ug/performance.html
This should help:
Regards,
Aditya Garg
EBS will always be more faster if you compare EBS vs S3 for data transfer, as EBS is directly attached to the EC2 instance. But again the storage limitation comes in picture for EBS.
So if you want to achieve both less latency and unlimited storage, I will suggest attaching S3 bucket as a volume to your EC2 instance, gave a very good data transfer speeds in my case.
Here is a guide that is really good for achieving.
https://fullstacknotes.com/mount-aws-s3-bucket-to-ubuntu-file-system/
For a web app with a moderate amount of static media content which will be deployed on an Amazon EC2 instance, does it make sense to include the static content inside the war file or to host it on a separate S3 instance.
What will be the differences in terms of performance?
What will be the differences in terms of service costs?
What will be the differences in terms of complexity?
Absolutely. For 4+ reasons:
Amazon S3 is almost management-free, so no hassles on provisioning, scaling, etc.
You will reduce EC2 server load
The storage is cheaper in S3 than in EC2 EBS volumes, as in S3 you only pay for what you consume, in EC2 you pay for the whole EBS provisioned storage (so there is some free space which you are paying for)
You could eventually add a CloudFront distribution to approach the static content to your users wherever they are (http://aws.amazon.com/cloudfront)
probably more ...
In terms of costs:
the data transfer from S3 to internet would be the same as you would pay on EC2
you will probably reduce the cost of the storage
you will have an additional cost for the number of requests made to your S3 files (http://aws.amazon.com/s3/#pricing)
on high traffic loads, you will also probably need less EC2 instances / resources (this is obviously not a fact, as it depends 100% on your app)
You will also have an overhead of complexity when releasing a new version of the app, because besides deploying it into the EC2 instances, you will also have to upload the new static file versions to S3. But you could automate this with a pretty simple script.