I am new to AWS and trying to understand it.
Presently i have a small Django App.
I am planning to have an EC2 instance of type t3.small (2 cpus and 2 GB ram)
I will have the following EBS storages:
1) EBS root system: 10GB (mainly used for OS and other config files)
2) Attached EBS: 20GB (mainly used to store all the code and images and videos etc)
I will keep taking snapshot of both the EBS volumes daily.
I want my images and videos only accessible from the EC2 instance.
So Is this approach a good approach. or i have to use S3/
I'll go for storing in S3 for many reasons :
Duarbility : S3 is 99,99999999 (11x9). Data is replicated in all AZ of the selected region. You can also easily setup a Cross Region Replication for better data durability and security.
Price: You pay only for the disk space you're using it. You don't have to specify a volume size like with EBS.
Scalibility : You add data as you go and the bucket will scale to infinite.
If your application become successful and you have more traffic you should go to a horizontal scaling using AutoScaling and LoadBalancer and that means multiple EC2 instances. In that case it will be so easier to access Data on S3 instead on an EBS attached to an instance.
Related
I have installed MariaDB on an EC2 instance. EC2 instance is small in size as our product is in development phase and we have very less traffic.
I want to keep an EBS volume for MariaDB data storage, so that as our traffic increases we can attach the EBS volume to a bigger EC2 instance and my data get transferred automatically.
Is keeping data storage on EBS(network) is a good approach? Will it not lead to more latent read/write?
If it won't impact my performance, what is the way to configure it?
An EC2 instance is made of 2 things - Compute & Storage.
Storage is basically an EBS volume (kind of Hard Disk of your laptop) that includes your OS files as well as other files for the software installed on your EC2.
Since, you are running your database on an EC2 instance, this EBS volume will also store your MariaDB database files on the same EBS where your OS is installed. It is not like, you can store MariaDB data on some other EBS.
So, when your data grows, you can expand EBS volume of your existing EC2 instance. You can follow this tutorial to learn how to do that - Increase the size of an Amazon EBS volume on an EC2 instance
Second thing is about performance. If your traffic increases, and you feel, you need more processing power (RAM or CPU), in that case you can upgrade your instance family. Refer this to see how to do that - Change the instance type.
Recommendation
As you will be spending a fixed monthly cost to run the EC2, my recommendation would be to use Aurora Serverless v2 with minimum capacity you need initially. Later, it can automatically scale itself when the traffic grows.
Is it more efficient to store icons and small decorative images (eg arrow.jpg or Submit.jpg etc) on S3 vs on the hosting server's local disk ?
images/arrow.jpg (relative url to the hosting server)
vs
https://aws.com/images/arrow.jpg
I am not concerned about the storage/space but the number of hits to the server because every page may contain 50 to 100 such icons and small images. I am not using CDN
When you say hosting server, you are ultimately going to store your volume on EBS, EFS ,( not instance store because it is not persistent).
There are many factors to distinguish based on efficiency but ultimately it will boil down to COST.
S3 is way cheaper, highly available, and more durable than EBS.
EBS is AZ bound but s3 is not. if you want to access to ebs in another az you will need to copy and snapshot and restore to another AZ.
with s3 you can utilize intelligent teiring with lifecycle policies to move your files automatically between different storage tiers for cost optimization.
however, if you want to fast access and low latency then EBS and EFS is a better option without a doubt.
s3 gives you unlimited storage but individual element size is limited to 5TB
For performance go for ebs , for cost go for s3.
NOTE : if you want to access your data on ebs or efs you would need to spin up an ec2 instance which also add cost
FAq for comparison
s3 v/s ebs v/s efs
This post on stack is also a good read AWS EFS vs EBS vs S3 (differences & when to use?)
I am currently looking for a way of easily distributing customised volumes to clients.
An approach I am looking at is creating RAW disk images, saving them to S3 and having clients import them as snapshots using the AWS CLI.
My question is - who pays for the data access request/data transfer?
...I'm assuming its bucket owner as there is no "requester-pays" option for the Import-Snapshot command. Has anybody done anything similar?
Another approach is directly sharing snapshots to a clients account - but this involves an added charge on our part to create the ideal sized volumes + generate the snapshots to share.
Is there a better method of generating + sharing data (essentially what would become EBS volumes) of varying sizes and content?
The easiest method would be to create an Amazon Machine Image (AMI), which is a bootable snapshot. You can list this it as a Community AMI.
Your clients can select the AMI when launching an Amazon EC2 instance. The boot disk will be exactly as you configured -- with the operating system, your application and all configurations that were saved on the disk.
There is no cost to you when a client uses the AMI.
See: Make an AMI public - Amazon Elastic Compute Cloud
I am new to cloud environment. I do understand the definition and storage types EBS and S3. I wanted to understand the application of EBS as compared to S3.
I do understand EBS looks like a device for heavy though put operations. I cannot find any application where this can be used in comparison to S3. I could think of putting server logs on EBS on magnetic storage, as one EBS can be attached to one instance.
S3 you can use the scaling property to add some heavy data and expand in real time. We can deploy our slef managed dbs on this service.
Please correct me if I am wrong. Please help me understand what is best suited for what and application of them in comparison with one another.
As you stated, they are primarily different types of storage:
Amazon Elastic Block Store (EBS) is a persistent disk-storage service, which provides storage volumes to a virtual machine (similar to VMDK files in VMWare)
Amazon Simple Storage Service (S3) is an object store system that stores files as objects and optionally makes them available across the Internet.
So, how do people choose which to use? It's quite simple... If they need a volume mounted on an Amazon EC2 instance, they need to use Amazon EBS. It gives them a C:, D: drive, etc in Windows and a mountable volume in Linux. Computers traditionally expect to have locally-attached disk storage. Put simply: If the operating system or an application running on an Amazon EC2 instance wants to store data locally, it will use EBS.
EBS Volumes are actually stored on two physical devices in case of failure, but an EBS volume appears as a single volume. The volume size must be selected when the volume is created. The volume exists in a single Availability Zone and can only be attached to EC2 instances in the same Availability Zone. EBS Volumes persist even when the attached EC2 instance is Stopped; when the instance is Started again, the disk remains attached and all data has been presrved.
Amazon S3, however, is something quite different. It is a storage service that allows files to be uploaded/downloaded (PutObject, GetObject) and files are replicated across at least three data centers. Files can optionally be accessed via the Internet via HTTP/HTTPS without requiring a web server. There are no limits on the amount of data that can be stored. Access can be granted per-object, per-bucket via a Bucket Policy, or via IAM Users and Groups.
Amazon S3 is a good option when data needs to be shared (both locally and across the Internet), retained for long periods, backed-up (or even for storing backups) and made accessible to other systems. However, applications need to specifically coded to use Amazon S3 and many traditional application expect to store data on a local drive rather than on a separate storage service.
While Amazon S3 has many benefits, there are still situations where Amazon EBS is a better storage choice:
When using applications that expect to store data locally
For storing temporary files
When applications want to partially update files, because the smallest storage unit in S3 is a file and updating a part of a file requires re-uploading the whole file
For very High-IO situations, such as databases (EBS Provisioned IOPS can provide volumes up to 20,000 IOPS)
For creating volume snapshots as backups
For creating Amazon Machine Images (AMIs) that can be used to boot EC2 instances
Bottom line: They are primarily different types of storage and each have their own usage sweet-spot, just like a Database is a good form of storage depending upon the situation.
I'm thinking about saving data from EC2 instances to the EBS and later save the result on S3. I don't have a lot of experience working with EBS, so my questions are:
How stable they are? I mean how often (if any) you had problem with EBS. Do they crash if overloaded or something like this?
What are the chances of loosing data from EBS?
Is it possible to mount one EBS to the multiple Instances? (let's say two ec2 share the same ebs )
I assume you've read AWS's take on EBS
Pretty stable. Last year, 10% of EBS volumes failed in 2-3 data centers in us-east for a couple hours. This is the only issue I've ever had with them.
I've never lost data from EBS. Even if I had, I take hourly snapshots (stored in s3), so I would have been just fine.
Not at the same time. To attach it to another instance, you must detach from the currently attached one.
Perhaps what you're look is s3fs - a way to mount s3 as a filesystem.
EBS is quite stable and every data you write is redundantly copied in 3 disks inside a AZ. If you take regular snapshots of your EBS volumes you can protect your data more. Since EBS operate in AZ scope it is recommended to moves assets like user documents, images, videos to Amazon S3. S3 offer more redundancy and availability than an EBS Volume.
You cannot mount single EBS volume to Multiple EC2 instances. You will have to use Solution like GlusterFS on AWS so that multiple EC2 instances can talk to common storage pool.