I use AWS and have automatic backup enabled.
For one of our client, we need to know exactly where the backup data is stored.
From the AWS FAQ website, I can see that:
Q: Where are my automated backups and DB Snapshots stored and how do I manage their retention?
Amazon RDS DB snapshots and automated backups are stored in S3.
My understanding is that you can have a S3 instance located anywhere you want, so it's not clear to me where this data is.
Just to be clear, I'm interested by the physical location (is it Europe, US....)
It is stored in the same AWS region where the RDS instance is located.
When you directly store data in S3, you store it in an S3 container called a bucket (S3 doesn't use the term "instance") in the AWS region you choose, and the data always remains only in that region.
RDS snapshots and backups are not something you store directly -- RDS stores it for you, on your behalf -- so there is no option to select the bucket or region: it is always stored in an S3 bucket in the same AWS region where the RDS instance is located. This can't be modified.
The data from RDS backups and snapshots is not visible to you from the S3 console, because it is not stored in one of your S3 buckets -- it is stored in a bucket owned and controlled by the RDS service within the region.
According to this :
Your Amazon RDS backup storage for each region is composed of the automated backups and manual DB snapshots for that region. Your backup storage is equivalent to the sum of the database storage for all instances in that region
I think that means that it is stored in that region only and s3 stores data like this :
Amazon S3 redundantly stores data in multiple facilities and on multiple devices within each facility. To increase durability, Amazon S3 synchronously stores your data across multiple facilities before confirming that the data has been successfully stored.
https://aws.amazon.com/rds/details/backup/
...By default, Amazon RDS creates and saves automated backups of your DB instance securely in Amazon S3 for a user-specified retention period.
...Database snapshots are user-initiated backups of your instance stored in Amazon S3 that are kept until you explicitly delete them
Related
I see that AWS DocumentDB is creating automatic snapshots daily and I myself can create manual snapshots from AWS Console. The documentation says that the snapshot is saved in S3 but it is not visible on S3 to me.
I basically want to move the DocumentDB data to S3 in order to propagate it further to other AWS services for monitoring purposes. I was thinking if I can trigger a manual snapshot daily and have a lambda trigger on S3 file upload by DocumentDB.
How can I see the automatic and manual snapshot created by DocumentDB on S3?
Backups in Amazon DocumentDB are stored in service-managed S3 buckets and thus there is no way to access the backups directly.
Two options here are:
1/use mongodump/mongoexport on a schedule: https://docs.aws.amazon.com/documentdb/latest/developerguide/backup_restore-dump_restore_import_export_data.html
2/use change streams to incrementally write to S3: https://docs.aws.amazon.com/documentdb/latest/developerguide/change_streams.html
We have S3 replication infrastructure in place to redirect PUTs/GETs to replica (destination) S3 bucket if primary (source) is down.
But I'm wondering how to copy objects from destination bucket to source once primary is restored.
You can use Cross-Region Replication - Amazon Simple Storage Service.
This can also be configured for bi-directional sync:
Configure CRR for bucket-1 to replicate to bucket-2
Configure CRR for bucket-2 to replicate to bucket-1
I tested it and it works!
CRR requires that you have Versioning activated in both buckets. This means that if objects are overwritten, then the previous versions of those objects are still retained. You will be charged for storage of all the versions of each object. You can delete old versions if desired.
I have created an S3 Bucket and mounted into one of my EC2 servers in the same region. Then I put data into the bucket using FTP account created for that EC2 instance. Finally, I access the data by Http request.
I'm not accessing S3 bucket directly from Internet, either for writing or accessing. All the data transferred through EC2 instance.
So, I assume per month charges as below, for fully used up 1TB S3 bucket (standard storage),
Storage Pricing - $0.0300*1024 = $30.72
Request Pricing - $0.005*10 = $0.05 (Assumed 10,000 request per month )
Data Transfer Pricing - Nill (Since the bucket is not being accessed directly)
Is that correct? or data transfer pricing is applicable?
Ref: Pricing Details
You do not pay for data transfer between S3 and EC2 in the same region, however you pay for Data Transfer OUT From Amazon EC2 To Internet or EC2 instance in a different availability zone in the same region.
See EC2 pricing for more details.
If you transfer 1TB of data OUT to Internet from AWS, either directly from S3 or through EC2 instance, you will pay the same price.
TIP:
If you are transferring big amount of data from S3 out to Internet, look into CloudFront. Data transfer EC2/S3/ELB -> CloudFront is free of charge and CloudFront has cheaper rates per Gb compared to downloading files directly from S3.
EDIT:
see #Michael - sqlbot's comment, this is often but not always true depending on S3 Bucket's region and CloudFront edge location serving the content.
TIP 2:
For really large amounts of data it might be worth setting up DirectConnect connection (private connection from your office / on-premise setup to AWS). Then Data Transfer becomes even cheaper per Gb, however you start paying hourly rate for your DirectConnect link. Do the math to calculate what's best for you.
If you are reading data from S3 to your EC2 instance, and the S3 bucket is in the same region as your EC2 instance, then there are no data transfer costs.
Broken down:
There is no “data transfer in” costs to your EC2 instance if the data is coming from an S3 bucket in the same region: EC2 Instance Pricing – Amazon Web Services (AWS)
There is no “data transfer out” costs from your S3 bucket if the data is going to an EC2 instance in the same region: Cloud Storage Pricing – Amazon Simple Storage Service (S3) – AWS
There is no "data tansfer out" costs from EC2 to S3.
More info:
https://www.quora.com/In-AWS-EC2-what-counts-towards-data-transfer-costs
I would like to backup a snapshot of my Amazon Redshift cluster into Amazon Glacier.
I don't see a way to do that using the API of either Redshift or Glacier. I also don't see a way to export a Redshift snapshot to a custom S3 bucket so that I can write a script to move the files into Glacier.
Any suggestion on how I should accomplish this?
There is no function in Amazon Redshift to export data directly to Amazon Glacier.
Amazon Redshift snapshots, while stored in Amazon S3, are only accessible via the Amazon Redshift console for restoring data back to Redshift. The snapshots are not accessible for any other purpose (eg moving to Amazon Glacier).
The closest option for moving data from Redshift to Glacier would be to use the Redshift UNLOAD command to export data to files in Amazon S3, and then to lifecycle the data from S3 into Glacier.
Alternatively, simply keep the data in Redshift snapshots. Backup storage beyond the provisioned storage size of your cluster and backups stored after your cluster is terminated are billed at standard Amazon S3 rates. This has the benefit of being easily loadable back into a Redshift cluster. While you'd be paying slightly more for storage (compared to Glacier), the real cost saving is in the convenience of quickly restoring the data in future.
Any use case to take a backup as Redshift automatically keeps snapshots. Here is a reference link
I want to keep a backup of an AWS s3 bucket. If I use Glacier, it will archive the files from the bucket and moved to the Glacier but it will also delete the files from s3. I don't want to delete the files from s3. One option is to try with EBS volume. You can mount the AWS s3 bucket with s3fs and copy it to the EBS volume. Another way is do an rsync of the existing bucket to a new bucket which will act as a clone. Is there any other way ?
What you are looking for is cross-region replication:
https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/
setup versioning and setup the replication.
on the target bucket you could setup a policy to archive to Glacier (or you could just use the bucket as a backup as is).
(this will only work between 2 regions, i.e. buckets cannot be in the same region)
If you want your data to be present in both primary and backup locations then this is more of a data replication use case.
Consider using AWS Lambda which is an event driven compute service.
You can write a simple piece of code to copy the data wherever you want. This will execute every time there is a change in S3 bucket.
For more info check the official documentation.