I'm researching about AWS RDS bidirectional replication. I know that RDS has the read replica, but I need the bidirectional replication for disaster recovery.
Would anyone have the experience with this before? I'm very appreciated for your help.
AWS RDS does have multi master capability with Aurora. See documentation below:
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-multi-master.html
It supports replication across multiple master across multiple AZ. This is for a region only. Since you mentioned disaster recovery, it might not satisfy your requirement but you can evaluate it to decide as per your requirements.
See the blog post below which talks about replication architecture of multi master Auora
https://aws.amazon.com/blogs/database/building-highly-available-mysql-applications-using-amazon-aurora-mmsr/
For cross region, you can use Auora Global database but that supports only 1 writer instance.
What db engine are you using in RDS? is that mysql, pgsql , sql server or oracle? Aurora multi-master only supports MySQL, and even multi-master we are talking about both master in same AWS region, which may not satisfy your requirement of "DR". Why do you need bidirectional in first place? If the writes from "master A" and "master B" are isolated, i.e. A and B did not update same portion of same table at the same time, then you may consider to use AWS DMS bi-directional replication to setup a DR db for your primary DB, but notice that AWS DMS bi-directional replication is not a multi-master db solution, it won't handle any data conflict / inconsistency if primary and replica db update same rows. For detail please refer to AWS doc: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Task.CDC.html#CHAP_Task.CDC.Bidirectional
Related
We need to know what are the best options to set AWS RDS instance (Aurora mysql) that is standalone and does not get traffic from actual RDS cluster.
Requirement is for our data team to write analytical queries but we do not want it to impact actual application and DB performance. Hence we need a DB which always has near to live data but live traffic or application does not connect to this instance.
Need to know which fits better, DL clone OR AWS Pilot light OR AWS Warn standby OR AWS hot standby OR
multi-AZ configuration.
Kindly let us know which one would fit our requirement better.
We have so far read about below 3 options,
AWS Amazon Aurora DB clone, https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Clone.html
AWS Pilot light or AWS Warn standby or AWS hot standby
. https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iii-pilot- light-and-warm-standby/
With multi-AZ configuration, we can create a new instance in new AZ, so that his instance will have a different host (kind off, a fail over strategy), where traffic to his instance will be from our queries and not from live prod application, unless there is some fail over issue.
Option 1, Aurora cloning says
Run workload-intensive operations, such as exporting data or running analytical queries on the clone.
...which seems to be your use case here.
Just be aware that the clone will not see any changes to the original data after it is made. So you will need to periodically delete and re-clone to get the updated data
Regarding option 2, I wrote those blog posts, and I do not think that approach suits your use case. That approach is for disaster recovery
Option 3 may work. To modify it a bit, the concept here is to create an Aurora Replica, which as you say is a separate instance. The problem here is the reader endpoint for your production workload, it may hit that instance (which is not what you want)
EDIT: Adding new option 4
Option 4. Check out Amazon Aurora zero-ETL integration with Amazon Redshift. This zero-ETL integration also enables you to analyze data from multiple Aurora database clusters in an Amazon Redshift cluster.
I would like to setup a PostgreSQL database with read-only standby with automatic failover ability.
Does Amazon RDS with Multi-AZ have automatic failover ability?
If yes, will the endpoint/DNS automatically point to the new standby database?
There are 2 scenarios to this question as it was little ambiguous in the question.
scenario 1 :- rds with multi AZ.
Here standby instance will have automatic failover.
Here the endpoint/DNS automatically point to the new standby database, no manual intervention is needed.
But read and write is being taken by only the master DB and sync replication takes with standby database.
scenario 2 :- rds with multi AZ where read replicas is also there.
In this scenario read replicas are not treated like standby database database by default.howver they can be used like standby.
failover is not automatic.
You will need to manually update your dns.
article from aws says
A Read Replica in a different region than the source database can be used as a standby database and promoted to become the new production database in case of a regional disruption.
also a similar question on stack should also help you Difference between "Multi-AZ Deployment" and "Read Replica Verison Multi-AZ Deployment"
Scenario 2 seems to have slightly evolved since 2021.
Now this has become Multi-AZ DB Cluster
Reader DB instances act as automatic failover targets and also serve
read traffic to increase application read throughput. If an outage
occurs on your writer DB instance, RDS manages failover to one of the
reader DB instances. RDS does this based on which reader DB instance
has the most recent change record.
I have a Amazon RDS MySQL database in the us-east-1 region. I have a few read replicas in that region for the master db as well. I'm working on a new app which will require the same data from that db but I would like to host the app and create a new db instance in us-west-2 for latency reasons. I still want to host the original application in us-east 1 though.
Is it possible in Amazon RDS to create a cross region bidirectional database so the db in us-west-2 is a replica of the one in us-east-1 that I can also write to?
I rather not use Amazon Aurora right now if possible but if that's the only answer then I guess it's something I will have to do.
Amazon RDS does not support multi-master clusters across regions.
From Working with Aurora Multi-Master Clusters - Amazon Aurora:
You can't enable cross-region replicas from multi-master clusters.
Both Aurora and the non-Aurora databases in Amazon RDS can provide multi-region read replicas, but these are one-way replicas.
If you happen to be using Aurora MySQL - take a look at Write Forwarding.
This allows replicas to send writes to the primary cluster and then (depending on your selected consistency option) waits for relevant changes to be replicated back to itself and other region replicas.
This can help get around the cross-region limitations of Multi-Master and lets you at least simulate a 2-way replication across AWS regions.
Anyone knows what AWS uses to do RDS DB instance sync. replication? DRBD or any other low level device block transfer or something else?
cause there are situations where the standby DB instance fails when a failure occurs on master/primary DB instance?
Note: claimed in RDS section "AWS Cloud Practitioner Essentials (Second Edition): AWS Integrated Services" digital training video
AWS utilizes database physical and logical database replication as appropriate for them.
As per the official documentation
Multi-AZ deployments for the MySQL, MariaDB, Oracle, and PostgreSQL
engines utilize synchronous physical replication to keep data on the
standby up-to-date with the primary. Multi-AZ deployments for the SQL
Server engine use synchronous logical replication to achieve the same
result, employing SQL Server-native Mirroring technology. Both
approaches safeguard your data in the event of a DB Instance failure
or loss of an Availability Zone.
Official slide shows that RDS actually utilizes DRBD for physical replication.
https://www.slideshare.net/AmazonWebServices/amazon-rds-with-amazon-aurora-aws-public-sector-summit-2016
How to scale horizontally amazon RDS instance? EC2 and load balancer+autoscaling is extremly easy to implement, but if I want scaling amazon RDS?
I can ugrade my RDS instance with more powerfull instance or I can create a read replica and I can direct SELECT queries to it. But in this mode I don't scale anything if I have a read-oriented web application. So, can I create RDS read replica with autoscaling and balance them with load balancer?
You can use a HAProxy to load-balance Amazon RDS Read Replica's. Check this http://harish11g.blogspot.ro/2013/08/Load-balancing-Amazon-RDS-MySQL-read-replica-slaves-using-HAProxy.html.
Hope this helps.
Note RDS covers several database engines- mysql, postgresql, Oracle, MSSQL.
Generally speaking, you can scale up (larger instance), use readonly databases, or shard. If you are using mysql, look at AWS Aurora. Think about using the database optimally- perhaps combining with memcached or Redis (both available under AWS Elasticache). Think about using a search engine (lucene, elasticsearch, cloudsearch).
Some general resources:
http://highscalability.com/
https://gigaom.com/2011/12/06/facebook-shares-some-secrets-on-making-mysql-scale/
If you are using PostgreSQL and have a workload that can be partitioned by a certain key and doesn't require complex transactions, then you could take a look at the pg_shard extension. pg_shard lets you create distributed tables that are sharded across multiple servers. Queries on the distributed table will be transparently routed to the right shard.
Even though RDS doesn't have the pg_shard extension installed, you can set up one or PostgreSQL servers on EC2 with the pg_shard extension and use RDS nodes as worker nodes. The pg_shard node only needs to store a tiny bit of metadata which can be backed up in one of the worker nodes, so they are relatively low maintenance and can be scaled out to accommodate higher query rates.
A guide with a link to a CloudFormation template to set everything up automatically is available at: https://www.citusdata.com/blog/14-marco/178-scaling-out-postgresql-on-amazon-rds-using-masterless-pg-shard