I've read the Amazon's RDS Multi-AZ documentation which explains how a new primary instance is created during a failure of the primary instance. But then after the standby becomes the primary, how does the process work for creating a new standby? Is it created fresh? Is the old primary now used as a standby? What if the old primary volume failed? Do they just replace it with a new one?
Related
I noticed when a Cross Region Read Replica was created for my AWS Aurora cluster - it has both a Write and Reader instance (Similar to my Primary which naturally has a writer in addition to a reader instance). In the Cluster Configuration for the Cross Region Replica cluster - I can see this "Replica" cluster indeed has the Replication source tag and its correctly getting all data flowed asynchronously.
Couple of questions I need help understanding this:
should a Cross Region Replica have a Writer?
Should I write to it in case of a disaster in the source region?
I had to explicitly make the read-replica instance (in replica region) as read_only. Otherwise read-replica accepts writes to it.
Found relevant answer in AWS re:Post:
"You're replicating to a new cluster, and each cluster needs a writer
instance. It is recommended that customers apply the read_only
parameter to the replica, but by default they are able to write to
this instance. This allows for architectures where the replica is read
& writable.
https://repost.aws/questions/QUrCbnj0u4TWaz-A1uR-QDPQ/aurora-create-cross-region-read-replica-vs-add-region
I think the name "writer" is a bit misleading.
In the doc for Aurora endpoint:
A cluster endpoint (or writer endpoint) for an Aurora DB cluster connects to the current primary DB instance for that DB cluster. This endpoint is the only one that can perform write operations such as DDL statements. Because of this, the cluster endpoint is the one that you connect to when you first set up a cluster or when your cluster only contains a single DB instance. Each Aurora DB cluster has one cluster endpoint and one primary DB instance.
So the writer instance is the same entity as the primary instance of the cluster.
In the doc for Cross-Region Replication:
When you create a cross-Region read replica for Aurora MySQL by using the AWS Management Console, Amazon RDS creates a DB cluster in the target AWS Region, and then automatically creates a DB instance that is the primary instance for that DB cluster.
For cross region replication, the new cluster (writer instance) will be created.
You don't need to take care of the writer instance. For the disaster recovery, promote read replica.
I would like to setup a PostgreSQL database with read-only standby with automatic failover ability.
Does Amazon RDS with Multi-AZ have automatic failover ability?
If yes, will the endpoint/DNS automatically point to the new standby database?
There are 2 scenarios to this question as it was little ambiguous in the question.
scenario 1 :- rds with multi AZ.
Here standby instance will have automatic failover.
Here the endpoint/DNS automatically point to the new standby database, no manual intervention is needed.
But read and write is being taken by only the master DB and sync replication takes with standby database.
scenario 2 :- rds with multi AZ where read replicas is also there.
In this scenario read replicas are not treated like standby database database by default.howver they can be used like standby.
failover is not automatic.
You will need to manually update your dns.
article from aws says
A Read Replica in a different region than the source database can be used as a standby database and promoted to become the new production database in case of a regional disruption.
also a similar question on stack should also help you Difference between "Multi-AZ Deployment" and "Read Replica Verison Multi-AZ Deployment"
Scenario 2 seems to have slightly evolved since 2021.
Now this has become Multi-AZ DB Cluster
Reader DB instances act as automatic failover targets and also serve
read traffic to increase application read throughput. If an outage
occurs on your writer DB instance, RDS manages failover to one of the
reader DB instances. RDS does this based on which reader DB instance
has the most recent change record.
Because there is no election involved in the fail over, so when the primary is down, it will auto switch to the only one slave automatically?
The answer to your question is within the docs. https://docs.aws.amazon.com/documentdb/latest/developerguide/replication.html
Specifically :
If a failure occurs in the primary instance (AZ1), a failover is triggered, and one of the existing replicas is promoted to primary. When the old primary recovers, it becomes a replica in the same Availability Zone in which it was provisioned (AZ1). When you provision a three-instance cluster, Amazon DocumentDB continues to preserve that three-instance cluster. Amazon DocumentDB automatically handles detection, failover, and recovery of instance failures without any manual intervention.
and
You can specify Amazon DocumentDB replicas as failover targets. That is, if the primary instance fails, the specified Amazon DocumentDB replica or replica from a tier is promoted to the primary instance. There is a brief interruption during which read and write requests made to the primary instance fail with an exception. If your Amazon DocumentDB cluster doesn't include any Amazon DocumentDB replicas, when the primary instance fails, it is re-created. Promoting an Amazon DocumentDB replica is much faster than re-creating the primary instance.
Read more about fault tolerance here.
https://docs.aws.amazon.com/documentdb/latest/developerguide/db-cluster-fault-tolerance.html
When I restore a MySQL snapshot, I'm given the option to make the new instance Multi-AZ. However for some reason when I restore to Aurora, the "Multi-AZ Deployment" selection is disabled.
I thought possibly this meant it was automatically enabled. However when I go to the running instance details, it specifically lists Multi-AZ: "No".
Is there something I'm missing? Is there some other step I need to do to make Aurora Multi-AZ?
Multi-AZ doesn't mean the same thing in Aurora as it does for MySQL and MariaDB. With Aurora, any replica in the cluster can take over for the master on failure (though the selection is based on priority, so it isn't just a randomly selected replica, if there is more than one replica), and the storage is already/always multi-AZ in Aurora because that's a core part of the Aurora design. So, Aurora can be "converted" to Multi-AZ.
Apparently, that's the only way it is done when creating an Aurora instance from a snapshot, based on this:
You can migrate a DB snapshot of an Amazon RDS MySQL DB instance to create an Aurora DB cluster.
...
You can migrate either a manual or automated DB snapshot. After the DB cluster is created, you can then create optional Aurora Replicas.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Migrate.RDSMySQL.html
If you create a multi-AZ Aurora cluster without a snapshot, that's all that happens -- a master and a replica are created. In that case, you can actually delete the replica and make it non-multi-AZ, which is a different process than what's involved with MySQL or MariaDB.
"Create a cluster" -- as used in the quote, above -- is a potentially confusing term, since you would naturally assume a cluster means two or more, but in fact an Aurora cluster can technically be a "cluster" of just one instance. Every Aurora instance is part of exactly one cluster. One instance is the primary (master) and any additional instances are replicas.
If the DB cluster doesn't contain any Aurora Replicas, then the primary instance is recreated during a failure event.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Managing.html#Aurora.Managing.FaultTolerance
We can easily promote RDS read replica using console and CLI, but is there a way to give master access means promote to primary using cloud formation?
CloudFormation uses the property "SourceDBInstanceIdentifier" to determine whether a DBInstance is a read replica or not. If this property has any non-empty value, the DBInstance will be created as a read replica.
If you remove this property once a read replica is created and update your CloudFormation stack, a new DBInstance will be created that won't be a read replica, it will be it's own standalone database. However, the new standalone database will not have the contents of the read replica.
The solution to this will be to regularly back up your read replica. When you want to promote your read replica, remove "SourceDBInstanceIdentifier" and set "DBSnapshotIdentifier" to the most recent snapshot of the read replica. This should effectively promote your read replica to master with most the data still intact
The RDS standby replica (multi-az) will automatically be switched to master in case of failures on the master node.
How you setup the multi-az replica (could be through cloudformation) will determine the automatic switch over. Please note that the standby replica becomes the master only when there is an AZ outage, primary instance fail, software patching and not when there is scenarios of locks or database level operation timeouts.