RDS - MySQL
We recently had a database outage due to a minor engine version upgrade.
We have RDS multi AZ setup. I can clearly see in master instance that that is saying Multi AZ yes and it is in a different availability zone.
In master I see events as
DB instance has a DB engine update
backing up DB
Finished backup
DB instance shut down
DB instance resumed
In the replica I am also seeing an event at the time of the incident
"Slave is disconnected from the master and is attempting to reconnect.
I have also seen in following event in replica prior to incident
"The free storage for DB instance is low at 5%."
I am trying to understand why didn't master failover to standby during the DB instance patched up.
What would be the issue? how can I find out why the failover didn't happen?
Was this MySQL?
If so, see Will a Multi-AZ deployment help reduce downtime during an Amazon RDS MySQL modification?
Because RDS MySQL doesn’t automate rolling upgrades, the DB engine version upgrade happens to both the primary and standby hosts at the same time. Therefore, a DB engine version upgrade doesn't benefit from a Multi-AZ deployment.
Also, see Best Practices for Upgrading Amazon RDS for MySQL and Amazon RDS for MariaDB:
One common fallacy is that Multi-AZ configurations prevents downtime during an upgrade. We do recommend that you use Multi-AZ for high availability, because it can prevent extended downtime due to hardware failure or a network outage. However, in the case of a MySQL or MariaDB engine upgrade, Multi-AZ doesn’t eliminate downtime. The slow shutdown and the physical changes made on the active server by the mysql_upgrade program require this downtime.
Related
RDS Multiple-AZ gives me the fail-over automatically when the master is down the standby instance will be promoted as mater, OFC they are synchronized,
With a few Read-Replica the over-all performance will be good.
So can I have them both at the same time, as shown below, for example:
Master --(across AZ)-- standby
|
Read-Replica instance1
|
Read-Replica instance2
In the above case, I believe if Master is down, the standby will be as master, but question is whether the Read-Replica will break or not?
It depends what flavor of RDS you are using - If you are using RDS for MySQL, MariaDB or PostgreSQL, you can use RDS read replicas with Multi-AZ configurations.
See Amazon RDS Read Replicas Now Support Multi-AZ Deployments
I am testing a few different scenarios for RDS implementation. For regular RDS our development account has a lambda function that will stop all RDS instances if they have a certain tag identifying them as development DBs. When it ran today on a Custom RDS for Oracle DB it shut down the EC2 instance connected with it but it only said Configuration Not Available on the RDS side. The whole idea behind this shutdown lambda is to save money at the end of each day and shut down development machines. My question is the Custom Oracle RDS instance shut down or not? If not then how does one shut one down? Thanks
I have a Multi-AZ database instance in AWS RDS. I would like to connect to its secondary server located in the Secondary Zone for the instance using the MySQL CLI. Is this possible to do? If so, how do I find the correct endpoint to connect to?
I would like to use this access to the secondary server to run nightly analytics queries that I would prefer to avoid hitting the primary database server. It's okay for these queries to see a lag in replicated data from the primary database server. And I already have a database user created that limits those analytics queries to truly be read-only, so I don't need to worry about the analytics process accidentally issuing writes to the secondary database.
You can't connect to the secondary server in an RDS MySQL Multi-AZ setup. You would have to add a read-replica server in order to do this.
Note that if you switched to Aurora MySQL a read-replica would act as your Multi-AZ failover server as well. There is a good table here listing the function of Multi-AZ standby servers and read-replica servers in RDS and Aurora databases.
If I launch a RDS instance in one of my private VPCs, will that be able to update itself provided that I have enabled Auto Minor Version Upgrade?
Amazon RDS is a fully-managed service. All maintenance is performed by AWS "in the background" and does not involve traffic going through your VPC. (This also applies to snapshots and instance modifications, such as changing the DB instance type.)
Minor version upgrades may involve an outage period.
If you are running in Multi-AZ mode, then the Secondary server is upgraded, the Secondary server becomes the new Master server and then the old Master is updated and becomes the new Secondary server. This means that applications will be disconnect and need to reconnect to the database, but there is no outage period.
I'm currently using RDS MultiAZ from Amazon Web Services on my project, and I was hoping to use ElastiCache to improve the speed of my queries. However I noticed that on ElastiCache I have to define which zone I'm interesting in using.
Just to check if I got it right: MultiAZ means that I have 2 database servers on 2 zones (I'm using South America region): on zone A I have a read and write server (Master) and on zone B I have a read server (Slave). If for any reason zone A goes down, zone B becomes the Master until Zone A returns.
Now how do use ElastiCache (I'm using Memcache) in this case? I can't create a cache cluster with a single endpoint to connect, and 2 nodes (one in each zone). I need to have 1 cache cluster for each zone, and 2 codes for my application so they'll connect to the correct zone?
Already asked that on AWS forums a month ago, but had no response.
Thanks!
Amazon ElastiCache clusters are per-AZ and there is no Multi-AZ for ElastiCache as there is for RDS (you are right, that is master/slave replication). So you would need to design around that. This is very context dependent, but here are three ideas:
Failure Recovery: monitor your cache cluster and, in the event of a failure, spin a new one in another AZ.
Master/Slave: have a standby cache cluster and, in the event of a failure, reroute and scale to the slave.
Multi master: have per-AZ cache clusters always up under a Elastic Load Balancer.
EDIT
This answer considers ElasticCache for Memcached. For Redis there is Multi-AZ (master/slave) support.