We have 4 standalone, non-multiAZ aurora DB instances in a VPC and we want to move them to aurora instances in another VPC.
As I understand there are 3 ways to migrate DB instances:
1) Modify the DB instance's Subnet group to change the VPC.
However this is not supported for Aurora instances yet.
2) Create a read replica, and when slave catches up, stop the slave and take it's snapshot to create a DB instance in the different VPC and use external slave then to again resume the replication.
I have a few question around this second method. As Aurora is using some different
replication method, the result of show slave status; command is empty.
Also by default the binlog_format is OFF so not sure if I have to modify it and
then restart the instance to take note of the binlog etc.
Have anyone done this before and can guide me? I don't want to restart the instance
to later find out that it is not working, as it is a very critical DB and I want to
minimize the downtime.
3) Use Amazon DMS service, however I cannot find the source DB details for Aurora in the documentation here.
I need to find out what all permissions to give to the replication user
I'll create for this. This command in aurora is not working,
GRANT REPLICATION CLIENT, REPLICATION SLAVE ON . TO 'replication_user'#'%'
IDENTIFIED BY 'aaaaaa';
Any help would be appreciated.
You should be able to restore from a snapshot. I migrated aurora across VPCs using this approach. I hoped we could create a read-replica in a different VPC but at the moment I think you can only create one in a different region.
One strange side effect I have found and i'm not sure why this happens is that in the orginal cluster multi-az was 2 Zones and in the restored cluster multi-az is No. This hasn't affected things that I can see. I still have a cluster with writer and reader
Create a snapshot and restore it to a new cluster. When you do that, use a new subnet group created for the second VPC. That's the best way to achieve this. Like you called out, you cannot change subnet group for an existing cluster.
In your approach #2, you mention having to create a slave and then taking a snapshot. That's not required. All instances in a cluster are connected to the same shared volume, so you can just go ahead and create a snapshot from your single instance cluster directly. Just make a note that snapshots are a cluster level action, and not an instance level action in Aurora.
From https://aws.amazon.com/premiumsupport/knowledge-center/rds-vpc-aurora-cluster/
It states that
Create a clone in a different VPC
If you clone a database in an Aurora cluster, you can change the VPC of the clone. > However, the subnets in the VPC must map to the same set of Availability Zones. For more information, see Cloning Databases in an Aurora DB Cluster.
It does work though can't see a way via the console to break the replication. We are using this as a faster way of migrating than snapshot and restore. Stopping the original master would break replication but though you could do it cleaner
Related
We need to know what are the best options to set AWS RDS instance (Aurora mysql) that is standalone and does not get traffic from actual RDS cluster.
Requirement is for our data team to write analytical queries but we do not want it to impact actual application and DB performance. Hence we need a DB which always has near to live data but live traffic or application does not connect to this instance.
Need to know which fits better, DL clone OR AWS Pilot light OR AWS Warn standby OR AWS hot standby OR
multi-AZ configuration.
Kindly let us know which one would fit our requirement better.
We have so far read about below 3 options,
AWS Amazon Aurora DB clone, https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Managing.Clone.html
AWS Pilot light or AWS Warn standby or AWS hot standby
. https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-iii-pilot- light-and-warm-standby/
With multi-AZ configuration, we can create a new instance in new AZ, so that his instance will have a different host (kind off, a fail over strategy), where traffic to his instance will be from our queries and not from live prod application, unless there is some fail over issue.
Option 1, Aurora cloning says
Run workload-intensive operations, such as exporting data or running analytical queries on the clone.
...which seems to be your use case here.
Just be aware that the clone will not see any changes to the original data after it is made. So you will need to periodically delete and re-clone to get the updated data
Regarding option 2, I wrote those blog posts, and I do not think that approach suits your use case. That approach is for disaster recovery
Option 3 may work. To modify it a bit, the concept here is to create an Aurora Replica, which as you say is a separate instance. The problem here is the reader endpoint for your production workload, it may hit that instance (which is not what you want)
EDIT: Adding new option 4
Option 4. Check out Amazon Aurora zero-ETL integration with Amazon Redshift. This zero-ETL integration also enables you to analyze data from multiple Aurora database clusters in an Amazon Redshift cluster.
I am working on a process to accomplish the following goal:
Copy a DB snapshot from one region to the next.
After copying the DB to the next region, I want to restore it.
I want to also put the newly restored DB instance into a separate VPC other than "default", so that I can have my AWS Workspaces clients get to it.
I've got steps 1 and 2 working perfectly; however, it doesn't seem like the process to restore a DB instance from a snapshot gives you a lot of options from the API. Here's the API process according to AWS's documentation: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/RDS/Client.html#restore_db_instance_from_db_snapshot-instance_method
According to those options, the only option I see available anywhere remotely related to the VPC is the vpc_security_ids; however, I tried specifying a vpc_security_id that belongs into the VPC that I want the RDS instance to be restored to, btu I got the following error in the console:
Aws::RDS::Errors::InvalidParameterCombination: The DB instance and EC2 security group are in different VPCs. The DB instance is in vpc-76cf380e and the EC2 security group is in vpc-012999f6551c713c6
from /Users/nutella/.gem/ruby/2.6.0/gems/aws-sdk-core-3.113.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
It's clearly possible to do this from the RDS console when restoring a snapshot, as you can just simply select the VPC that you want to restore the db instance into, but it's not clear on how to do this with the SDK.
Our current server consisting of an 2x EC2 instances and RDS (Read/Write) database is in Mumbai Region. However I would like to copy everything (2x EC2 & RDS (R/W)) across to Sydney, and other to other regions.
Ideally I would like to replicate the contents in those instances as well.
Does anyone know a quick and easy way of doing this?
Edit 25/01/2019:
However I would like to copy everything including what ever is inside the instances (2x EC2s and the RDSs)
Edit 29/01/2019:
The purpose is to "scale/expand out". I want to have the same infrastructure replicated 1-to-1 (exactly/identically) across various regions.
It is simple!
- For EC2 - you need to create an AMI of those instances then right click on the AMI you've just created and choose "copy AMI" to the designated region.
For RDS
If you just wanna copy data to another region then take a snapshot then copy that snapshot to destination region
If you want to make the RDS replicate to another region continuously then you need to create a read-replica from your RDS instance.
Option for replicating environment depends on how much downtime can you tolerate.
If you are okay with downtime
1. Copy the AMI of EC2 instance and snapshot of RDS to another regions
2. Bring up your new environment.
This is perfect for non critial workload
If this is critical application
1. Copy the AMI of ec2 instance ( I am assuming this would be your web/app instnaces) For real time replication use rsync or robocopy .. or solution like cloudendure .
2. Create a new RDS instance in sydney
3. USE DMS migration tool .. create source and target relationship
4. once insync cut off the relation bring new environment in sydney
As suggested by previous answers for EC2 you can create AMIs and then move the AMI to a different region.
For RDS, you can either create read replicas (and read replicas of read replicas, but beware of latency), read replicas are used to mainly improve read performance of your app.
You can also create a Multi AZ backup which will act as a disaster recovery site. However, note that Multi-AZ is only used in case of a failover. Moreover, Multi-AZ involves Synchronous data copy and read replicas are asynchronous, so read replicas can demonstrate eventual consistency behavior.
But the real question here is - What are you trying to achieve?
Are you trying to "scale out" your infrastructure to support huge traffic to your application? Or are you simply trying to setup disaster recovery (DR)?
If your answer is DR, then the approach is pretty straight forward with Multi AZ and EC2 instance snapshots. But if the answer is scaling out and performance, you really need to be thinking of better strategies such as using Cloudfront (CDN) if it is a web app, using Elasticache in-memory cache for frequently read data, or RDS read replicas, using Elastic Load Balancers with Dynamic/Step scale-out/scale-in. Other, methods would be to evaluate the type of RDS storage subsystem used i.e. using Provisional IOPs vs. Using General Purpose SSD, checking if there are any NAT “instance” bottlenecks in your VPC and so on.
It may be tempting to spin up all these redundant copies of EC2 AMIs or RDS read replicas with a click of a button, but you really need to be thinking about the cost you are going to incur on a monthly basis for completely un-used resources.
I modified my RDS instance to "Multi AZ : Yes". My primary RDS instance is in us-west-1a and for multi-AZ the secondary zone is shown as us-west-1c. I wanted to verify if whatever changes I am making on my primary database are getting copied to the Multi-AZ standby database quickly.
But I am not able to understand what endpoint URL should I use to login into Multi-AZ database. I am thinking the end point URL would be different from primary. Could you please help me on this?
You do not have access to the secondary RDS instance in a Multi-AZ configuration. You just need to trust that AWS is replicating data correctly. In a Multi-AZ configuration, RDS will write to both replicas syncronously. It will not return the write request until both replicas have written correctly.
To access a Multi-AZ instance, you issue your reads and writes to the single RDS endpoint. In case of an issue, AWS will modify the DNS entry for that endpoint to point to the secondary replica. So as long as you are using the endpoint DNS record, and not caching the IP address when accessing the RDS instance, the failover process should be transparent to you with only a minute or so of "downtime".
take a look at https://aws.amazon.com/rds/details/multi-az/. You don't typically interact with the replica(s) of RDS resources directly; AFAIK ( I'm not an rds expert ) you can't do what you're describing. The idea is that RDS does that for you, automatically keeping a consistent replica in a different AZ, and providing to you a consistent DNS endpoint.
Although OP asks for "verify data is copied quickly", Google pointed me here to "verify a multi-AZ RDS deploy". I'll share what I found in hopes that it's halfway helpful.
In the RDS console, there is an option on reboot to Reboot from failover which doesn't appear on a standard deploy.
Standard deploys do not have this option, which was a small but satisfying indication that the multi-AZ was acting as expected.
Source (and generally a pretty good read)
Q: Can I initiate a “forced failover” for my Multi-AZ DB instance
deployment?
Amazon RDS will automatically fail over without user intervention
under a variety of failure conditions. In addition, Amazon RDS
provides an option to initiate a failover when rebooting your
instance. You can access this feature via the AWS Management Console
or when using the RebootDBInstance API call.
can someone help me with these questions please :
1- the documentation stated that Aurora will automatically fail-over to the read replicas, my question is how does it select the replica which will be promoted if you have more than one with different instances class?
2- can I disable this automatic fail-over (just asking, not stating that I will do it)
3- what is the purpose of multi-az in Aurora if you can have the same effect with much more control on instances classes while creating replicas and let Aurora do the auto fail-over for you. please correct me if I am wrong with this assumption.
thanks in advance
The algorithm for election of a new master in case of failure is not really documented... but it doesn't seem to matter, because Aurora replicas seem to be different than other RDS replicas, because all the instances in the cluster are necessarily of the same instance class.
Unlike other RDS offerings, read replicas in Aurora do not appear to have an independent copy of the backing store -- instead, the backing store itself provides redundancy, being replicated at the storage level with two copies in each of three availability zones.
The cluster volume is made up of multiple copies of the data for the DB cluster, but the data in the cluster volume is represented as a single, logical volume to the primary and Aurora Replicas in the DB cluster.
Because the cluster volume is shared among all instances in your DB cluster, no additional work is required to replicate a copy of the data for each Aurora Replica.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Replication.html
Multi-AZ in Aurora is also not the same thing -- with other RDS products, multi-AZ involves a second, invisible instance, running in parallel with the master. The Aurora literature uses the phrase "multi-AZ technology," but the meaning appears to be different. Note that the Aurora pricing tables don't show a separate pricing rate for "multi-AZ" the way MySQL and MariaDB do.
Failover doesn't appear to be something that can be disabled. Even if you have no replicas, Aurora will still "fail over" if the master fails -- but it does it by spinning up a replacement master using the existing cluster volume as the backing store.
The above answer is no longer valid anymore.
Multi AZ = Aurora Cluster with at least one Read Replica in a
different AZ.
You can still create multiple read replicas for a cluster but if you create them within the same AZ of your writer, cluster will not be multi AZ.
Within each AWS Region, Availability Zones (AZs) represent locations that are distinct from each other to provide isolation in case of outages. We recommend that you distribute the primary instance and reader instances in your DB cluster over multiple Availability Zones to improve the availability of your DB cluster. That way, an issue that affects an entire Availability Zone doesn't cause an outage for your cluster.
You can set up a Multi-AZ cluster by making a simple choice when you create the cluster. The choice is simple whether you use the AWS Management Console, the AWS CLI, or the Amazon RDS API. You can also make an existing Aurora cluster into a Multi-AZ cluster by adding a new reader instance and specifying a different Availability Zone.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Concepts.AuroraHighAvailability.html