I have a VPC which has 2 Private subnets i.e. subnet 1 and subnet 2. My redshift cluster sits in subnet 2 and has data. I want to move the redshift from subnet 2 to subnet 1 within the same VPC (Which can be done easily). But I have few doubts related to data migration:
Does data migration happens automatically without any data loss or do I need to take the backup, create the cluster in subnet 1 and then again push the backed up data to the cluster.
Any leads would be appreciated.
From Amazon Redshift Snapshots - Amazon Redshift:
Restoring a Cluster from a Snapshot
A snapshot contains data from any databases that are running on your cluster, and also information about your cluster, including the number of nodes, node type, and master user name. If you need to restore your cluster from a snapshot, Amazon Redshift uses the cluster information to create a new cluster and then restores all the databases from the snapshot data. The new cluster that Amazon Redshift creates from the snapshot will have same configuration, including the number and type of nodes, as the original cluster from which the snapshot was taken. The cluster is restored in the same region and a random, system-chosen Availability Zone, unless you specify another Availability Zone in your request.
So, you should take a Snapshot of the existing Redshift cluster, then create a new cluster in the other AZ by Restoring a Cluster from the Snapshot. Once everything seems to be okay, you can delete the old cluster.
I know this is an old question but looking at the comments in the other answer I ran into the same issue. To move from one subnet to another you do the following:
Make sure your subnet group has both subnets in it
Take a snapshot of the current cluster
Restore the snapshot to a new cluster using the same settings as the original cluster but select the other AZ that you want it to be in. Name the cluster something like "{ORIGINAL_NAME}-new"
After the new cluster is up, go to the old cluster, edit it, and append "-old" to the name (it can be anything but just to keep them straight).
After the rename is complete go back to the new cluster, edit it, and change the name to be exactly what the original name was.
After the cluster is up make sure you can connect to it correctly and then delete the original cluster (the "-old" one).
CloudFormation hooks to the cluster by the cluster name. So as long as the settings are identical to the original one then there should be no drift in the CF stack and it will be linked to the new cluster after the names are changed.
Related
Setting AWS Redshift disaster recovery plan. Ideally I would like to have the ability to restore table data to a new cluster (provisioned by IAC) in my DR account.
Sharing snapshots with the DR account I don't believe will work as restoring tables needs to take place within the cluster the snapshots were created from.
Restoring snapshots to a new provisioned cluster isn't ideal as that cluster creation takes place outside of our IAC.
I believe my only other option would be to use the COPY/ UNLOAD sql commands?
You can have your Redshift cluster automatically backed up to a second AWS Region. In case of failover to the second Region, you can restore the Redshift cluster there
If your IaC is CloudFormation, you can then bring the newly restored cluster into your IaC stack
well I have couple of questions. I have a aurora cluster with a single MySQL RDS instance which has 450GB of data. we use this cluster only when we are doing some specific testing.so I want to delete this cluster but keep its data available to me so I can make a new cluster whenever we need any testing to be done.
there are couple of ways this can be done as far as I know
take a snapshot of the cluster and restore the cluster from the
snapshot whenever required.
backup the cluster to s3 and restore the
cluster from s3 when required
which way is more faster and which one is more cost efficient?
can an entire cluster be restored from s3 if so what are the steps involved ? , I found the aws documentation bit too messy.
If we stop a aurora cluster, it again automatically restarts within 7 days , is there a way to prevent this automatic restart and keep it stopped when it is not required and start when required ?
I have currently a cluster HA (with three multiple masters, one for every AZ) deployed on AWS through kops. Kops deploys a K8S cluster with a pod for etcd-events and a pod for etcd-server on every master node. Every one of this pods uses a mounted volume.
All works well, for example when a master dies, the autoscaling group creates another master node in the same AZ, that recovers its volume and joins itself to the cluster. The problem that I have is respect to a disaster, a failure of an AZ.
What happens if an AZ should have problems? I periodically take volume EBS snapshots, but if I create a new volume from a snapshot (with the right tags to be discovered and attached to the new instance) the new instance mounts the new volumes, but after that, it isn't able to join with the old cluster. My plan was to create a lambda function that was triggered by a CloudWatch event that creates a new master instance in one of the two safe AZ with the volume mounted from a snapshot of the old EBS volume. But this plan has errors because it seems that I am ignoring something about Raft, Etcd, and their behavior. (I say that because I have errors from the other master nodes, and the new node isn't able to join itself to the cluster).
Suggestions?
How do you recover theoretically the situation of a single AZ disaster and the situation when all the master died? I have the EBS snapshots. Is it sufficient to use them?
I'm not sure how exactly you are restoring the failed node but technically the first thing that you want to recover is your etcd node because that's where all the Kubernetes state is stored.
Since your cluster is up and running you don't need to restore from scratch, you just need to remove the old node and add the new node to etcd. You can find out more on how to do it here. You don't really need to restore any old volume to this node since it will sync up with the other existing nodes.
Then after this, you can start other services as kube-apiserver, kube-controller-manager, etc.
Having said that, if you keep the same IP address and the exact same physical configs you should be able to recover without removing the etcd node and adding a new one.
We have 4 standalone, non-multiAZ aurora DB instances in a VPC and we want to move them to aurora instances in another VPC.
As I understand there are 3 ways to migrate DB instances:
1) Modify the DB instance's Subnet group to change the VPC.
However this is not supported for Aurora instances yet.
2) Create a read replica, and when slave catches up, stop the slave and take it's snapshot to create a DB instance in the different VPC and use external slave then to again resume the replication.
I have a few question around this second method. As Aurora is using some different
replication method, the result of show slave status; command is empty.
Also by default the binlog_format is OFF so not sure if I have to modify it and
then restart the instance to take note of the binlog etc.
Have anyone done this before and can guide me? I don't want to restart the instance
to later find out that it is not working, as it is a very critical DB and I want to
minimize the downtime.
3) Use Amazon DMS service, however I cannot find the source DB details for Aurora in the documentation here.
I need to find out what all permissions to give to the replication user
I'll create for this. This command in aurora is not working,
GRANT REPLICATION CLIENT, REPLICATION SLAVE ON . TO 'replication_user'#'%'
IDENTIFIED BY 'aaaaaa';
Any help would be appreciated.
You should be able to restore from a snapshot. I migrated aurora across VPCs using this approach. I hoped we could create a read-replica in a different VPC but at the moment I think you can only create one in a different region.
One strange side effect I have found and i'm not sure why this happens is that in the orginal cluster multi-az was 2 Zones and in the restored cluster multi-az is No. This hasn't affected things that I can see. I still have a cluster with writer and reader
Create a snapshot and restore it to a new cluster. When you do that, use a new subnet group created for the second VPC. That's the best way to achieve this. Like you called out, you cannot change subnet group for an existing cluster.
In your approach #2, you mention having to create a slave and then taking a snapshot. That's not required. All instances in a cluster are connected to the same shared volume, so you can just go ahead and create a snapshot from your single instance cluster directly. Just make a note that snapshots are a cluster level action, and not an instance level action in Aurora.
From https://aws.amazon.com/premiumsupport/knowledge-center/rds-vpc-aurora-cluster/
It states that
Create a clone in a different VPC
If you clone a database in an Aurora cluster, you can change the VPC of the clone. > However, the subnets in the VPC must map to the same set of Availability Zones. For more information, see Cloning Databases in an Aurora DB Cluster.
It does work though can't see a way via the console to break the replication. We are using this as a faster way of migrating than snapshot and restore. Stopping the original master would break replication but though you could do it cleaner
can someone help me with these questions please :
1- the documentation stated that Aurora will automatically fail-over to the read replicas, my question is how does it select the replica which will be promoted if you have more than one with different instances class?
2- can I disable this automatic fail-over (just asking, not stating that I will do it)
3- what is the purpose of multi-az in Aurora if you can have the same effect with much more control on instances classes while creating replicas and let Aurora do the auto fail-over for you. please correct me if I am wrong with this assumption.
thanks in advance
The algorithm for election of a new master in case of failure is not really documented... but it doesn't seem to matter, because Aurora replicas seem to be different than other RDS replicas, because all the instances in the cluster are necessarily of the same instance class.
Unlike other RDS offerings, read replicas in Aurora do not appear to have an independent copy of the backing store -- instead, the backing store itself provides redundancy, being replicated at the storage level with two copies in each of three availability zones.
The cluster volume is made up of multiple copies of the data for the DB cluster, but the data in the cluster volume is represented as a single, logical volume to the primary and Aurora Replicas in the DB cluster.
Because the cluster volume is shared among all instances in your DB cluster, no additional work is required to replicate a copy of the data for each Aurora Replica.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Replication.html
Multi-AZ in Aurora is also not the same thing -- with other RDS products, multi-AZ involves a second, invisible instance, running in parallel with the master. The Aurora literature uses the phrase "multi-AZ technology," but the meaning appears to be different. Note that the Aurora pricing tables don't show a separate pricing rate for "multi-AZ" the way MySQL and MariaDB do.
Failover doesn't appear to be something that can be disabled. Even if you have no replicas, Aurora will still "fail over" if the master fails -- but it does it by spinning up a replacement master using the existing cluster volume as the backing store.
The above answer is no longer valid anymore.
Multi AZ = Aurora Cluster with at least one Read Replica in a
different AZ.
You can still create multiple read replicas for a cluster but if you create them within the same AZ of your writer, cluster will not be multi AZ.
Within each AWS Region, Availability Zones (AZs) represent locations that are distinct from each other to provide isolation in case of outages. We recommend that you distribute the primary instance and reader instances in your DB cluster over multiple Availability Zones to improve the availability of your DB cluster. That way, an issue that affects an entire Availability Zone doesn't cause an outage for your cluster.
You can set up a Multi-AZ cluster by making a simple choice when you create the cluster. The choice is simple whether you use the AWS Management Console, the AWS CLI, or the Amazon RDS API. You can also make an existing Aurora cluster into a Multi-AZ cluster by adding a new reader instance and specifying a different Availability Zone.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Concepts.AuroraHighAvailability.html