I have created postgresql db with replica on GCP using tier "db-custom-2-7680". Primary instance and replica reflecting same configuration. but, when i try to update the tier from "db-custom-2-7680" to "db-custom-8-16384" facing an error message below-
Failed to update instance steeting, Invalid request- postgresql master can not be larger than replica...
can anyone help on this issue?
Related
I'm trying to create an Aurora read replica from a RDS MySQL database, but I'm getting the following error:
You must enable backups on the master DB instance in order to set up DB instance to DB cluster replication...
I double checked the source instance and I already have backups enabled.
What else could be the problem?
It turns out that the problem was that my Backup retention period was set to 0 days.
I am having an AWS RDS Aurora PostgreSQL cluster with four instances with a Multi-AZ deployment serving in Production. Encryption-at-rest hasn't been enabled on this cluster. Now I have to enable the encryption on this existing cluster. AWS docs suggest me to create a snapshot of that cluster and then restore the cluster again with the encryption enabled this time. Ref: Here
Since my cluster is serving in production and no downtime or I/O suspension is acceptable to me. Here are some questions that I would like to get answered before I plan about encrypting the existing cluster:
Is there any downtime during the creation of the snapshot assuming there is a lot of data and a snapshot will take time.
What about the new data that is being written on to the database during the snapshot creation? Is the snapshot creation real-time or I will lose my new data during the time till the snapshot is being taken?
Is this the only way for me to enable encryption on the production cluster knowing that it will result in some database outage?
There is a way to encrypt your AWS RDS Amazon Aurora with PostgreSQL compatibility Cluster with no or minimum downtime, but it will take a bit of effort.
You need to take the following steps:
For the source DB, you have to take snapshot.
Then copy that snapshot, and check Enable Encryption and select Default Encryption Key or select your Custom AWS KMS CMK, now you have an encrypted copy of your DB snapshot.
Restore this encrypted snapshot to the new DB instance, and you can enable Multi-AZ and add Read Replicas now or modify them after migration.
Now you have two DB instances Encrypted and Unencrypted, but the data mismatched as it is a production database.
We will use AWS DMS to make synchronous replication of data, or ou can use PostgreSQL logical replication with Aurora instead of AWS DMS, it will be better, both will works.
Go to AWS DMS console, create an AWS DMS task.
For migration type, choose Migrate existing data and replicate ongoing changes.
For target table preparation mode, choose Truncate.
Under Advanced Task Settings, enable the awsdms_status table if you want to verify replication status.
Run the migration task and wait until all the records are updated. AWS DMS will then determine the size of the data to migrate.
Then, you need to verify the data in the Encrypted DB instance after migration is the same as the Unencrypted DB instance.
Check replication status in AWS DMS, by checking the migration task and awsdms_status.
You can now route traffic to the new endpoint.
For a smooth cutover, use Amazon Route 53 to route traffic by changing the DNS TTL to a short value, and eventually replacing the endpoint names in Route 53.
Now replying to your questions,
Is there any downtime during the creation of the snapshot assuming there is a lot of data and a snapshot will take time.
According to you cluster setup, you are running a Multi-AZ deployment, automated backups and DB Snapshots are simply taken from the standby to avoid I/O suspension on the primary. Please note that you may experience increased I/O latency (typically lasting a few minutes) during backups for both Single-AZ and Multi-AZ deployments.
What about the new data that is being written on to the database during the snapshot creation? Is the snapshot creation real-time or I
will lose my new data during the time till the snapshot is being
taken?
You will lose your data written after the snapshot has been taken, so you will use AWS DMS to replicate synchronous data to your encrypted DB instances.
Is this the only way for me to enable encryption on the production cluster knowing that it will result in some database outage?
Yes this is the only way, but it will result in no or little downtime.
I am using AWS ELK(amazon managed elastic) and my Kibana health status is red, trying to browse to the kibana URL i get "Kibana server is not ready yet".
I have tried to fix the problem but without luck, I think it all start when I changed my ELK settings from 1 availability zones with 1 instances to 2 Availability zones or another option is that I have streamed large amount of data in the last day.
As part of trials to fix the problem I returned to 1 availability zones with 1 instances but that didn't fix the problem.
Also I have Enabled the error logs and seen that I receive in cloudwatch:
"publishing cluster state with version [68816] failed for the
following nodes"
"failed to connect to node"
Any help solving this problem will help.
More info(about my current setup):
Domain status:Active
Elasticsearch version: 6.7
Availability zones:1
Instance type:r5.large.elasticsearch
Number of instances:1
Storage type:EBS
EBS volume type:General Purpose (SSD)
EBS volume size:1000 GB
Encryption at rest:Disabled
Node-to-node encryption:Disabled
Amazon Cognito for authentication:Disabled
Service software release:R20190724-P1
in the cluster health tab of the domain I can see:
Cluster status:green
MasterReachableFromNode:green
AutomatedSnapshotFailure:green
KibanaHealthyNodes:red
and in the InvalidHostHeaderRequests I have about 60% of requests that are InvalidHostHeaderRequests out of ElasticsearchRequests (but I guess that is unrelated):
CPUUtilization: is about 8%
JVMMemoryPressure: is about 20%
SysMemoryUtilization:98%
KibanaHealthyNodes is red possibly your kibana is down. Have you updated to AWS Elasticsearch v6.7 recently? Looks like the kibana needs to be restarted on the elasticsearch cluster for which AWS support team can help you with. Or in case you dont have support plan might be if you post on AWS forum someone from AWS can take a look and assist you with the same.
InvalidHostHeaderRequests will not cause the issue with kibana. AWS ES will throw this error when your application is trying to send the request on IPs of the nodes. Please check and use the domain endpoint in the request else this error will come up.
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html
During CloudFormation stack-creation I get a CREATE_FAILED error for Aurora with reason DB Instance quota exceeded.
I assume it means there are too many DB instances running (I'm on a company account with a federated login, and not the only one creating stacks and launching instances).
How do I fix it? Do need to delete some running instances? Also, how do I find out what the allowed quota is?
Limits in Amazon RDS shows a limit of 40 Amazon RDS DB Instances per Region.
You can contact AWS Customer Service to request a limit increase. Here's the Support screen:
I am trying Amazon Aurora instance and I can not see an option to stop it. The only options are Delete and Reboot.
Am I missing something.
Edit: 2018/09/25 - Amazon Aurora Now Supports Stopping and Starting of Database Clusters
Per this announcement, Aurora now supports starting and stopping the db instance. This feature was released for other RDS configurations last year and generally behaves the same in this implementation. Stopping the database cluster stops the primary instance and any Aurora replicas. The database will remain stopped for 7 days, after which it will be automatically restarted.
Additionally, see:
AWS Documentation - Stopping and Starting a DB Instance
AWS CLI Documentation - aws rds start-db-instance
AWS CLI Documentation - aws rds stop-db-instance
Edit: 2018/08/13 - Serverless Aurora supports infrequent access workloads
Per this announcement, Serverless Aurora has been released and has a feature to pause compute capacity after N consecutive minutes of inactivity. Here's an example of this configuration from the announcement blog:
This value is configurable up to 1440 minutes (24 hours), and would be comparable to "stopping" an Aurora Instance because you are not paying for compute capacity while compute capacity is paused, only database storage.
If your use case includes infrequent access (example: a dev instance that is not used after-hours), then this is currently the easiest option available.
Edit: 2017/06/01 - RDS can now be stopped/started, per this announcement, but does not support Aurora yet.
In the AWS console, a 'Stop' option has been mentioned to the same menu described in the question body. A stopped instance retains its storage but cannot be modified until it is started again. Instances cannot be permanently stopped -- after seven days, a stopped instance will be automatically started again.
Unfortunately, some configurations still don't support this feature. See below the break for an alternative.
Additionally, see:
AWS Documentation - Stopping and Starting a DB Instance
AWS CLI Documentation - aws rds start-db-instance
AWS CLI Documentation - aws rds stop-db-instance
Alternative suggestion for incompatible configurations:
This includes:
Aurora via old SDKs prior to September 2018
Old SDKs prior to June 2017
RDS with Multi-AZ
RDS with read replica
RDS with SQL Server Mirroring
RDS instances in these situations cannot be stopped and started at-will like an EC2 instance. You can, however, approximate this behavior by deleting the RDS instance with a final snapshot, and then returning later to restore an RDS instance from that DB snapshot.
This is approximate to "stopping" because no new data will be written once your final snapshot is taken, and you also won't be paying for a running RDS instance. In the same way that you would pay for any EBS volumes attached to your stopped EC2 instance, you will likewise continue paying for the storage of the DB snapshot while it exists.
This behavior diverges from stopping an EC2 instance inthat the underlying infrastructure will no longer exist. When you restore from snapshot, that will be on a new RDS instance and that restoration will take some time to complete, depending on the size of both your infrastructure and snapshot.
Further Reading
AWS RDS Documentation - Delete Instance with Final Snapshot
AWS RDS Documentation - Restoring From a DB Snapshot
They did add support for stopping/starting RDS instances this month (June 2017). The key bit of info from #Venkata's link is: "The stop/start feature is available for database instances running in a Single-AZ deployment which are not part of a Read Replica (both source and replica) configuration."
This means it does not work for Aurora because it is Multi-AZ by default. You do not even receive the option in the interface.
It also will not work for MySQL if a read replica is configured. You will not get the option on the slave and you get the option on the master but it fails if you try to use it.
You can achieve the same thing with a restore from a snapshot... but the down side is that takes roughly an hour with our large db instance whereas stopping it and starting it can happen in ~5 minutes.
They just added support for this today:
https://aws.amazon.com/about-aws/whats-new/2017/06/amazon-rds-supports-stopping-and-starting-of-database-instances/
Enjoy! :)