Due to high i/o cost we want to migrate from aurora(Postgres engine) to Postgres. Is there any way to do this.
If no is rebuilding the entire dataset is the only option?
There are a couple of options you can look into for this migration:
Use AWS Database Migration Service to use Aurora PostgreSQL as source, and Postgres(RDS or any other) as target. Refer to the documentation to check on version support and configurations.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.RDSPostgreSQL
Test setting up logical replication from Aurora PostgreSQL to move data changes.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Replication.Logical.html
If the above options are unsuccessful, you can try to re-build with a native dump and restore which could be a more time taking process.
Related
I have an AWS Aurora MySQL database on my production environment, and a separate AWS Aurora MySQL database on my performance environment. Periodically, I'll create a copy the production database, and use the copy as my database in my Performance environment, switching out the old performance database and replacing it with the new one.
Does AWS Glue provide the ability to move data from one Aurora MySQL database to another Aurora MySQL database? Could I use it to periodically (maybe once a week) copy over data from the Prod database to the Perf database? Also, if this is possible, would I be able to selectively copy data over from the prod MySQL, without necessarily losing data that was only added on the perf MySQL?
May I suggest not to use Glue for a full copy of a database, but AWS DMS (Database Migration Service) instead.
You can do very quick 1-to-1 migrations between two databases with DMS. You spin a DMS instance (Linux server, low cost, turn it off when not in use), set up a source and a target endpoint, and a replication task, and you're good to go.
Here is a guide you can follow: https://docs.aws.amazon.com/dms/latest/sbs/chap-rdsoracle2aurora.html
Can some one please advise the steps required for migrating a web application which is currently running on tomcat server at onpremise to AWS ec2 instance. I understand this is not a straight forward and requires some detailed process.
The code is wrriten in Java and database used as oracle.
So it would be helpfull if someone can suggest me any relavent document or any website which gives some demo to refer me and proceed with this scenario.
If it's a personal project then I would recommend Lightsail as the simplest way to deploy existing Java application.
For a database a small instance of MySQL or if relational database is not needed then a document database like DynamoDB. https://aws.amazon.com/products/databases/?nc2=h_m1
There are multiple choices one how to migrate a Java application to AWS.
You could potentially use existing AWS services like:
Lightsail - https://aws.amazon.com/lightsail/
Beanstock - https://aws.amazon.com/elasticbeanstalk/
or
EC2 instance and install Tomcat manually
Use ECS with Docker https://aws.amazon.com/getting-started/tutorials/deploy-docker-containers/?nc2=type_a
As for Database solution Oracle is an option but quite expensive one.
When moving to AWS it's better to use one of the RDS managed databases like MySQL, Postgress or more expensive like Aurora.
In order to propose an architecture some details would be needed on predicted load, the size of the application and volume of data. Is the product regional or global, are there any additional issues that need to be addressed while moving to a cloud (performance, availability etc), how users are authenticated (are any other services needed).
I'm facing the following situation:
I have to do a logical replication of my Cloud SQL PostgreSQL instance to an external PostgreSQL database and vice-versa. More specifically, the CloudSQL and the external instance will contain some master and slave tables that must be replicated between each database. However, PostgreSQL logical and external replication are currently not supported by the GCP.
Since it is not supported, I've searched for an alternative, and I noticed that Debezium + Kafka can be used with CloudSQL MySQL to capture database changes and do logical replication. I asked the same question on the Debezium google group (link below)
https://groups.google.com/forum/#!topic/debezium/yS61un46x8k
And they've answered to me:
"Debezium requires a specific logical decoding plug-in (either ProtoBufs or wal2json) installed within the source Postgres, I'm not sure whether you have that flexibility with your cloud SQL provider (but e.g. on Amazon RDS, wal2json is installed by default)."
Can anybody answer me if one of these plugins are available or can be installed within Cloud SQL PostgreSQL? If not, are there any alternative approach for logical replication while it is not supported by the platform? Thanks in advance.
Cloud SQL Postgres does not support logical replication right now. You also have no way to use Debezium + Kafka because of missing logical replication.
I have to migrate from Cloud SQL Postgres to on-premises Postgres.
Took them a long time, but is supported now
https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication
I have a local MySQL database and I want to migrate the data inside of it to a remote MySQL database (using RDS on AWS). How can I migrate my data between the two instances?
AWS DMS helps you migrate large, terabyte-scale databases to the AWS
Cloud easily and securely. During migration, the source database
remains fully operational, minimizing downtime.
But judging form your question you want homogeneous data migration and as per AWS Documentation:
If you're performing a homogeneous migration, use your engine’s native
tools, such as MySQL dump or MySQL replication.
Refer to this answer for using SQL Dump on larger data.
Thanks
Use AWS database migration service that is available in Aws. You need to provide your database end-point I,e your on premises data- base server end-point in it and also set your db engine parameters to your requirement and launch. it talks 10-15 minutes to migrate your data to cloud and from there you can continue accessing your database from the AWS it self.
The other method is, take a recent back up of your on premises database. Launch an instance in aws EC2 and install the db that you are using on premises.copy the back file in your system to cloud.using the backup file available launch the database.set up an RDS instance of the type that you have installed in EC2, and connect the end points.
We are using AWS RDS Aurora MySQL 5.6 for our production database.
AWS launched MySQL 5.7 compatible Aurora engine on 6th Feb, 2018.
I dont see any option in "modify instance" to change engine to MySQL 5.7
I dont see any option in restore snapshot to database with MySQL 5.7 either.
We want to do this upgrade with least downtime. Pls suggest what could be done here.
According to this link, you cannot upgrade an in-place database, you will need to restore a snapshot of the existing database and change the engine version during that process. These restrictions appear to be only temporary and may be lifted at a later point to allow for in-place upgra
The comments above are true; there is still no in place upgrade for 5.6 to 5.7; the process is still pretty easy though;
1) Go to the RDS dashboard, in the left hand menu there is a menu item called 'Snapshots'; you can either click on this if you are ok using a recent snapshot; otherwise select your database & on the actions drop down & choose 'Take Snapshot'
2) In snapshots simply select your snapshot & choose 'Restore Snapshot' from the actions drop down; it will automatically duplicate a bunch of your previous settings. Its at this juncture you can select the new database engine of 5.7
All and all you should allow for at least a half an hour of downtime for the whole process. Probably a couple of hours to be on the safe side.
You can now perform in-place upgrade from Aurora MyQSL from 5.6 to 5.7
Only a matter of invoking modify-db-cluster or modify-global-cluster (if you are using global clusters of course).
More in the docs (including how to do this using the AWS console).
Easiest way is:
Take a manual snapshot first of Aurora MySQL 5.6 cluster.
Then, create a new Aurora MySQL 5.7 using that manual snapshot which was taken in step 1
Your credentials would be same as that of the older 5.6 cluster.
Verify if data is correct.
Need was to upgrade the AWS RDS Aurora MySQL from 5.6 to 5.7 without causing any downtime to our production. Being a SaaS solution, we could not afford any downtime.
Background
We have distributed architecture based on micro services running in AWS Fargate and AWS Lambda. For data persistency AWS RDS Aurora MySQL is used. While there are other services being used, those are not of interest in this use case.
Approach
After a good deliberation on in place upgrade by declaring a downtime and maintenance window, we realized that having zero downtime upgrade is the need. As without which we would have created a processing backlog for us.
High level approach was:
Create an AWS RDS Cluster with the required version and copy the data from the existing RDS Cluster to this new Cluster
Setup AWS DMS(Data Migration Service) between these two clusters
Once the replication is done and is ongoing then switch the application to point to the new DB. In our case, the micro-services running in AWS Fargate has to upgraded with the new end point and it took care of draining the old and using the new.
For Complete post please check out
https://bharatnainani1997.medium.com/aws-rds-major-version-upgrade-with-zero-downtime-5-6-to-5-7-b0aff1ea1f4
To manage an update for a DB instance or DB cluster
Sign in to the AWS Management Console and open the Amazon RDS console at https://console.aws.amazon.com/rds/.
In the navigation pane, choose Instances to manage updates for a DB instance, or Clusters to manage updates for an Aurora DB cluster.
Select the checkbox for the DB instance or DB cluster that has a required update.
Choose Instance actions for a DB instance, or Actions for a DB cluster, and then choose one of the following:
Upgrade now
Upgrade at next window
Note: If you choose Upgrade at next window and later want to delay the update, you can select Defer upgrade.