I want to periodically sync my Cloud SQL Postgres DB with an Amazon Aurora DB.
Setting up a Lambda that does pgdump and psql from one to the other should work but is there a cleaner solution?
You can take a look at the Database Migration Service in AWS. As long as one of the databases is in AWS (which Aurora is) you can use it to perform full-loads + continuous migration using Change Data Capture. PostgreSQL is supported, so you'd have to create a migration instance, configure the Postgres and Aurora endpoints and set up a replication task.
Reference
Creating tasks for ongoing replication using AWS DMS
I ended up running a pg_dump and psql cron job on an EC2 instance. I could not access GCP db as endpoint as from DMS for some reason.
Related
can we consider RDS MYSQL/POSTGRES read replica's as source endpoint for AWS DMS ?
I hava a requirement werein i have to use replica's of RDS MYSQL and RDS POSTGRES as source endpoint in AWS DMS and REDSHIFT cluster as target endpointfor AWS DMS(we have to use FULL load with CDC).
please suggest if it's possible to use replica's of RDS MYSQL and RDS POSTGRES.
Thanks
On Using a MySQL-compatible database as a source for AWS DMS - AWS Database Migration Service, there is a reference to read replicas:
If you are using an Amazon RDS MySQL or Amazon RDS MariaDB read replica as a source, enable backups on the read replica, and ensure the log_slave_updates parameter is set to TRUE.
Therefore, it seems to be possible with MySQL on Amazon RDS.
However, Using a PostgreSQL database as an AWS DMS source - AWS Database Migration Service says:
You can't use RDS PostgreSQL read replicas for CDC (ongoing replication).
However, it seems possible for the initial (full) load.
I created a snapshot of my RDS Postgres DB, and I am trying to use that snapshot to create an AWS Aurora DB using the migrate snapshot option in AWS console, however the option is disabled for me, and I can't understand why?
According to this official AWS blog, I should be able to migrate my RDS postgres DB into Aurora DB using the migrate snapshot option.
I created this snapshot from a Postgres DB running on version 12.5. I have created another snapshot from another Postgres DB running on version 9.6.20, and I face the same issue there as well.
12.5 and 9.6.20 versions of PostgresSQL are not supperted by aurora. What is supported is listed here.
I have an AWS Aurora MySQL database on my production environment, and a separate AWS Aurora MySQL database on my performance environment. Periodically, I'll create a copy the production database, and use the copy as my database in my Performance environment, switching out the old performance database and replacing it with the new one.
Does AWS Glue provide the ability to move data from one Aurora MySQL database to another Aurora MySQL database? Could I use it to periodically (maybe once a week) copy over data from the Prod database to the Perf database? Also, if this is possible, would I be able to selectively copy data over from the prod MySQL, without necessarily losing data that was only added on the perf MySQL?
May I suggest not to use Glue for a full copy of a database, but AWS DMS (Database Migration Service) instead.
You can do very quick 1-to-1 migrations between two databases with DMS. You spin a DMS instance (Linux server, low cost, turn it off when not in use), set up a source and a target endpoint, and a replication task, and you're good to go.
Here is a guide you can follow: https://docs.aws.amazon.com/dms/latest/sbs/chap-rdsoracle2aurora.html
I have a web application that uses two databases. I want to schedule an SQL query to be executed in set intervals to select data from the first database and put it in a table in the second one and the web application takes it's data from here.
You can do that using a scheduled Lambda function.
From Schedule jobs for Amazon RDS and Aurora PostgreSQL using Lambda and Secrets Manager
For on-premises databases and databases that are hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances, database administrators often use the cron utility to schedule jobs. For example, a job for data extraction or a job for data purging can easily be scheduled using cron. For these jobs, database credentials are typically either hard-coded or stored in a properties file. However, when you migrate to Amazon Relational Database Service (Amazon RDS) or Amazon Aurora PostgreSQL, you lose the ability to log in to the host instance to schedule cron jobs. This pattern describes how to use AWS Lambda and AWS Secrets Manager to schedule jobs for Amazon RDS and Aurora PostgreSQL databases after migration.
https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/schedule-jobs-for-amazon-rds-and-aurora-postgresql-using-lambda-and-secrets-manager.html
We need to replicate data from an Amazon RDS (MySQL) databaes to an Aurora PostgreSQL database. Each database is on a different AWS account and region.
The data must be replicated in the PostgreSQL instance every 6 hours, so we need the guarantee that the data doesn't duplicate even if a field updated for a record.
Which method, tool, or design is better to do that? (Could be different to AWS.)
You could facilitate one of the following services provided by AWS:
Database Migration Service
Glue
The AWS Database Migration Service supports:
Using an Amazon-Managed MySQL-Compatible Database as a Source for AWS DMS
Using a PostgreSQL Database as a Target for AWS Database Migration Service