I am looking to migrate some databases (in Terabytes) from oracle, MS SQL and Sybase to PostgreSQL
I came across - SCT as one of the tools which might be useful
I have a few questions regarding the viability and ability of SCT for the task:
Questions:
Is AWS SCT able to migrate databases (Data and Schema both) from Oracle, MS SQL and Sybase to PostgreSQL ?
Is AWs SCT able to migrate Triggers, Queries, Views and Stored Procedures ?
Does this migration happen on-premises or I will have upload/transfer to upload my data to AWS cloud compute or S3 instances, migrate/convert and download back ?
What is the largest database size that can be migrated using AWS SCT (Giga/Terabytes/Records) ?
How fast is the database migrated (Data per Time) ?
What is the accuracy of the migration (Percent) ?
how do I use the AWS SCT tool ? ( example: Is it available for public download? Are there any Paid licenses? Does it come bundled with some other service which we have to buy? )
I would also appreciate if there is any extra information about AWS services which might help my project.
The AWS Schema Conversion Tool only converts the schema. It does not transfer the data.
From What Is the AWS Schema Conversion Tool?:
You can use the AWS Schema Conversion Tool (AWS SCT) to convert your existing database schema from one database engine to another. You can convert relational OLTP schema, or data warehouse schema. Your converted schema is suitable for an Amazon Relational Database Service (Amazon RDS) MySQL, MariaDB, Oracle, SQL Server, PostgreSQL DB, an Amazon Aurora DB cluster, or an Amazon Redshift cluster. The converted schema can also be used with a database on an Amazon EC2 instance or stored as data on an Amazon S3 bucket.
To transfer the data, you could use the AWS Database Migration Service (AWS DMS).
From What is AWS Database Migration Service?:
AWS Database Migration Service (AWS DMS) is a cloud service that makes it easy to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the AWS Cloud, between on-premises instances (through an AWS Cloud setup), or between combinations of cloud and on-premises setups.
The Schema Conversion Tool can convert triggers, but please consult the documentation for capabilities/limits.
The Schema Conversion Tool is free. See: Installing, verifying, and updating the AWS SCT
The Database Migration Service involves a server, so it is has associated costs. See: AWS Database Migration Service Pricing - Amazon Web Services
Related
I need to transfer data from DWH(AWS RDS MS SQL Server) to Amazon S3. Data in DWH can be updated, not only added. Did someone do such a pipeline? Data in DWH is updated every 10-15 minutes.
AWS Database Migration Service can help you with that, it also supports ongoing replication and SQL Serve is fully suported
there's a full walktrought solution here: https://dms-immersionday.workshop.aws/en/sqlserver-s3.html
they are using SQL Server on EC2, but you can change the source to RDS
I am trying to migrate data from GCP cloud sql server to AWS aurora MySQL using DMS CDC, for this I need to enable CDC on DMS source database which cloud SQL. As per the AWS documentation I need to enable CDC by executing "sp_cdc_enable_db" SP and for this I need sysadmin access but google cloud doesn't support sysadmin access. So, in this scenario how to enable CDC?
As you know, Cloud SQL doesn't support sysadmin access, and CDC feature too.
So you have to use different method to the migration process.
If you really want to use CDC, I recommend to use middle-man replication between GCP cloud sql and AWS aurora mysql.
Just replicate your cloud sql to on-premise or somewhere else where you can execute cdc.
And then migrate the SQL server replication to AWS aurora using AWS DMS, But aurora will not be synced with source DB in cloud sql.
Or if you just want to both DBs are in sync, Have you tried steps AWS document described in here?
I think "Migrating existing data and replicating ongoing changes" section worked exactly you want.
I have a web application that uses two databases. I want to schedule an SQL query to be executed in set intervals to select data from the first database and put it in a table in the second one and the web application takes it's data from here.
You can do that using a scheduled Lambda function.
From Schedule jobs for Amazon RDS and Aurora PostgreSQL using Lambda and Secrets Manager
For on-premises databases and databases that are hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances, database administrators often use the cron utility to schedule jobs. For example, a job for data extraction or a job for data purging can easily be scheduled using cron. For these jobs, database credentials are typically either hard-coded or stored in a properties file. However, when you migrate to Amazon Relational Database Service (Amazon RDS) or Amazon Aurora PostgreSQL, you lose the ability to log in to the host instance to schedule cron jobs. This pattern describes how to use AWS Lambda and AWS Secrets Manager to schedule jobs for Amazon RDS and Aurora PostgreSQL databases after migration.
https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/schedule-jobs-for-amazon-rds-and-aurora-postgresql-using-lambda-and-secrets-manager.html
We need to replicate data from an Amazon RDS (MySQL) databaes to an Aurora PostgreSQL database. Each database is on a different AWS account and region.
The data must be replicated in the PostgreSQL instance every 6 hours, so we need the guarantee that the data doesn't duplicate even if a field updated for a record.
Which method, tool, or design is better to do that? (Could be different to AWS.)
You could facilitate one of the following services provided by AWS:
Database Migration Service
Glue
The AWS Database Migration Service supports:
Using an Amazon-Managed MySQL-Compatible Database as a Source for AWS DMS
Using a PostgreSQL Database as a Target for AWS Database Migration Service
We are currently using AWS RDS as our databases. In tables, we defined some insert or update triggers on tables. I would like to know if Bigquery also support triggers?
thanks
BigQuery is a data warehouse product, similar to AWS Redshift and AWS Athena and there is no trigger support.
If you used AWS RDS so far, you need to check Google CloudSQL.
Google Cloud SQL is an easy-to-use service that delivers fully managed
SQL databases in the cloud. Google Cloud SQL provides either MySQL or
PostgreSQL databases.
If you have a heavy load, then check out Google Cloud Spanner it's even better for full scalable relational db.
Cloud Spanner is the only enterprise-grade, globally-distributed, and
strongly consistent database service built for the cloud specifically
to combine the benefits of relational database structure with
non-relational horizontal scale.
Big Query doesn't have the feature as stated by the colleague above.
However it has an event api based on it's audit logs. You can inspect it and trigger events with cloud functions as per:
https://cloud.google.com/blog/topics/developers-practitioners/how-trigger-cloud-run-actions-bigquery-events
Regards