I'm facing the following situation:
I have to do a logical replication of my Cloud SQL PostgreSQL instance to an external PostgreSQL database and vice-versa. More specifically, the CloudSQL and the external instance will contain some master and slave tables that must be replicated between each database. However, PostgreSQL logical and external replication are currently not supported by the GCP.
Since it is not supported, I've searched for an alternative, and I noticed that Debezium + Kafka can be used with CloudSQL MySQL to capture database changes and do logical replication. I asked the same question on the Debezium google group (link below)
https://groups.google.com/forum/#!topic/debezium/yS61un46x8k
And they've answered to me:
"Debezium requires a specific logical decoding plug-in (either ProtoBufs or wal2json) installed within the source Postgres, I'm not sure whether you have that flexibility with your cloud SQL provider (but e.g. on Amazon RDS, wal2json is installed by default)."
Can anybody answer me if one of these plugins are available or can be installed within Cloud SQL PostgreSQL? If not, are there any alternative approach for logical replication while it is not supported by the platform? Thanks in advance.
Cloud SQL Postgres does not support logical replication right now. You also have no way to use Debezium + Kafka because of missing logical replication.
I have to migrate from Cloud SQL Postgres to on-premises Postgres.
Took them a long time, but is supported now
https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication
Related
Due to high i/o cost we want to migrate from aurora(Postgres engine) to Postgres. Is there any way to do this.
If no is rebuilding the entire dataset is the only option?
There are a couple of options you can look into for this migration:
Use AWS Database Migration Service to use Aurora PostgreSQL as source, and Postgres(RDS or any other) as target. Refer to the documentation to check on version support and configurations.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html#CHAP_Source.PostgreSQL.RDSPostgreSQL
Test setting up logical replication from Aurora PostgreSQL to move data changes.
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraPostgreSQL.Replication.Logical.html
If the above options are unsuccessful, you can try to re-build with a native dump and restore which could be a more time taking process.
I deploy my app to AWS.
On AWS there are RDS which support some industrial standard DBMS like PostgreSQL/MySQL/Oracle.
These dbms can be make available on development machine (docker) as well, make it easy to achieve dev/prod parity.
I'm looking for a time series specialized database that I can achieve dev/prod as well.
AWS has Timestream that is specialized for time series, but I'm clueless of a local equivalent database for it.
There probably some EC2-hosted database possible, but I prefer to be lazy and have Amazon take care of manage the database cluster for me.
What options do I have?
Apache Druid is a very good Time-Series Database that can be deployed on local development environments and on multiple cloud environments easily.
Druid is ofered as a fully managed cloud service, on AWS, by Imply.
The fully managed variant of Druid is called Imply Cloud.
More information: https://imply.io/product/imply-cloud
You should try Amazon Timestream
It is as a nonrelational, fully managed service built specifically to collect, store and process time-series data. The arrival of masses of IoT data is expected to push time-series technology into wider use,that's why Amazon came with Timestream.
Can some one please advise the steps required for migrating a web application which is currently running on tomcat server at onpremise to AWS ec2 instance. I understand this is not a straight forward and requires some detailed process.
The code is wrriten in Java and database used as oracle.
So it would be helpfull if someone can suggest me any relavent document or any website which gives some demo to refer me and proceed with this scenario.
If it's a personal project then I would recommend Lightsail as the simplest way to deploy existing Java application.
For a database a small instance of MySQL or if relational database is not needed then a document database like DynamoDB. https://aws.amazon.com/products/databases/?nc2=h_m1
There are multiple choices one how to migrate a Java application to AWS.
You could potentially use existing AWS services like:
Lightsail - https://aws.amazon.com/lightsail/
Beanstock - https://aws.amazon.com/elasticbeanstalk/
or
EC2 instance and install Tomcat manually
Use ECS with Docker https://aws.amazon.com/getting-started/tutorials/deploy-docker-containers/?nc2=type_a
As for Database solution Oracle is an option but quite expensive one.
When moving to AWS it's better to use one of the RDS managed databases like MySQL, Postgress or more expensive like Aurora.
In order to propose an architecture some details would be needed on predicted load, the size of the application and volume of data. Is the product regional or global, are there any additional issues that need to be addressed while moving to a cloud (performance, availability etc), how users are authenticated (are any other services needed).
just wondering if this is possible? I see some older links stating this is not possible from 2015 or so but wondering if it is now and if there is any available documentation that states yay/nay. Thanks!
It is not possible to queries across or JOIN logical databases created via the CREATE DATABASE command.
This is the same for PostgreSQL, on which Amazon Redshift was based. See: Joining Results from Two Separate Databases
While PostgreSQL has the dlink module that can join separate databases (on the same cluster or otherwise), this capability is not available for Amazon Redshift.
I have not used AWS yet and want to get to kinda assessment about its data center options.
My guess is that you can have a MySQL or SQL Server on an EC2 virtual box.
If DBS is Amazon data specific service, what are the differences between these two approaches:
Using DBS vs DB on EC2?
Check out this related question first.
According to Amazon's documentation:
Connect to your DB Instance using your favorite database tool or programming language. Since you have direct access to a native MySQL, Oracle or SQL Server database engine, most tools designed for these engines should work unmodified with Amazon RDS.
Basically, going the EC2 route requires a lot more maintenance in the long run. If you're using Amazon's services, and you need any one of those kind of databases, you might as well use RDS.