How to approach schema/data migration when upgrading to Couchbase 7.x from Couchbase 5.x ?
By schema I mean indexes, that must be created before any document inserted,
so have to
create new collection,
define indexes,
move documents from older collection except for ref docs.
For SQL there is database migration tool like Flyway/Liquibase
P.S. I am not quite good with Couchbase.
Is there any other general practices to keep db/data in good shape in update in environments pipeline DEV->UAT->PREPROD->PROD ?
There's a tool called CouchVersion.
CouchVersion is a Java framework which helps you to manage changes in
your Couchbase and synchronize them with your application. The concept
is very similar to other db migration tools such as Liquibase or
Flyway but without using XML/JSON/YML files.
You don't have to use any 3rd party tool for this! Couchbase offers everything out of the box.
First some background knowledge, couchbase is designed to work in a cluster that consists of at least two nodes, ideally in production you should use at least 3 nodes. Given that, if you want to upgrade couchbase you can do that live on the cluster! You simply remove a node from the cluster, upgrade the node, then joint it back into the cluster, that's it! The cluster continues to work while you remove and upgrade it node by node. Nothing is lost, no downtime, your clients will not even feel it.
Second, you can upgrade a couchbase cluster node only from one major version to the next (+1) major version, so in your case you have to upgrade all nodes from major version 5.x to 6.x then when all nodes in the cluster have version 6.x you upgrade all nodes from 6.x to 7.x
Note that couchbase cluster can run different node versions in the same cluster (called mixed mode), only when all nodes are upgraded to the latest version only then all version features are available.
I personally upgraded couchbase in test and production from version 6.0.3 to 6.5, from 6.5 to 6.6 and from 6.6 to 7.1
There is also data migration tool
https://github.com/tchlyah/couchmove
But specifically our current top option is to use solution provided by own CI/CD team
(that similarly defines SQL++ statements to execute)
Related
I do not know why AWS DocumentDb does not support MongoDB version that above 3.6? Should I use mongo 3.6 or 4.0 above?
Amazon DocumentDB now supports MongoDB 4.0 compatibility including transactions: https://aws.amazon.com/about-aws/whats-new/2020/11/amazon-documentdb-with-mongodb-compatibility-adds-support-for-mongodb-4-and-transactions/
Document DB is compatible with only MongoDB 3.6.
See: https://aws.amazon.com/documentdb/features/
Whether you want to use 3.6 vs 4.0 or even 4.2 or 4.4 is very subjective to what you want to do with the DB.
The pros of using Document DB is that its a managed service and hence you dont have to worry a lot about setting it up.
The con is you will not get features present in version 3.6 onwards example multi document transactions, new operators in the agg pipeline, bug fixes etc.
To figure out the exact changes check - https://docs.mongodb.com/manual/release-notes/
You can install mongodb on EC2 instances, this will ensure you will get the latest and greatest mongodb. However this comes with the added work of managing the mongodb instance, its backups, High availability considerations etc.
Do note: No matter what you decide I would recommend that you try to use the latest drivers that are present today so that you have the freedom to go to the latest version of self installed mongo or even upgrade document db engine versions as they become available.
AWS DocumentDb uses its own database engine compatible with MongoDB 3.6 API.
MongoDB 4.0 API is not supported as of yet.
We have MongoDB Atlas available in AWS - AWS also supports fully managed database service - MongoDB Atlas - https://aws.amazon.com/de/quickstart/architecture/mongodb/
AWS enables to set up the infrastructure to support MongoDB deployment in a flexible, scalable, and cost-effective manner on the AWS Cloud.
https://aws.amazon.com/de/quickstart/architecture/mongodb/
We have an Openstack infrastructure consisting of one controller node, eight compute nodes and a network node. This last node is having hardware problems (disk write failures). Unfortunately it has only one disk without replication. And there's no option now to modify it for HA support.
We already tried to "dd" that disk to another but it didn't bring up. So, we agreed that the better choice was to build a new network node (using the same hardware specs)
Failing network node is running the following:
CentOS 7.1.1503
Openstack-neutron-openvswitch-2014.2.2-1 (Juno release)
Openvswitch-2.1.2-2
New network node:
CentOS 7.3.1611
Openstack-neutron-openvswitch-2014.2.3-1 (Juno release)
Openvswitch-2.3.1-2
We managed to export the database, just copying the conf.db file located in /etc/openvswitch into the new node. We had to convert the db to a newer schema since the nodes have different ovs versions. But we can't make it work like the old one, since it adds new interfaces to the database record and doesn't use the ones imported from the old hardware, even having the same exact names.
Is there a way to replicate the ovs configuration in the new node and make it work? That considering that both hardware are the same. Any of you had any experience trying to move/import/export an ovs database? I can attach the database dump if necessary.
I think you must config manually from the beginning, because that database is hashed and encrypted,
when you install openstack throught different machine, the openstack generate keystone randomly,
if you use packstack it the keystone stone can be same with you generate the answer-file
packstack --gen-answer-file=openstack.conf
and adit the option you want in openstack.conf
We tried to go directly from Flyway 3.0 to 4.0 to get access to Repeatable Migrations. (Thus skipping versions 3.1 and 3.2.)
The SQL run via MetaDataTableImpl.upgradeIfNecessary drops the indexes schema_version_vr_idx and schema_version_ir_idx which did not exist in version 3.0's schema, so Postgres is not happy.
A way to support serveral version upgrade could be to use DROP INDEX IF EXISTS schema_version_vr_idx;, but that needs to be adopted to the different databases of course.
For now I solved the problem by creating these two indexes before running Flyway to upgrade the schema.
Is there a better way to do it?
Actually those indexes did exists by default: https://github.com/flyway/flyway/blob/flyway-3.0/flyway-core/src/main/resources/org/flywaydb/core/internal/dbsupport/postgresql/createMetaDataTable.sql
No idea why or how they were removed on your DB instance.
I am trying to develop server-side using loopback with database connector.
However, I am quite confused with installing loopback on AWS.
reference for installing loopback on AWS
This website mentioned that only loopback of version 2.0 could be installed.
Yet, when I browse through loopback website, https://strongloop.com/strongblog/how-to-setup-push-notifications-private-mbaas-amazon-aws-part-1/, this website shows that it seems possible to install loopback of version higher than 2.0 on AWS. Since there are some features only available after version 2.1x, it would be nice if AWS allows installation of loopback of version higher than 2.0. Could anyone help me solve the problem? BTW, I am only using free tier of AWS and do not intend to pay at this moment.
Even if you install the image that comes preconfigured with Loopback 2, you should be able to upgrade to newer versions using npm as you normally would (sudo npm install -g strongloop and the like). Imagine if there's a security issue that you'd need that wasn't backported for whatever reason...Loopback is just files and the image is just linux. You have free reign to update/upgrade whatever you need.
My recommendation would be to start out with a minimal Ubuntu image and install everything with npm. You'll understand the ecosystem better and won't be surprised by something you don't remember installing specifically.
One caveat that a bunch of preconfigured images have is they are only available on older instance types (m1 for instance--pun not intended ;)). They are slower and more expensive than newer instances.
You can probably install whatever you want.
The difference is that if you want a machine image that contains everything already there and don't want to build it yourself you're stuck to 2.0 here (https://aws.amazon.com/marketplace/pp/B00PG9I0M0)
What I would do is use the provided AMI and after that upgrade loopback manually.
Amazon Web Services offer a number of continuous deployment and management tools such as Elastic Beanstalk, OpsWorks, Cloud Formation and Code Deploy depending on your needs. The basic idea being to facilitate code deployment and upgrade with zero downtime. They also help manage best architectural practice using AWS resources.
For simplicity lets assuming a basic architecture where you have a 2 tear structure; a collection of application servers behind a load balancer and then a persistence layer using a multi-zone RDS DB.
The actual code upgrade across a fleet of instances (app servers) is easy to understand. For a very simplistic overview the AWS service upgrades each node in turn handing connections off so the instance in question is not being used.
However, I can't understand how DB upgrades are managed. Assume that we are going from version 1.0.0 to 2.0.0 of an application and that there is a requirement to change the DB structure. Normally you would use a script or a library like Flyway to perform the upgrade. However, if there is a fleet of servers to upgrade there is a point where both 1.0.0 and 2.0.0 applications exist across the fleet each requiring a different DB structure.
I need to understand how this is actually achieved (high level) to know what the best way/time of performing the DB migration is. I guess there are a couple of ways they could be achieving this but I am struggling to see how they can do it and allow both 1.0.0 and 2.0.0 to persist data without loss.
If they migrate the DB structure with the first app node upgrade and at the same time create a cached version of the 1.0.0. Users connected to the 1.0.0 app persist using the cached version of the DB and users connected to the 2.0.0 app persist to the new migrated DB. Once all the app nodes are migrated, the cached data is merged into the DB.
It seems unlikely they can do this as the merge would be pretty complex but I can't see another way. Any pointers/help would be appreciated.
This is a common problem to encounter once your application infrastructure gets into multiple application nodes. In the olden days, you could take your application offline for "maintenance windows" during which you could:
Replace application with a "System Maintenance, back soon" page.
Perform database migrations (schema and/or data)
Deploy new application code
Put application back online
In 2015, and really for several years this approach is not acceptable. Your users expect 24/7 operation, so there must be a better way. Of course there is, the answer is a series of patterns for Database Refactorings.
The basic concept to always keep in mind is to assume you have to maintain two concurrent versions of your application, and there can be no breaking changes between these two versions. This means that you have a current application (v1.0.0) currently in production and (v2.0.0) that is scheduled to be deployed. Both these versions must work on the same schema. Once v2.0.0 is fully deployed across all application servers, you can then develop v3.0.0 that allows you to complete any final database changes.