We are using Flyway DB for a long-lived project, which currently has more than 100 migrations.
In our automated tests, we want to reset the DB before running tests, so that we start with a known state in the db AND we test migrations as well.
The problem we encountered is that running more than 100 migrations is too slow. We think running a single script that creates the current schema, instead of evolving it through all the migrations, would be quite faster. And we don't need the ability to go from version n to n+1 (for arbitrary n), as the only production db we have is already up to date.
Is there any way to "replace" the older migrations with one "snapshot" migration capable of taking a db from version 0 to version 100 in just one script? I guess we could just remove all migrations up to the last one (let's say it is number 100) and replace the migration number 100 with a migration that creates the whole db in one script. Any other ideas?
I know we could just keep the db schema in tests and just wipe the data out before running them, but this is difficult due to some of the rows (just a few) are like constants that where inserted during a migration, and just wiping everything out leaves the db in an inconsistent state.
Thanks in advance!
You can follow a process like the one describe here to condense your existing migrations: http://flywaydb.org/documentation/existing.html
Related
Overview
I have an application that uses Doctrine migrations. The application is a software code base used to manage multiple businesses who all use the same code instance in their own unique environments. i.e. Each code base is identical other than configurations.
Mistakes we made
One of the mistakes the dev team made was to include core migrations in the customer folders. i.e. With each new application we have to drop in the migration files or run a migration:diff to get going which I feel is not efficient and can lead to a mess.
What I would prefer is to have core migrations as part of the core code since it rarely changes and custom migrations on the client app side.
What I want to do
I want to move all core structure migrations to our code files
I want to have a single custom migration to drop in customised data in the client migration folder.
The problem
The problem I face is pretty much how to reorganize the migrations without breaking databases on existing client applications.
I have thought of two solutions:
Solution 1:
Add blank migrations as a placeholder for the new migrations I want.
Commit these to the repo and deploy to our environments.
They will be run, nothing will be changed, the migraitons table will store them as having been executed.
Next, Update the blank migrations to the actual code I want, and empty all other migration files. Commit this to the environments.
Finally - remove the unwanted migration files, remove the unwanted database migration records.
Solution 2
Change the migration location in the db to a new location
Remove all migration files and add blank migrations for the new ones I want
Commit this to the repo, allow to run and record the migrations as being run in the new table.
Add migration code.
Now all new applications will have the updated migration files and the old apps will have the new migration files...
Question:
Am I re-inventing the wheel? Is there a standard on how to do this as I am certain I am not the first to bump into this problem?
So for anyone who finds themselves in a similar position where they need to tidy up a mess of doctrine migrations, this should serve as a simple pattern to follow.
In our development environment we use continuous integration/git/kubernetes etc. and the following process works well with our environment.
The steps:
Update the migrations table name, this you can do in the configs quite easily.
'table_storage' => [
'table_name' => 'migration_version',
'version_column_name' => 'version_timestamp',
],
Next, delete your old migrations (delete the files) and run migrations:diff to generate a new one which will be a combination of all your changes.
Now comment out the code in the new file so that it's essentially an empty migration file.
On local, delete the old migrations table and run your build process which will add the new migration to the new table.
Commit to develop/staging/live etc. and repeat the process.
Now that the db in all your environments has the updated migrations file in it. You can now uncomments the code which will not be executed when you commit the file since it exists in your migrations table.
Hope this helps someone!
I'm using Django and Postgresql to develop a web service.
Suppose we've 3~4 branch which for the different features or old-version bugfix purpose.
Then, I met a problem, when I was in branch A and change django model, and run migrate to change database in my local test desktop.
When I switch to another branch which has no migration file, database will inconsistent and cannot work when I try to run django, I've to delete the database and recreate it.
In general, what's the best/common way to deal with this kind demands for developer environment?
I understand your situation well and have been in same shoe several times.
Here is what I prefer(/do):
I am in branch bug-fix/surname_degrade
I changed the user data model [which generated user_migration_005] and then migrated the DB.
Then my boss came and pointed out that the user is not able to login due to login degrade.
So I have to switch branch and fix that first.
I can rollback the migration[user_migration_005] which I have done few moments back. With something like this python manage.py migrate user_migration_004
Switched branch and started working on hot-fix/login_degrade
When I switch back to my previous task , I can just do migration and proceed.
With this procedure I don't need to delete my all tables or restore old database or anything like that.
I am a newbie, will be extremely happy to hear your thoughts.
The major issue here is that, you database will change everytime You migrate,so either you mantain you database consistency among different branches, or You can do One thing, while using/testing (after declaring all the models)
1) Delete all database tables ( If you have a backup or dummy data )
2) Delete all existing migration files in you branch
3) Create new migrations
4) Migrate to new migrations
The above steps can also be done if the models are re modified, after modification just repeat the steps.
Run a different test database in each branch.
When you fork the design, fork the database
Make a clone of the database and migrate that.
Make sure when you push to git, you include your migrations, that wait when someone else pulls the branch and does a migrate django knows what changes were made to the database.
I've been looking at Flyway as a database migration tool.
The one thing that I have been unable to find a definite answer for is the following:
Can I force Flyway to run all as-of-yet unapplied migrations in a single transaction, instead of having each migration be its own transaction?
In a dev environment it's not an issue, but in a production environment where you would potentially perform multiple migrations from one update to the next, one of the migrations failing would leave the database in a 'half-migrated' state, where some migrations were committed and some not - quite a bad thing.
A workaround would be to simply cram all the SQL required in a single file, but there are issues with that:
The production migrations and dev migrations would end up being performed differently, since you cannot know in advance what will be in the migration on dev environment. I guess you could always do a clean and then a new migrate, but this seems to be against the spirit of the flyway design with regard to incremental migrations.
The checksums will be different as soon as a new change is added.
Does Flyway still not support such a feature? Does Liquibase, or any other migration tool?
There is no such feature out of the box. It's a great question though and I'd bet it was thought about since Flyway provides the transactional boundaries per migration - hopefully Axel Fontaine will chime in on the technical / design considerations that resulted in this not being a feature.
The FAQ have this and this to say regarding downgrading / failures. The policy boils down to:
Maintain backwards compatibility between the DB and all versions of
the code currently deployed in production.... Have a well tested,
backup and restore strategy.
In my case, we have being using Flyway for almost 3 years and have abided by the quoted policy. At any given deployment we could have 100 or more migrations running against many databases and happy to say have never had anything untoward happen in production. This all comes down to minimizing the opportunity for failure in your release process.
I used Liquibase on a much smaller project prior to that and don't recall any such feature apart from providing the rollback procedures.
Many moons ago I used commands like ./manage.py reset appname to DROP and then recreate the database tables for a single App. This was handy for when other developers had inadvertently but manually broken something in the database and you wanted to reset things back without affecting other apps (or needing to go through a lengthy dump/load process).
The advent of Django 1.7 and its builtin migrations support seems to have removed and renamed a lot of these commands and I'm going crosseyed with all the shared prefixes in the documentation. Can somebody spell this out for me?
How do I reset the tables for a single application (one with migrations)?
If your Django migration subsystem is not broken in itself, the normal way to reset an app is to run manage.py migrate <app> zero.
This will run all of the app's migrations backwards, so a few things are noteworthy:
if some of the app's migrations are not reversible, the process will fail. Should not happen normally as Django only creates reversible migrations. You can build irreversible ones yourself, though - usually when you create data migrations.
if some other app has a dependency on this app, it will also be migrated backwards up to the last migration that did not depend on it.
You can then run migrate again, so it is run forwards.
In any case, remember migrations introduce a risk for your data, so backup your database before touching anything.
What is the best practice for removing a database table when database upgrades are handled in 'migrations' fashion?
We use Flyway for database migrations. Every time there is a database change, a migration script (which takes care of the delta change) gets added.
After a round of refactoring to remove obsolete code, couple of tables are no more needed.
Options I can think of are:
Leave those tables alone. I don't like clutter, so prefer not to go with this option.
Add migration script to delete these tables. Creating and later deleting a few tables will add to app installation time, again not preferable to us.
Edit one of the initial migration scripts, so the table doesn't get created for new installations. Problem: Flyway will complain that one of the migrations was tampered with.
Are there other options?
Don't worry so much about #2. The overhead is little when the tables are empty and it's not very often that one needs to rebuild the full DB anyway.