By design the flyway migrations run in transactions and synchronously pretty early during startup of an application. This is usually desired, to ensure before business logic starts to execute the database is in a consistent state (migrated) or the migration fails and the app crashes.
In some cases I'd really like to be able to start the application without it waiting for some migrations to be completed (long running migrations, creating indexes or materialized views, etc.). This might also be needed when deploying from a CI-Server and using deploy-timeouts / health-checks (that can not be raised indefinitely) to ensure the deployment worked as expected.
Is there any configuration / convention / best practice to enable async migrations? (i.e. naming the migration A2_00__UpdateSthLong.sql instead of V2 (standard) or R2 (Repeatable migration).
Seems like its not possible (yet):
There is an issue requesting this on github:
https://github.com/flyway/flyway/issues/950
Flyway for micronaut, seems to already support this: https://github.com/flyway/flyway/issues/950
A temp solution until support is integrated could be to use Java Migrations and spawn an async-Task from within the migrate method yourself. The migration wouldn't be transactional then of course :(
Another solution might be to do the migration before actually starting up the application (i.e. by using the maven task).
Related
So, I often am working with projects where there will be multiple dockerized django application servers behind a loadbalancer, and frequently I will need to deploy to them. I frequently use Watchtower to do pull-based deploys. I build a new image, push it to dockerhub, and watchtower is responsible for pulling those images down to the servers and replacing the running containers. That all works pretty well.
I would like to start automating the running of django migrations. One way I could accomplish this would be to simply add a run of the manage.py migrate to the entrypoint, and have every container automatically attempt a migration when the container comes online. This would work, and it would avoid the hassle of needing to come up with a way to do a lockout or leader election; but without some sort of way to prevent multiple runs, there is a risk that multiple instances of the migration could run at the same time. If I went this route, is there any chance that multiple migrations running at the same time could cause problems? Should I be looking for some other way to kick off these migrations once and only once?
Running migrations at the same time in parallel is not safe. I've tested it by running the migrate command in parallel on a data migration, and Django will run the migration twice. It will add 2 rows to the django_migrations table.
Check out this Google Groups post discussing the issue
And this article about running migrations on container statup.
We have a Clojure service that runs in a Docker Container running in Amazon ECS. When the container is deployed, the Clojure services connects to the database and always runs migrations on startup.
The problem is that if we need to rollback the code deploy, the deployed container has the old code in it, and does not have access to the rollback migrations that the latest container has.
This is a problem that doesn't happen often, but when it does, how do we perform the DB Rollback?
The best we can think of right now is to do it manually.
Anyone have experience doing this programatically?
It seems like you should consider separating your migrations from your actual deployable. Everyone will have their own preference for managing migrations but you lose flexibility when you package your migrations into your application. A dedicated migration tool can act more intelligently when on its own. For example, some database migrations are impossible to rollback without some sort of snapshot system, ex any migration that removes data. Additionally, its bad practice for your application to have the permissions necessary to perform migrations. You also cannot easily audit which user performed the migration.
I'm building an e-commerce website with Django 1.8 and PostgreSQL 9.4. I'm interested in learning what techniques I can use when I change my database to avoid having problems with Django migrations, particularly in the event that I can't get migrations to run and I have to delete my migrations, rebuild my database, and restore from backups.
In development, I've found that when I change my database schema and re-run migrations, they only run successfully about 50% of the time. Clearly I'm doing some things wrong. What's worse is that when migrations don't work, it's not always easy to understand exactly why they failed and how to modify my migration files so that they will run. In those situations, I always have to delete the database and start over. This is acceptable in development but it's not a good strategy when I go into production.
What are some "best practices" or "do's and don'ts" you follow when you modify your model classes/database schema so as to increase the probability that your Django migrations will run? And are there any steps you take to ensure that you can restore your database in the event that your migrations won't run and you have to rebuild the database from scratch? I should add that I'm a one-person startup so I don't have the conflict issues that a team working from the same code base would have.
These Techniques are what I'm using
Work locally in the same environment what I'm working in it on server. Same version of the Django and database server then push the migrations itself, don't ignore it, and migrate on the server using there migrations.
This one I used once that I migrate manually, I created the tables, indices, relations using sql commands manually and it worked properly too.
I prefer the first one more
I've been looking at Flyway as a database migration tool.
The one thing that I have been unable to find a definite answer for is the following:
Can I force Flyway to run all as-of-yet unapplied migrations in a single transaction, instead of having each migration be its own transaction?
In a dev environment it's not an issue, but in a production environment where you would potentially perform multiple migrations from one update to the next, one of the migrations failing would leave the database in a 'half-migrated' state, where some migrations were committed and some not - quite a bad thing.
A workaround would be to simply cram all the SQL required in a single file, but there are issues with that:
The production migrations and dev migrations would end up being performed differently, since you cannot know in advance what will be in the migration on dev environment. I guess you could always do a clean and then a new migrate, but this seems to be against the spirit of the flyway design with regard to incremental migrations.
The checksums will be different as soon as a new change is added.
Does Flyway still not support such a feature? Does Liquibase, or any other migration tool?
There is no such feature out of the box. It's a great question though and I'd bet it was thought about since Flyway provides the transactional boundaries per migration - hopefully Axel Fontaine will chime in on the technical / design considerations that resulted in this not being a feature.
The FAQ have this and this to say regarding downgrading / failures. The policy boils down to:
Maintain backwards compatibility between the DB and all versions of
the code currently deployed in production.... Have a well tested,
backup and restore strategy.
In my case, we have being using Flyway for almost 3 years and have abided by the quoted policy. At any given deployment we could have 100 or more migrations running against many databases and happy to say have never had anything untoward happen in production. This all comes down to minimizing the opportunity for failure in your release process.
I used Liquibase on a much smaller project prior to that and don't recall any such feature apart from providing the rollback procedures.
During development, I like the idea of frameworks like Entity Framework 4.3 Migrations (although I need it to work with sql scripts instead of Migration classes) keeping all developer databases up-to-date. Someone does an update to get the latest source, they try to run the application and get an error that they need to update their database to the latest migration (or have the migration happen automatically). Migration files are timestamped so devs don't have to worry about naming two files the same or the sequence the files need to be executed in.
When I get ready to build a WebDeploy deployment package, I want the package to contain the scripts required to move the production database to the latest db version. So somehow, MSBuild or WebDeploy need to make a decision about which scripts must be packaged. For deployment I don't want the application to try to update itself like what EF offers. I want to either hand the package to the IT department or do an auto-deploy through a deployment server.
Some of my questions are:
Can EF 4.3 work with sql scripts instead of DBMigration classes for my development needs (I'm already using it for my ORM so it would be nice if it could)?
Does MSBuild or WebDeploy understand the concept of database migrations (e.g. does it recognize the EF. 4.3 migrationHistory table) or do I have to make sure to give it just the scripts it needs to run that will bring my prod db to the latest migration? Manually deciding which scripts should be pakaged is not something I want to do so is there a MS WebDeploy extension that understands migrations?
Are my concerns and ideas valid? I'm just researching this stuff so I don't really know.
I think that your concerns are valid. During development anything that keeps the developer machines in sync will do. When deploying, more control is needed.
In my project we've decided to only use code-based migrations, to ensure that all databases are migrated in the same order with the same steps. Automatic migration and db creation is disabled by a custom initialization strategy that only checks that the db exists and is valid.
I've written more about the details in Prevent EF Migrations from Creating or Changing the Database. I've also looked a bit into merge conflicts which will eventually happen with multiple developers.
You can run SQL in the classes by calling the Sql method, or generate SQL from a migration by using the -script parameter with the update-database command.
No. They were looking at adding support for webdeploy but apparently decided against it before rtm. They did however release a command line app (and powershell script obviously) which you could call from either.
What I do in my projects is create an startup module in my applications and run any migrations that haven't been deployed automatically - Triggering EF migration at application startup by code. It's not perfect, developers have to run the app after getting latest before changing the db, but it works for both get latest and deployment.