There are certain migrations that insert a lot of data into our production database. They slow down the integration tests and are completely not needed during the testing phase. I'd rather not solve this by boilerplate conditionals in SQL.
Is there a way to apply profiles to Flyway scripts and omit chosen migrations when necessary?
You can split them out in a separate directory and configure flyway.locations to selectively include it.
Related
I'm using a 3rd party dependency modeltranslation-lokalise which requires me to run python manage.py makemigration. This will generate a migration file that sits within the package directory once generated, and therefore are not source controlled.
For all other migration files I'm keeping them in source control and would like to do the same for these, as it will allow us to track changes to the database in the environments we use, ensuring they can be easily replicated and that operations are not duplicated on the same database (e.g. by having the same operation exist in differently-named migrations).
My question: How are these 3rd party migrations meant to be handled when building and deploying to production (e.g. via a Dockerfile in a build pipeline?
My guess would be to find a way to reliably extract the migration files from the package and add it to source control.
The alternative would be: run makemigrations within the Dockerfile that builds the deployed image, however that could result in the same migration being run twice on the database should the name of that migration change (e.g. if there's a timestamp in the name). So I'm thinking best to avoid that.
Note: this question is similar but it considers the use of the third part optional and doesn't seem to consider the above alternatives.
I solve this using the answer of this question:
Django migration file in an other app?
By using the MIGRATIONS_MODULE settings, I was able to point django to create the migrations in my source directory, and thereby commit them to source control.
I am using flyway database migration tool. Let's assume I have 100 sql scripts under a folder and I migrated them by applying on a server. Later I added 50 more sql scripts. Can I be able to move these 100 old sql scripts away and archive them somewhere (artifactory or remote share). That way I can only have 50 sql scripts which are new and needed for current migration.
Is this possible? Or all sql scripts has to be present under the directory?
It would be useful if you could edit your question to state the reason why you want to only include the 50 scripts for the current migration, as this would help me recommend an approach.
There are two ways that I know of that you could use.
Create subfolders for each release, rather than have one single folder with an unmanageable list of scripts. This approach means that all migrations remain in the project, which has advantages such as being able to rebuild a test database from scratch (using flyway clean migrate) and also being able to reuse the project later on to repeat earlier deployments. If you do it this way, you will need to reference each subfolder using the flyway.locations parameter.
Remove the sql scripts that have already been run, as you have suggested. If you do this, you will need to run flyway baseline on any targets. This command will ensure that the targets don't expect migrations below a specific version, so the missing migration files won't confuse flyway.
My personal preference is the first option. If you can't use this for whatever reason, we'd be interested to understand why.
I am trying to create a management command which will create two migrations, one for adding models, fields etc, and the other for deleting, so that I can apply one of the migrations before deploying the app and the other after deployment has been on all the servers.
Is there any simple way of achieving this without human intervention?
You could import the migration file, iterate over Migration.operations and split it into two files according to your requirements. You will also have to check if there any dependencies on that migration and adjust them.
This would still be a hack and most likely doesn't cover all the edge cases. A proper solution would probably involve rewriting large chunks django.db.migrations.
I've been looking at Flyway as a database migration tool.
The one thing that I have been unable to find a definite answer for is the following:
Can I force Flyway to run all as-of-yet unapplied migrations in a single transaction, instead of having each migration be its own transaction?
In a dev environment it's not an issue, but in a production environment where you would potentially perform multiple migrations from one update to the next, one of the migrations failing would leave the database in a 'half-migrated' state, where some migrations were committed and some not - quite a bad thing.
A workaround would be to simply cram all the SQL required in a single file, but there are issues with that:
The production migrations and dev migrations would end up being performed differently, since you cannot know in advance what will be in the migration on dev environment. I guess you could always do a clean and then a new migrate, but this seems to be against the spirit of the flyway design with regard to incremental migrations.
The checksums will be different as soon as a new change is added.
Does Flyway still not support such a feature? Does Liquibase, or any other migration tool?
There is no such feature out of the box. It's a great question though and I'd bet it was thought about since Flyway provides the transactional boundaries per migration - hopefully Axel Fontaine will chime in on the technical / design considerations that resulted in this not being a feature.
The FAQ have this and this to say regarding downgrading / failures. The policy boils down to:
Maintain backwards compatibility between the DB and all versions of
the code currently deployed in production.... Have a well tested,
backup and restore strategy.
In my case, we have being using Flyway for almost 3 years and have abided by the quoted policy. At any given deployment we could have 100 or more migrations running against many databases and happy to say have never had anything untoward happen in production. This all comes down to minimizing the opportunity for failure in your release process.
I used Liquibase on a much smaller project prior to that and don't recall any such feature apart from providing the rollback procedures.
I want to speed up my tests by running only necessary migrations to build database. Currently, Django runs all of them.
I know how to specify dependencies among migrations and actively use it. I watch for dependencies, most of my tests don't depend even on Django. I want some dividends for that.
Is there any way to specify particular apps or migrations on which test depends?
I found available_apps property of TransactionTestCase, which is marked as private API, but it doesn't work for me. Even if I run single test class or test method.