Adding a non-nullable field on a production Django deployment - django

Question

I have a production Django deployment (Django 1.11) with a PostgreSQL database. I'd like to add a non-nullable field to one of my models:
class MyModel(models.Model):
new_field = models.BooleanField(default=False)
In order to deploy, I need to either update the code on the servers or run migrations first, but because this is a production deployment, requests can (and will) happen in between my updating the database and my updating the server. If I update the server first, I will get an OperationalError no such column, so I clearly need to update the database first.
However, when I update the database first, I get the following error from requests made on the server before it is updated with the new code:
django.db.utils.IntegrityError: NOT NULL constraint failed: myapp_mymodel.new_field
On the surface, this makes no sense because the field has a default. Digging into this further, it appears that defaults are provided by Django logic alone and not actually stored on the SQL level. If the server doesn't have the updated code, it will not pass the column to SQL for the update, which SQL interprets as NULL.
Given this, how do I deploy this new non-nullable field to my application without my users getting any errors?

Answer 1

Migrations should always be run at the beginning of deployments or else you get other problems. The solution to this problem is to split the changes into two deployments.
In deployment 1, the field needs to be nullable (either a NullBooleanField or null=True). You should make a migration for the code in this state and make sure the rest of your code will not crash if the value of the field is None. This is necessary because requests can go to servers that do not yet have the new code; if those servers create instances of the model, they will create it with the field being null.
In deployment 2, you set the field to be not nullable, make a migration for this, and remove any extra code you wrote to handle cases where the value of the field is None. If the field does not have a default, the migration you make for this second deployment will need to fill in values for objects that have None in this field.
The two deployments technique is needed to safely delete fields as well, although it looks a bit different. For this, use the library django-deprecate-fields. In the first deployment, you deprecate the field in your models file and remove all references to it from your code. Then, in deployment 2, you actually delete the field from the database.

Answer 2

You can accomplish this by starting with a NullBooleanField:
Add new_field = models.NullBooleanField(default=False) to your model
Create schema migration 1 with makemigrations
Change model to have new_field = models.BooleanField(default=False)
Create schema migration 2 with makemigrations
Run schema migration 1
Update production code
Run Schema migration 2
If the old production code writes to the table between steps 5 and 6, a null value of new_field will be written. There will be a time between steps 6 and 7 where there can be null values for the BooleanField, and when the field is read, it will be null. If your code can handle this, you'll be ok, and then step 7 will convert all of those null values to False. If your new code can't handle these null values, you can perform these steps:
Add new_field = models.NullBooleanField(default=False) to your model
Create schema migration 1 with makemigrations
Run schema migration 1
Update production code
Change model to have new_field = models.BooleanField(default=False)
Create schema migration 2 with makemigrations
Run schema migration 2
Update production code
*note that these methods were only tested with Postgres.

Answer 3

Typically, a django upgrade process looks as follows:
LOCAL DEVELOPMENT ENV:
Change your model locally
Migrate the model (python manage.py makemigrations)
Test your changes locally
Commit & push your changes to (git) server
ON THE PRODUCTION SERVER:
Set ENV paramaters
Pull from your version control system (git fetch --all; git reset --hard origin/master)
update python dependencies (eg pip install -r requirements.txt)
migrate (manage.py migrate_schemas)
update static files (python manage.py collectstatic)
restart django server (depends on the server but could be something like 'python manage.py runserver')