I've encountered a very strange bug. I have a django migration that runs this query:
venues = Venue.objects.annotate(num_activities=Count('activities'))\
.filter(num_activities=1, activities__activity__code="GOLF", timings=None)\
.all()
The query itself works when the database is already up to date, but when initialising the database from scratch (running all migrations from 0001_initial), I get the error:
*** django.core.exceptions.FieldError: Cannot resolve keyword 'activities' into field.
So that migration fails. But if I run it again, with the database now in a partially migrated state (i.e. going only from 0065 -> 0066), it works just fine.
Any ideas about why it wouldn't work when running from scratch? Is there a bug in the django migration script? Seems to be due to the .annotate() function.
Related
[edit] solution found, details below
There are some interesting suggestions in a similar question but it doesn't help with my problem. Also, I have this app in production in several countries.
We are migrating an app and started with managed=False on all models. We then had a claim_batch app that has a migration 0002_capitationpayment:
dependencies = [
('product', '__first__'),
('claim_batch', '0001_initial'),
]
# Some models with foreign keys to "product.Product"
The product module doesn't have migrations but it has models. When I do a showmigrations in this situation, I have:
product
(no migrations)
And it works fine somehow.
The problem is that I now need to add migration with a managed table in product. When I do the makemigrations and try to migrate, I'm getting:
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration claim_batch.0002_capitationpayment is applied before its dependency product.0001_initial on database 'default'.
It makes sense. The claim_batch.0002 is referencing a migration that is now existing but was not applied and it should not be possible to apply it after its dependency. Except in this scenario.
If I try to remove the dependency from the already applied migration, I'm getting:
ValueError: The field claim_batch.CapitationPayment.product was declared with a lazy reference to 'product.product', but app 'product' isn't installed.
I'm guessing that a clean install with this would work fine but I have production systems in various countries running in this state. Starting from scratch is not really an option.
[EDIT]
I can get it to work by manually adding the initial migration of the product app into the django_migrations table. It's not the best solution but it's manageable.
I have updated my model, but I couldn't do the migrations, the problem is:
I am getting an error like: column "blah blah" of relation "blah blah blah" already exists
The mentioned column should exist in DB, but it shouldn't exist on migration file because I did not do any addition or modification about that model field and it was already successfully created in one of the previous migrations, even used frequently without any error.
When I do the migration with --fake, this time it doesn't create the really unexisting field of migration file which is defined with that model update.
Deployed on Heroku, it may be caused by rollbacks of code layer, since after rollbacks code gets to older versions but DB stays same.
What is the best way without losing any data from production DB?
Following is a screenshot of the bash; timezone, endtime and start time fields already exist on model and DB before this migration, created on one of the previous successful migrations
Click here to screenshot of Heroku Bash when I try to run migrations
Thanks
You should not be running makemigrations on Heroku. Do it locally, then commit the result, deploy, and then run migrate only.
As it is, you've got completely out of sync; if you don't have any data you need to keep, the easiest thing to do is to delete your db and start again.
I have a production Django deployment (Django 1.11) with a PostgreSQL database. I'd like to add a non-nullable field to one of my models:
class MyModel(models.Model):
new_field = models.BooleanField(default=False)
In order to deploy, I need to either update the code on the servers or run migrations first, but because this is a production deployment, requests can (and will) happen in between my updating the database and my updating the server. If I update the server first, I will get an OperationalError no such column, so I clearly need to update the database first.
However, when I update the database first, I get the following error from requests made on the server before it is updated with the new code:
django.db.utils.IntegrityError: NOT NULL constraint failed: myapp_mymodel.new_field
On the surface, this makes no sense because the field has a default. Digging into this further, it appears that defaults are provided by Django logic alone and not actually stored on the SQL level. If the server doesn't have the updated code, it will not pass the column to SQL for the update, which SQL interprets as NULL.
Given this, how do I deploy this new non-nullable field to my application without my users getting any errors?
Migrations should always be run at the beginning of deployments or else you get other problems. The solution to this problem is to split the changes into two deployments.
In deployment 1, the field needs to be nullable (either a NullBooleanField or null=True). You should make a migration for the code in this state and make sure the rest of your code will not crash if the value of the field is None. This is necessary because requests can go to servers that do not yet have the new code; if those servers create instances of the model, they will create it with the field being null.
In deployment 2, you set the field to be not nullable, make a migration for this, and remove any extra code you wrote to handle cases where the value of the field is None. If the field does not have a default, the migration you make for this second deployment will need to fill in values for objects that have None in this field.
The two deployments technique is needed to safely delete fields as well, although it looks a bit different. For this, use the library django-deprecate-fields. In the first deployment, you deprecate the field in your models file and remove all references to it from your code. Then, in deployment 2, you actually delete the field from the database.
You can accomplish this by starting with a NullBooleanField:
Add new_field = models.NullBooleanField(default=False) to your model
Create schema migration 1 with makemigrations
Change model to have new_field = models.BooleanField(default=False)
Create schema migration 2 with makemigrations
Run schema migration 1
Update production code
Run Schema migration 2
If the old production code writes to the table between steps 5 and 6, a null value of new_field will be written. There will be a time between steps 6 and 7 where there can be null values for the BooleanField, and when the field is read, it will be null. If your code can handle this, you'll be ok, and then step 7 will convert all of those null values to False. If your new code can't handle these null values, you can perform these steps:
Add new_field = models.NullBooleanField(default=False) to your model
Create schema migration 1 with makemigrations
Run schema migration 1
Update production code
Change model to have new_field = models.BooleanField(default=False)
Create schema migration 2 with makemigrations
Run schema migration 2
Update production code
*note that these methods were only tested with Postgres.
Typically, a django upgrade process looks as follows:
LOCAL DEVELOPMENT ENV:
Change your model locally
Migrate the model (python manage.py makemigrations)
Test your changes locally
Commit & push your changes to (git) server
ON THE PRODUCTION SERVER:
Set ENV paramaters
Pull from your version control system (git fetch --all; git reset --hard origin/master)
update python dependencies (eg pip install -r requirements.txt)
migrate (manage.py migrate_schemas)
update static files (python manage.py collectstatic)
restart django server (depends on the server but could be something like 'python manage.py runserver')
I have a migration that loads a fixture for populating the database with a basic site structure (from Loading initial data with Django 1.7 and data migrations
). After that migration ran, my test adds a (custom) NewsPage. THis yields an "IntegrityError at /admin/pages/add/website/newspage/5/
duplicate key value violates unique constraint "wagtailcore_page_pkey"
DETAIL: Key (id)=(3) already exists." The same happens when i add the page through the admin interface.
It's a bit suspicious that the Page with pk=3 is the first one that is created in my fixture. The other two pk's were already created by Wagtail's migrations.
I've read up about fixtures an migrations, and it seems Postgres won't reset the primary key sequences. I'm assuming this is also my problem here.
I found a possible solution in Django: loaddata in migrations errors, but that failed (with "psycopg2.ProgrammingError: syntax error at or near "LINE 1: BEGIN;"). Trying to execute the gist of it, I ran the sqlsequencereset management command (./manage.py sqlsequencereset wagtailcore myapp), but i still get the error, although now for id=4.
Is my assumption correct that Postgres not resetting the primary key sequences is my problem here?
Does anyone know how to reliably fix that from/after a migration loaded fixtures?
Would it maybe be easier / more reliable to create content in Python code?
Edit (same day):
If i don't follow the example in Loading initial data with Django 1.7 and data migrations, but just run the management command, it works:
def load_fixture(fixture_file):
"""Load a fixture."""
commands = StringIO()
call_command('loaddata', fixture_file, stdout=commands)
I don't know what the drawbacks of this more simple approach are.
Edit 2 (also same day):
Ok i do know, the fixtures will be based on the current model state, not the state that the migration is for, so it will likely break if your model changes.
I converted the whole thing to Python code. That works and will likely keep working. Today i learned: don't load fixtures in migrations. (Pity, it would have been a nice shortcut.)
I'm fairly new to Django, I am using 1.10 version. I reached a point where I was unable to migrate because of an error, and so backed up to an earlier migration using ./manage.py migrate myapp 0003_auto_20160426_2022 and deleted the later migration files. I then repaired my models.py and ran makemigrations, which worked fine. But when I attempted to migrate, I received the following error (only showing last few lines)
File
"/Users/wahhab/Sites/rts/env/lib/python3.5/site-packages/MySQLdb/connections.py",
line 280, in query
_mysql.connection.query(self, query) django.db.utils.OperationalError: (1022, "Can't write; duplicate key in table '#sql-72_4a6'")
I don't know how to move forward from this point so that I can continue working on my project. I have data in other apps but only a little test data in this new app so far, so I have considered deleting all migrations and the MySQL tables for this app and starting over, but I don't want to create a worse mess than I have and don't know what is causing this error. Any advice is welcome. Thanks!
Okay so a hackish solution has already been suggested by #raratiru.
Now the explanation for WHY you were facing the above problem is that when you deleted you migrations, Django resets all its counters, which also includes the counter of the key, but because the deletion of migrations was a manual one, the data still persists in your DB.
So there already exists objects in your DB with key = 1, 2, 3 ....and so on. But Django doesn't know this and hence when you have deleted just the migrations and not the data, Django faces a clash of key as Django again starts assigning automatic key values from 1, which already exists in the DB. But as the key needs to be unique, it gives you an error.
Hence if you delete both the migrations and the data, you don't get thee error.