Keyerror 'include' on migrating models to sql server - django

I have a django application and have backend as Microsoft sql server. For that, i have installed "pip install django-mssql-backend".
I have extended User model and added one additional field of confirm_password and same i am migrating, but i am getting below error
Operations to perform:
Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
Applying contenttypes.0001_initial...Traceback (most recent call last):
File "manage.py", line 22, in <module>
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\core\management\__init__.py", line 419, in execute_from_command_line
utility.execute()
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\core\management\__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\core\management\base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\core\management\base.py", line 398, in execute
output = self.handle(*args, **options)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\core\management\base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\core\management\commands\migrate.py", line 246, in handle
fake_initial=fake_initial,
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\db\migrations\executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\db\migrations\executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\db\migrations\executor.py", line 227, in apply_migration
state = migration.apply(state, schema_editor)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\db\migrations\migration.py", line 126, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\db\migrations\operations\models.py", line 531, in database_forwards
getattr(new_model._meta, self.option_name, set()),
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\sql_server\pyodbc\schema.py", line 156, in alter_unique_together
self.execute(sql)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\sql_server\pyodbc\schema.py", line 861, in execute
sql = str(sql)
File "C:\Users\RJhaveri\Documents\Ronak\SourceCode\Development\django\retailAudit\lib\site-packages\django\db\backends\ddl_references.py", line 201, in __str__
return self.template % self.parts
KeyError: 'include'
After this, i can see below two tables created in database
django_migrations
django_content_type
My Model is below:
from django.db import models
from django.contrib.auth.models import User
class UserMaster(models.Model):
user = models.OneToOneField(User, on_delete = models.CASCADE)
# Additional fields
confirm_password = models.CharField(max_length=10)
def __str__(self):
return self.user.username
Below is Forms.py
from django import forms
from django.contrib.auth.models import User
from retail_forms.models import UserMaster
class UserForm(forms.ModelForm):
password = forms.CharField(widget = forms.PasswordInput())
first_name = forms.CharField(required=True)
last_name = forms.CharField(required=True)
class Meta():
model = User
fields = ('first_name','last_name','email','username','password')
class UserMasterForm(forms.ModelForm):
confirm_password = forms.CharField(required=True)
class Meta():
model = UserMaster
fields = ('confirm_password',)
Django version: 3.2
Can someone help.

This issue is patched in pre-release version 1.0rc1 of mssql-django. GitHub issue
I recommend using mssql-django over django-pyodbc, django-pyodbc-azure-2019 or django-mssql-backend. As it's supported by microsoft and has better support imo.
To fix:
Install mssql-django pip install mssql-django==1.0rc1
Change the database engine in settings.py
DATABASES = {
'default': {
'ENGINE': 'mssql',
...
}
}

Dear I play with this alot and found that issue is django-pyodbc
I do the following:
Uninstall this using pip uninstall django-pyodbc or pip3 uninstall django-pyodbc (for Linux)
Install this using pip install django-pyodbc-azure-2019 or pip3 install django-pyodbc-azure-2019 (for Linux)
My Issue is resolved and hope your app will run smoothly after this. :-)

Diagnosis :
This is an incompatibility issue between django-mssql-backend and django. django-mssql-backend supports Django 3.0+ only.
Solution :
Rollback to Django 3.0.14 and then run your migrations. It works fine.
Alternatively, you can try this fork supporting 3.2 : https://github.com/microsoft/mssql-django/issues/50

Changing the Engine name work for me:
# settings.py
DATABASES = {
"default": {
"ENGINE": "mssql",
"NAME": "DATABASE_NAME",
"USER": "USER_NAME",
"PASSWORD": "PASSWORD",
"HOST": "HOST_ADDRESS",
"PORT": "1433",
"OPTIONS": {"driver": "ODBC Driver 17 for SQL Server",
},
},
}
For more details follow the link: https://learn.microsoft.com/en-us/samples/azure-samples/mssql-django-samples/mssql-django-samples/

Related

django.db.utils.OperationalError: near "[]": syntax error

There is this error showing after I have used makemigrations command
I have tried commenting different column for it but it wont work
C:\Users\Rushabh\Desktop\project\MyPrj>python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, paper, sessions
Running migrations:
Applying paper.0014_auto_20170405_1549...Traceback (most recent call last):
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\utils.py", line 62, in execute
return self.cursor.execute(sql)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\sqlite3\base.py", line 335, in execute
return Database.Cursor.execute(self, query)
sqlite3.OperationalError: near "[]": syntax error
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\core\management\__init__.py", line 367, in execute_from_command_line
utility.execute()
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\core\management\__init__.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\core\management\base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\core\management\base.py", line 345, in execute
output = self.handle(*args, **options)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\core\management\commands\migrate.py", line 204, in handle
fake_initial=fake_initial,
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\migrations\executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\migrations\executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\migrations\executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\migrations\migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\migrations\operations\fields.py", line 84, in database_forwards
field,
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\sqlite3\schema.py", line 231, in add_field
self._remake_table(model, create_fields=[field])
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\sqlite3\schema.py", line 191, in _remake_table
self.create_model(temp_model)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\base\schema.py", line 295, in create_model
self.execute(sql, params or None)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\base\schema.py", line 112, in execute
cursor.execute(sql, params)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\utils\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\utils.py", line 62, in execute
return self.cursor.execute(sql)
File "C:\Users\Rushabh\Anaconda3\lib\site-packages\django\db\backends\sqlite3\base.py", line 335, in execute
return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: near "[]": syntax error
Migration file
from __future__ import unicode_literals
import django.contrib.postgres.fields
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('paper', '0019_auto_20170405_1659'),
]
operations = [
migrations.AddField(
model_name='test',
name='checked',
field=models.BooleanField(default=False),
),
migrations.AddField(
model_name='test',
name='mark3',
field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=5), default=[], size=None),
),
migrations.AddField(
model_name='test',
name='mark4',
field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=5), default=[], size=None),
),
migrations.AddField(
model_name='test',
name='mark7',
field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=5), default=[], size=None),
),
migrations.AddField(
model_name='test',
name='request',
field=models.BooleanField(default=False),
),
]
The problem arose after I added the following field to my models
mark3=ArrayField(models.CharField(max_length=5),default=[])
The models.py file is
from django.db import models
from django.contrib.postgres.fields import ArrayField
class User(models.Model):
user_id=models.CharField(unique=True,max_length=50)
password=models.CharField(max_length=50)
role=models.IntegerField(blank=False)
def __str__(self):
return self.user_id
class Qbank(models.Model):
user=models.ForeignKey(User,on_delete=models.CASCADE)
qbank_id=models.CharField(unique=True,max_length=50,blank=False)
# subject_id=models.CharField(max_length=50)
qbank_file=models.FileField(upload_to= 'qbs/',blank=False)
ans_file=models.FileField(upload_to= 'ans/',blank=False)
uploaded_at = models.DateTimeField(auto_now_add=True)
def __str__(self):
return self.qbank_id
class Test(models.Model):
qbank=models.ForeignKey(Qbank,on_delete=models.CASCADE)
test_id=models.CharField(unique=True,max_length=50)
request=models.BooleanField(default=False)
checked=models.BooleanField(default=False)
mark3=ArrayField(models.CharField(max_length=5),default=[])
mark4=ArrayField(models.CharField(max_length=5),default=[])
mark7=ArrayField(models.CharField(max_length=5),default=[])
# true if send and checked respt
def __str__(self):
return self.test_id
I puzzled over this one for a little while too. The ArrayField is specific to Postgres, and is imported from a Postgres library:
import django.contrib.postgres.fields
It looks like you're trying to commit your migrations to SQLite. You should set up a local Postgres database, and update your settings.py file from:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
To:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'DATABASE NAME',
'USER': 'USER NAME',
'PASSWORD': 'USER PASSWORD',
'HOST': 'localhost',
'PORT': '5432',
}
}
Also, you are incorrectly setting the default value for your ArrayField. Per Django ArrayField documentation, you should not use [] as the default. It won't cause this problem, but it will probably create some others! You should use default=list instead of default=[], which will create a mutable default shared between all instances of ArrayField:
Instead of:
mark7=ArrayField(models.CharField(max_length=5),default=[])
Try:
mark7=ArrayField(models.CharField(max_length=5),default=list)
I stupidly used this specific posgresql field - ArrayField for the model and let the test run with sqlite. That caused the error when I pushed code to github with the travis-ci.
The ArrayField docs warn that you shouldn't use a mutable value like []. You can use the callable list for an empty default.
class Test(models.Model):
...
mark3=ArrayField(models.CharField(max_length=5), default=list)
mark4=ArrayField(models.CharField(max_length=5), default=list)
mark7=ArrayField(models.CharField(max_length=5), default=list)
After making this change, delete the old migration file and run makemigrations again.

Django 1.9 - created new app run migrate, tables are not created

ive added a new app and created some models. I have run makemigrations and migrate and all went successfully. However when i open the sqlite table, none of the tables are created.
sample from models:
from __future__ import unicode_literals
from django.db import models
from django.conf import settings
import string
import random
import time
import os
# Create your models here.
from service.models import ServiceContacts
class Subnets(models.Model):
subnet = models.GenericIPAddressField(protocol='IPv4',verbose_name="Subnet",blank=True,null=True)
subnet_mask = models.CharField(max_length=4,verbose_name="Subnet Mask",choices=settings.SUBNET_MASK_CHOICES,blank=True)
subnet_type = models.CharField(max_length=10 ,verbose_name='Subnet Type',choices=settings.SUBNET_TYPE_CHOICES,blank=True)
class Meta:
verbose_name = "Site Subnet Data"
verbose_name_plural = "Site Subnet Data"
class SiteContacts(models.Model):
name = models.CharField(max_length=200)
title = models.CharField(max_length=200)
mobile = models.CharField(max_length=200,blank=True, null=True)
ddi = models.CharField(max_length=200,blank=True, null=True)
notes = models.TextField(blank=True, null=True)
class Meta:
verbose_name = "Site Contact Data"
verbose_name_plural = "Site Contact Data"
sample from inital
# -*- coding: utf-8 -*-
# Generated by Django 1.9.6 on 2016-11-09 17:32
from __future__ import unicode_literals
from django.db import migrations, models
import django.db.models.deletion
import sites.models
class Migration(migrations.Migration):
initial = True
dependencies = [
('service', '0007_auto_20160701_0931'),
]
operations = [
migrations.CreateModel(
name='CircuitFiles',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('circuit_file', models.FileField(blank=True, upload_to=sites.models.service_upload_path)),
('file_name', models.CharField(max_length=200, verbose_name='File Name')),
],
options={
'verbose_name': 'Circuit Files',
'verbose_name_plural': 'Circuit Files',
},
),
migrations.CreateModel(
name='CircuitNotes',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('date', models.DateField()),
('notes', models.TextField(blank=True)),
],
options={
'verbose_name': 'Circuit Notes',
'verbose_name_plural': 'Circuit Notes',
},
),
tables from sqlite3db
[root#network-tools infternal]# sqlite3 db.sqlite3
SQLite version 3.7.17 2013-05-20 00:56:22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables
auth_group networks_circuitnotes
auth_group_permissions networks_configtemplates
auth_permission networks_configvariables
auth_user networks_majorsiteinfodata
auth_user_groups networks_networkstock
auth_user_user_permissions networks_networkstockusage
django_admin_log networks_showroomconfigdata
django_content_type networks_sitecontacts
django_migrations networks_sitefiles
django_session networks_sitefiletype
django_site networks_snmpdata
sqlite>
i should see sites_subnets and sites_sitecontacts in there, as well as many others.
the migrations table has the below in it, i cant see sites|0001_inital in there...
100|networks|0069_configvariables_type|2016-11-03 15:17:41.424747
101|networks|0070_circuitinfodata_circuit_preference|2016-11-09 09:11:29.358213
102|networks|0071_auto_20161109_0915|2016-11-09 09:15:22.455639
103|networks|0072_auto_20161109_0916|2016-11-09 09:16:25.962542
104|sites|0002_auto_20161110_0859|2016-11-10 08:59:31.071382
installed Apps:
INSTALLED_APPS = (
'home.apps.HomeConfig',
'oncall.apps.OncallConfig',
'networks.apps.NetworksConfig',
'sites.apps.SitesConfig',
Make Migrations:
[root#network-tools infternal]# python manage.py makemigrations
Migrations for 'networks':
0073_auto_20161110_1059.py:
- Alter field circuit_type on circuitinfodata
Migrations for 'sites':
0003_auto_20161110_1059.py:
- Alter field circuit_type on circuits
[root#network-tools infternal]# python manage.py migrate
Operations to perform:
Apply all migrations: service, sessions, admin, sites, auth, contenttypes, maintenance, oncall, networks
Running migrations:
Rendering model states... DONE
Applying networks.0073_auto_20161110_1059... OK
Applying sites.0003_auto_20161110_1059...Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/lib64/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/usr/lib64/python2.7/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib64/python2.7/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib64/python2.7/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/usr/lib64/python2.7/site-packages/django/core/management/commands/migrate.py", line 200, in handle
executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
File "/usr/lib64/python2.7/site-packages/django/db/migrations/executor.py", line 92, in migrate
self._migrate_all_forwards(plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/usr/lib64/python2.7/site-packages/django/db/migrations/executor.py", line 121, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/lib64/python2.7/site-packages/django/db/migrations/executor.py", line 198, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/lib64/python2.7/site-packages/django/db/migrations/migration.py", line 123, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/lib64/python2.7/site-packages/django/db/migrations/operations/fields.py", line 201, in database_forwards
schema_editor.alter_field(from_model, from_field, to_field)
File "/usr/lib64/python2.7/site-packages/django/db/backends/base/schema.py", line 482, in alter_field
old_db_params, new_db_params, strict)
File "/usr/lib64/python2.7/site-packages/django/db/backends/sqlite3/schema.py", line 253, in _alter_field
self._remake_table(model, alter_fields=[(old_field, new_field)])
File "/usr/lib64/python2.7/site-packages/django/db/backends/sqlite3/schema.py", line 184, in _remake_table
self.alter_db_table(model, temp_model._meta.db_table, model._meta.db_table)
File "/usr/lib64/python2.7/site-packages/django/db/backends/base/schema.py", line 359, in alter_db_table
"new_table": self.quote_name(new_db_table),
File "/usr/lib64/python2.7/site-packages/django/db/backends/base/schema.py", line 110, in execute
cursor.execute(sql, params)
File "/usr/lib64/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/usr/lib64/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/lib64/python2.7/site-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib64/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/lib64/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 323, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: sites_circuits
Try to migrate particular app using following process.
If you create initial migration and then run migrate command sometime migration not don completely so try to migrate using following command
python manage.py makemigrations
Initial migration created then run migrate command with app name
python manage.py migrate appname
Hope this is help you
ok heres what i did.
python manage.py sqlmigrate sites 0001
this gave me the sql for the migration, i then run
sqlite3 db.sqlite3
which opens the DB, i then just copied and pasted in the output from the sqlmigrate command into sql and its all working now

Django ProgrammingError in Fields on PostgreSQL

models.py
class Stop(models.Model):
idn = models.PositiveIntegerField(primary_key=True, unique=True)
label = models.CharField(null=False, blank=False, max_length=512)
coor_x = models.FloatField()
coor_y = models.FloatField()
buses = models.ManyToManyField(Bus)
latest_query_datetime = models.DateTimeField(default=datetime(2000, 1, 1, 0, 0, 0))
latest_query_data = JSONField(default={})
class Meta:
ordering = ["label"]
def __str__(self):
return self.label
When I run:
python3 manage.py makemigrations && python3 manage.py migrate
It raises a ProgrammingError saying that jsonb datatype does not exist:
Migrations for 'rest':
0007_auto_20160612_1301.py:
- Alter field latest_query_data on stop
Operations to perform:
Apply all migrations: contenttypes, rest, auth, sessions, admin
Running migrations:
Rendering model states... DONE
Applying rest.0005_auto_20160612_1237...Traceback (most recent call last):
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: type "jsonb" does not exist
LINE 1: ... TABLE "rest_stop" ADD COLUMN "latest_query_data" jsonb DEFA...
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 200, in handle
executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/migrations/executor.py", line 92, in migrate
self._migrate_all_forwards(plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/migrations/executor.py", line 121, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/migrations/executor.py", line 198, in apply_migration
state = migration.apply(state, schema_editor)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/migrations/migration.py", line 123, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/migrations/operations/fields.py", line 62, in database_forwards
field,
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 396, in add_field
self.execute(sql, params)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 110, in execute
cursor.execute(sql, params)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/home/erayerdin/.venv/eshot-api/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: type "jsonb" does not exist
LINE 1: ... TABLE "rest_stop" ADD COLUMN "latest_query_data" jsonb DEFA...
I use PostgreSQL to use JSONField and update it when a user requests a view. If I do not use default={}, it tells me to make one.
Further Analysis
I changed latest_query_data field to TextField so that I can store as string and convert to dict when I need. However, this also raised the same error.
Environment
django 1.9.6
psycopg 2.6.1
According to the Django docs, JSONField requires PostgreSQL ≥ 9.4 and Psycopg2 ≥ 2.5.4
What PostgreSQL version are you using?
See https://docs.djangoproject.com/en/dev/ref/contrib/postgres/fields/#django.contrib.postgres.fields.JSONField
Ubuntu 14.04 repositories contain only 9.3 version. You can review this to upgrade your version.
Based on the Anonymous comment, I found the following to work:
from django.contrib.postgres import fields
class OldJSONField(fields.JSONField):
def db_type(self, connection):
return 'json'
class Stop(models.Model):
...
latest_query_data = OldJSONField(default=dict)
...
If you are getting this error and you have installed Postgres > 9.4 then I would check that you aren't connecting to an older version of Postgres that is also installed on your instance.
To confirm what you are connecting to from Django you can use psycopg2 from the shell:
import psycopg2
conn = psycopg2.connect("dbname=<your database> user=<your user> password=<your password>")
cur = conn.cursor()
cur.execute("SELECT Version();")
cur.fetchone()
Make sure the version here is > 9.4. If not you probably have a couple of versions installed and your service configuration is pointing at the other version.
With older versions of PostgreSQL for example "PostgreSQL9.2.x",
We can use an alternative as
from jsonfield import JSONField
instead of:
from django.contrib.postgres.fields import JSONField
for example:
from django.db import models
from jsonfield import JSONField
class MyModel(models.Model):
json = JSONField()
Can install as:
pip install jsonfield
Check this out:
Add support for PostgreSQL 9.2+'s native json type. #32
This solution is especially good for some limited edition versions. For example,
on VPS & Cpanel that supports postgresql9.2 by default
more details!, see"rpkilby-jsonfield"
So, it seems a bug on psycopg2 or django, I will post an issue on both repositories. This is how I've solved (at least, ProgrammingError) the problem.
Change JSONField to TextField.
Flush your database.
Beware! This operation will erase all the data but the structure in your database.
Remove all migrations folder in all of your apps.
Run python3 manage.py makemigrations && python3 manage.py migrate in all of your apps.
Run python manage.py makemigrations <appname> && python3 manage.py migrate <appname> for each app you have.
Use built-in json module to convert between str and dict.
However, remember, this solution requires so much effort if you want to filter a QuerySet of a model. I do not recommend it, but there was no other solution to get rid of this error and all I needed to do was to save data and represent it.
! This answer will be accepted as default if there will not occur any other better solution in 48 hours.

Django migration fails with "__fake__.DoesNotExist: Permission matching query does not exist."

In a Django 1.8 project, I have a migration that worked fine, when it had the following code:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import migrations
from django.conf import settings
def update_site_forward(apps, schema_editor):
"""Add group osmaxx."""
Group = apps.get_model("auth", "Group")
Group.objects.create(name=settings.OSMAXX_FRONTEND_USER_GROUP)
def update_site_backward(apps, schema_editor):
"""Revert add group osmaxx."""
Group = apps.get_model("auth", "Group")
Group.objects.get(name=settings.OSMAXX_FRONTEND_USER_GROUP).delete()
class Migration(migrations.Migration):
dependencies = [
('auth', '0001_initial'),
]
operations = [
migrations.RunPython(update_site_forward, update_site_backward),
]
This group is created in a migration, because it shall be available in all installations of the web app. To make it more useful, I wanted to also give it a default permission, so I changed update_site_forward to:
def update_site_forward(apps, schema_editor):
"""Add group osmaxx."""
Group = apps.get_model("auth", "Group")
Permission = apps.get_model("auth", "Permission")
ContentType = apps.get_model("contenttypes", "ContentType")
ExtractionOrder = apps.get_model("excerptexport", "ExtractionOrder")
group = Group.objects.create(name=settings.OSMAXX_FRONTEND_USER_GROUP)
content_type = ContentType.objects.get_for_model(ExtractionOrder)
permission = Permission.objects.get(codename='add_extractionorder',
content_type=content_type) # line 16
group.permissions.add(permission)
and Migration.dependencies to:
dependencies = [
('contenttypes', '0002_remove_content_type_name'),
('excerptexport', '0001_initial'),
('auth', '0001_initial'),
]
While applying the migration (after first reverting it) (python3 manage.py migrate auth 0001 && python3 managy.py migrate) worked, migrating a newly created PostgreSQL database with this and all other migrations (python3 manage.py migrate) fails:
Operations to perform:
Synchronize unmigrated apps: debug_toolbar, django_extensions, messages, humanize, social_auth, kombu_transport_django, staticfiles
Apply all migrations: excerptexport, admin, sites, contenttypes, sessions, default, stored_messages, auth
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying auth.0002_add_default_usergroup_osmaxx...Traceback (most recent call last):
File "manage.py", line 17, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.4/dist-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.4/dist-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.4/dist-packages/django/core/management/base.py", line 393, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.4/dist-packages/django/core/management/base.py", line 444, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.4/dist-packages/django/core/management/commands/migrate.py", line 221, in handle
executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.4/dist-packages/django/db/migrations/executor.py", line 110, in migrate
self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python3.4/dist-packages/django/db/migrations/executor.py", line 148, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python3.4/dist-packages/django/db/migrations/migration.py", line 115, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python3.4/dist-packages/django/db/migrations/operations/special.py", line 183, in database_forwards
self.code(from_state.apps, schema_editor)
File "/home/osmaxx/source/osmaxx/contrib/auth/migrations/0002_add_default_usergroup_osmaxx.py", line 16, in update_site_forward
permission = Permission.objects.get(codename='add_extractionorder', content_type=content_type)
File "/usr/local/lib/python3.4/dist-packages/django/db/models/manager.py", line 127, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/django/db/models/query.py", line 334, in get
self.model._meta.object_name
__fake__.DoesNotExist: Permission matching query does not exist.
What am I doing wrong?
The default permissions are created in a post_migrate signal handler, after the migrations have run. This won't be a problem if your updated code runs as part of the second manage.py migrate run, but it is a problem in the test suite and any new deployment.
The easy fix is to change this line:
permission = Permission.objects.get(codename='add_extractionorder',
content_type=content_type) # line 16
to this:
permission, created = Permission.objects.get_or_create(codename='add_extractionorder',
content_type=content_type)
The signal handler that creates the default permissions will never create a duplicate permission, so it is safe to create it if it doesn't exist already.

Django migrate model with user in a foreignkey fails

Im trying to migrate my django model:
from django.contrib.auth.models import User
class Post(models.Model):
headline = models.CharField(max_length=200)
slug = models.SlugField(max_length=200)
body = models.TextField(blank=True, null=True)
author = models.ForeignKey(User, null=True, blank=True)
I added the author field after I created the model.
Here is the migration django creates:
# encoding: utf8
from django.db import models, migrations
class Migration(migrations.Migration):
dependencies = [('articles', '0002_auto')]
operations = [
migrations.AddField(
field = models.ForeignKey(to_field=u'id', to=u'auth.User', blank=True, null=True),
name = 'author',
model_name = 'post',
),
]
Here is my traceback when I try to run ./manage.py migrate:
Operations to perform:
Synchronize unmigrated apps: ckeditor, sessions, admin, messages, auth, staticfiles, contenttypes, django_extensions
Apply all migrations: articles
Synchronizing apps without migrations:
Creating tables...
Installing custom SQL...
Installing indexes...
Installed 0 object(s) from 0 fixture(s)
Running migrations:
Applying articles.0002_post_author...Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/core/management/__init__.py", line 397, in execute_from_command_line
utility.execute()
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/core/management/__init__.py", line 390, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/core/management/base.py", line 289, in execute
output = self.handle(*args, **options)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/core/management/commands/migrate.py", line 116, in handle
executor.migrate(targets, plan, fake=options.get("fake", False))
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/migrations/executor.py", line 60, in migrate
self.apply_migration(migration, fake=fake)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/migrations/executor.py", line 73, in apply_migration
migration.apply(project_state, schema_editor)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/migrations/migration.py", line 80, in apply
operation.database_forwards(self.app_label, schema_editor, project_state, new_state)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/migrations/operations/fields.py", line 22, in database_forwards
schema_editor.add_field(from_model, to_model._meta.get_field_by_name(self.name)[0])
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/backends/schema.py", line 349, in add_field
definition, params = self.column_sql(model, field, include_default=True)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/backends/schema.py", line 105, in column_sql
db_params = field.db_parameters(connection=self.connection)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/models/fields/related.py", line 1285, in db_parameters
return {"type": self.db_type(connection), "check": []}
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/models/fields/related.py", line 1276, in db_type
rel_field = self.related_field
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/models/fields/related.py", line 1183, in related_field
return self.foreign_related_fields[0]
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/models/fields/related.py", line 971, in foreign_related_fields
return tuple(rhs_field for lhs_field, rhs_field in self.related_fields)
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/models/fields/related.py", line 958, in related_fields
self._related_fields = self.resolve_related_fields()
File "/home/USER/.virtualenvs/PROJECT/src/django-trunk/django/db/models/fields/related.py", line 943, in resolve_related_fields
raise ValueError('Related model %r cannot been resolved' % self.rel.to)
ValueError: Related model u'auth.User' cannot been resolved
Anyone know what I'm doing wrong?
What helped me in this situation:
Delete all migration files except __init__.py (/%prjname%/migrations folder)
python manage.py makemigrations
python manage.py migrate
Not sure about exact cause, but i tried to use files, generated by my code-partner and it didn't work out.
Ok, this is another funky feature of Django which cost me hours to figure it out. According to https://docs.djangoproject.com/en/1.8/topics/auth/customizing/#substituting-a-custom-user-model:
Due to limitations of Django’s dynamic dependency feature for swappable models, you must ensure that the model referenced by AUTH_USER_MODEL is created in the first migration of its app (usually called 0001_initial); otherwise, you will have dependency issues.
So to solve this problem the best "clean" way is to put your custom user model creation in 0001_initial.py and it will simply work. And that's the real reason why Lebedev Sergey's delete/makemigrations trick can work.
If you make this change after you have applied ANY other migrations, you need to delete everything else in the migration folder and then run "python manage.py makemigrations". Then whatever you used for AUTH_USER_MODEL will be your first migration.
This probably isn't your problem if you're not using a custom user model, but remember to always use get_user_model() or when referencing the User class. Also, when defining a foreign key, settings.AUTH_USER_MODEL works, too, as in:
class MyModel(models.Model):
person = models.ForeignKey(settings.AUTH_USER_MODEL)