I have a Django site that's working fine. It has a small amount of data in the database which I want to use for testing. I've done dumpdata to generate a few .json fixtures. But, when I try to run a test on my app ("tagger") I get a Postgresql error, and I can't work out how to solve it:
(django-projectname)$ ./manage.py test tagger
Creating test database for alias 'default'...
Problem installing fixture '/Users/phil/Projects/RIG/projectname/django-projectname/projectname/tagger/fixtures/auth_testdata.json': Traceback (most recent call last):
File "/Users/phil/.virtualenvs/django-projectname/lib/python2.6/site-packages/django/core/management/commands/loaddata.py", line 174, in handle
obj.save(using=using)
File "/Users/phil/.virtualenvs/django-projectname/lib/python2.6/site-packages/django/core/serializers/base.py", line 165, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/Users/phil/.virtualenvs/django-projectname/lib/python2.6/site-packages/django/db/models/base.py", line 526, in save_base
rows = manager.using(using).filter(pk=pk_val)._update(values)
File "/Users/phil/.virtualenvs/django-projectname/lib/python2.6/site-packages/django/db/models/query.py", line 491, in _update
return query.get_compiler(self.db).execute_sql(None)
File "/Users/phil/.virtualenvs/django-projectname/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 869, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/Users/phil/.virtualenvs/django-projectname/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
cursor.execute(sql, params)
File "/Users/phil/.virtualenvs/django-projectname/lib/python2.6/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
return self.cursor.execute(query, args)
IntegrityError: duplicate key value violates unique constraint "auth_permission_content_type_id_codename_key"
DETAIL: Key (content_type_id, codename)=(3, add_htuser) already exists.
I tried doing dumpdata for the auth app using the --natural flag too, but that didn't change anything. I'm stumped, and can't work out where it's getting a duplicate key from.
The "htuser" class that it refers to is a Proxy model on the auth User class.
I'm using South for my apps' migrations and it sounds like https://stackoverflow.com/a/2888916/250962 may be the answer, but I don't understand how to "Remove the explicitly-set primary key values from the datamigrations"?
Related
I can run all unit tests successfully, I can even run selenium tests successfully if I run an independent server, but when I try to use LiveServerTestCases to test everything in a self-contained manner, each LiveServerTestCase test ends with the following error after completing the tearDown function:
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\db\backends\base\base.py", line 239, in _commit
return self.connection.commit()
django.db.utils.IntegrityError: FOREIGN KEY constraint failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\test\testcases.py", line 209, in __call__
self._post_teardown()
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\test\testcases.py", line 908, in _post_teardown
self._fixture_teardown()
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\test\testcases.py", line 943, in _fixture_teardown
inhibit_post_migrate=inhibit_post_migrate)
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
return command.execute(*args, **defaults)
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\core\management\base.py", line 353, in execute
output = self.handle(*args, **options)
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\core\management\commands\flush.py", line 80, in handle
emit_post_migrate_signal(verbosity, interactive, database)
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\core\management\sql.py", line 51, in emit_post_migrate_signal
**kwargs
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\dispatch\dispatcher.py", line 175, in send
for receiver in self._live_receivers(sender)
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\dispatch\dispatcher.py", line 175, in <listcomp>
for receiver in self._live_receivers(sender)
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\contrib\auth\management\__init__.py", line 79, in create_permissions
Permission.objects.using(using).bulk_create(perms)
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\db\models\query.py", line 471, in bulk_create
obj_without_pk._state.db = self.db
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\db\transaction.py", line 212, in __exit__
connection.commit()
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\db\backends\base\base.py", line 261, in commit
self._commit()
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\db\backends\base\base.py", line 239, in _commit
return self.connection.commit()
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\db\utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\db\backends\base\base.py", line 239, in _commit
return self.connection.commit()
django.db.utils.IntegrityError: FOREIGN KEY constraint failed
I'm pretty sure I don't have any errors in my database models, as all the unit tests run fine and the selenium tests run fine when I fire up a seperate server instance to run in parallel, so I'm guessing it has to do with selenium?
I've tried using the Chrome webdriver, IE webdriver and Firefox webdriver. Same results.
It doesn't appear to be related to my database as the error only occurs for LiveServerTestCases.
Environment Details
Django version 2.1
SQLite3 version 3.20.0
Some more information regarding the Django version, Database type and version along with your code trials would have helped us to debug this issue in a better way.
However, this error message...
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\core\management\sql.py", line 51, in emit_post_migrate_signal **kwargs
.
File "C:\Users\Win7\.virtualenvs\lang-QbOXb8q_\lib\site-packages\django\db\backends\base\base.py", line 239, in _commit
return self.connection.commit()
django.db.utils.IntegrityError: FOREIGN KEY constraint failed
...implies that an IntegrityError was raised while attempting to save an existing model instance.
As per Django 2.0 release notes:
Foreign key constraints are now enabled on SQLite: This was a backwards-incompatible change (IntegrityError: FOREIGN KEY constraint failed) if attempting to save an existing model instance that’s violating a foreign key constraint.
Foreign Keys are now created with DEFERRABLE INITIALLY DEFERRED instead of DEFERRABLE IMMEDIATE. So the tables may need to be rebuilt to recreate foreign keys with the new definition, particularly if you’re using a pattern as follows;
from django.db import transaction
with transaction.atomic():
Book.objects.create(author_id=1)
Author.objects.create(id=1)
If you don’t recreate the foreign key as DEFERRED, the first create() would fail as the foreign key constraints are enforced.
#dirkgroten in this discussion provided an example as follows:
Look for patterns like this in your code:
# in pagetree/models.py, line 810
#classmethod
def create_from_dict(cls, d):
return cls.objects.create() # what happens to d by the way?
This will definitely fail with a ForeignKey constraint error since a PageBlock must have section, so you can't call create without first assigning it.
An interesting issue.
Getting Unknown column exception -- Please find the stack trace
I try to get new leads list and responded leads. I merge them. When I merge them there is an exception.
After debugging its found that new_leads method has exclude of two fields collection and delivery . If we make it one exclude all is well . I mean dont check the other, if we include both the filters we have an issue.
I tried using filter/ exclude etc. but it didnt work.
Query Set contains following method
def all_leads_related_to_user(self, user):
""" User new and past leads
Use this queryset for performing lead search.
"""
new_leads = self.new_leads_for_user(user)
responded_leads = self.leads_responded_by_user(user)
all_leads = (new_leads | responded_leads).distinct() <= Issue is here.
return all_leads
def new_leads_for_user(self, user):
....
# User's location filter
if user.sub_region_excluded_list:
sub_region_exclude_list = [10, 12]
qs = qs.exclude( Q(collection_point__sub_region_id__in=sub_region_exclude_list) |
Q(delivery_point__sub_region_id__in=sub_region_exclude_list))
# <== Make it just one exclude it works.
Model
class Suburb(models.Model):
state = models.ForeignKey(State, blank=False)
sub_region = models.ForeignKey(SubRegion, blank=False)
postcode = models.CharField(_('postcode'), blank=False, max_length=10)
name = models.CharField(_('suburb name'), blank=False, max_length=200)
class Load(models.Model):
.....
collection_point = models.ForeignKey(Suburb, related_name='collection_point', on_delete=models.SET_NULL, null=True)
delivery_point = models.ForeignKey(Suburb, related_name='delivery_point', on_delete=models.SET_NULL, null=True)
Stack Trace:-
>>> Load.objects.all_leads_related_to_user(User.objects.all()[0])
Load.objects.all_leads_related_to_user(User.objects.all()[0])
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/data/fq/venv/lib/python3.4/site-packages/django/db/models/query.py", line 226, in __repr__
data = list(self[:REPR_OUTPUT_SIZE + 1])
File "/data/fq/venv/lib/python3.4/site-packages/django/db/models/query.py", line 250, in __iter__
self._fetch_all()
File "/data/fq/venv/lib/python3.4/site-packages/django/db/models/query.py", line 1103, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/data/fq/venv/lib/python3.4/site-packages/django/db/models/query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
File "/data/fq/venv/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 886, in execute_sql
raise original_exception
File "/data/fq/venv/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 876, in execute_sql
cursor.execute(sql, params)
File "/data/fq/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 80, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/data/fq/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/data/fq/venv/lib/python3.4/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/data/fq/venv/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/data/fq/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/data/fq/venv/lib/python3.4/site-packages/django/db/backends/mysql/base.py", line 101, in execute
return self.cursor.execute(query, args)
File "/data/fq/venv/lib/python3.4/site-packages/MySQLdb/cursors.py", line 250, in execute
self.errorhandler(self, exc, value)
File "/data/fq/venv/lib/python3.4/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
raise errorvalue
File "/data/fq/venv/lib/python3.4/site-packages/MySQLdb/cursors.py", line 247, in execute
res = self._query(query)
File "/data/fq/venv/lib/python3.4/site-packages/MySQLdb/cursors.py", line 411, in _query
rowcount = self._do_query(q)
File "/data/fq/venv/lib/python3.4/site-packages/MySQLdb/cursors.py", line 374, in _do_query
db.query(q)
File "/data/fq/venv/lib/python3.4/site-packages/MySQLdb/connections.py", line 292, in query
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (1054, "Unknown column 'locations_suburb.sub_region_id' in 'having clause'")
>>>
I'm using MySqlDB
Note:-
All migrations are applied and Db is in right state
Update
Issue is related to MYSQL madating columns to be available in select statements when its creating a Having clause . Refer to -
Unknown column in 'having clause'.
Django in this instance is not adding as the column to be selected. hence the error.
I need to some how find a way to add this in the select clause with other params.
This usually pops up when you haven't made or applied migrations. Specifically, when you modify the fields any model in Django (or any ORM), you need to inform the SQL server so it can reflect it in its tables. Modern Django implements this by a series of migrations, so that if you have data from any time in the life of your project, you can run it on code from any time in history by simply running the migrations forwards or backwards.
Long story short, MySQL claims the sub_region field doesn't exist. You need to sync the tables to reflect your models.
There are two steps, making the migrations and running them on your server, shown below for your locations app. Take a backup before running the second command, especially on MySQL or SQLite!
$ python manage.py makemigrations locations
$ python manage.py migrate locations
This will cause the database server to create the column, and you should no longer get an OperationalError.
It is not a migration issue.
When having a HAVING Clause MYSQL mandates those parameters to be in selected, if not selected it will throw an exception.
Therefore I had to force Django to include both the fields by:
qs = qs.annotate(collection_point__sub_region_id = F("collection_point__sub_region_id"),
delivery_point__sub_region_id = F("delivery_point__sub_region_id"))
This is the answer to this issue:
Step 1. You need to fake the migrations
python manage.py migrate --fake
Step 2. Migrate again
python manage.py migrate
Step 3. Comment out the column for which the error comes up e.g. "Can't DROP 'address'
Step 4. Make migrations and migrate again
python manage.py makemigrations
python manage.py migrate
Since my table didn't have any data, I deleted it and recreated it on MySQL and then deleted it from Django. Finally I recreated on Django thus solving the error.
I am getting this error after upgrading to django 1.4 while running tests using sqlite:
Creating test database for alias 'default'...
Problem installing fixture '/home/devasia/pyserver/project/mysite/app/fixtures/test_data.json': Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/loaddata.py", line 196, in handle
obj.save(using=using)
File "/usr/local/lib/python2.7/dist-packages/django/core/serializers/base.py", line 165, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 529, in save_base
rows = manager.using(using).filter(pk=pk_val)._update(values)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 557, in _update
return query.get_compiler(self.db).execute_sql(None)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 986, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 818, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 337, in execute
return Database.Cursor.execute(self, query, params)
IntegrityError: Could not load auth.Permission(pk=31): columns content_type_id, codename are not unique
----------------------------------------------------------------------
Ran 1 test in 0.011s
OK
Destroying test database for alias 'default'...
This is my sample test case:
class CheckTest(TestCase):
fixtures = ['test_data.json']
def test_check(self):
c = Client()
c.login(username = 'test', password = 'test')
The fixtures are not that of old version. I have created this fixture using django 1.4 starting from 'syncdb' command. The tests are running well if I use postgresql as my test database(the tests run really slow).
Did I miss something while upgrading? I found this https://code.djangoproject.com/ticket/14731 but couldn't resolve the issue. (Please note that the only code/data from old project is the models)
Thanks
I'm loading a fixture created with dumpdata, and getting the following exception:
Problem installing fixture 'db_dump.json': Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/django/core/management/commands/loaddata.py", line 174, in handle
obj.save(using=using)
File "/usr/lib/python2.6/site-packages/django/core/serializers/base.py", line 165, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/usr/lib/python2.6/site-packages/django/db/models/base.py", line 526, in save_base
rows = manager.using(using).filter(pk=pk_val)._update(values)
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 491, in _update
return query.get_compiler(self.db).execute_sql(None)
File "/usr/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 869, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/usr/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.6/site-packages/django/db/backends/sqlite3/base.py", line 234, in execute
return Database.Cursor.execute(self, query, params)
IntegrityError: columns app_label, model are not unique
This is with a sqlite3 backend.
Update: Using natural keys doesn't make a difference here.
What does it mean, and why is it happening?
Apparently one of the traps for the unwary is that one must exclude contenttypes when exporting fixtures. (Thanks to subsume on #django for the information).
To exclude content types use the -e option when running the dumpdata command.
$./manage.py dumpdata -e contenttypes > initial_data.json
python manage.py dumpdata --exclude=contenttypes --exclude=auth.Permission > initial_data.json
I am relatively new to django. I have defined my db schema and validated it with no errors (manage.py validate reports 0 errors found).
Yet when I run ./manage.py syncdb
I get the following stack trace:
Creating table demo_foobar_one
Creating table demo_foobar_two
<snip>...</snip>
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 218, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 347, in handle
return self.handle_noargs(**options)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/commands/syncdb.py", line 103, in handle_noargs
emit_post_sync_signal(created_models, verbosity, interactive, db)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/sql.py", line 185, in emit_post_sync_signal
interactive=interactive, db=db)
File "/usr/local/lib/python2.6/dist-packages/django/dispatch/dispatcher.py", line 162, in send
response = receiver(signal=self, sender=sender, **named)
File "/usr/local/lib/python2.6/dist-packages/django/contrib/auth/management/__init__.py", line 28, in create_permissions
defaults={'name': name, 'content_type': ctype})
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 135, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 373, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/base.py", line 435, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/base.py", line 528, in save_base
result = manager._insert(values, return_id=update_pk, using=using)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 195, in _insert
return insert_query(self.model, values, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 1479, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 783, in execute_sql
cursor = super(SQLInsertCompiler, self).execute_sql(None)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 727, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.6/dist-packages/django/db/backends/util.py", line 15, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.6/dist-packages/django/db/backends/mysql/base.py", line 86, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/cursors.py", line 176, in execute
if not self._defer_warnings: self._warning_check()
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/cursors.py", line 92, in _warning_check
warn(w[-1], self.Warning, 3)
_mysql_exceptions.Warning: Data truncated for column 'name' at row 1
I have checked (and double checked) my table schema. All name field are CharField type with maximum length = 64. The backend db I am using is MySQL, so I am sure that indexes can be created for strings of length 64.
What could be causing this error (I suspect it is a misleading error message - even though its coming from the db itself)...
The traceback is happening during the creation of a django.contrib.auth.Permission: some of these get created for your models automatically as part of syncdb.
Permission.name has max_length=50, so you probably have an application and/or model class with a really long name?
Try the following query in manage.py dbshell:
SELECT * FROM auth_permission WHERE LENGTH(name) = 50;
If you cannot change your model name, then you can fix this problem by reducing the length of the generated Permission.name by specifying verbose_name in the model Meta class (see here for more details):
class MyVeryLongModelNameThatIsBreakingMyMigration(models.Model):
class Meta:
verbose_name = 'my long model'
Update
There's an open (as of 2013) Django ticket to fix this:
#8162 (Increase Permission.name max_length)
As Piet Delport noted, the problem is that your model name is too long.
You're certainly going to have to shorten your model name, and then clean up your database. How you do that depends upon where you are in the development process.
If this is a brand new application, with a dedicated database, and no actual data, the simple answer is to drop and recreate the database, and then re-run python manage.py syncdb.
If you have other tables in the database that need to be left alone, but the tables for your Django have no 'real' data, and can thus be dropped without damage, then you can use manage.py sqlclear to generate SQL DDL to drop all of the Django-generated tables, constraints, and indexes.
Do the following:
apps="auth contenttypes sessions sites messages admin <myapp1> <myapp2>"
python manage.py sqlclear ${apps} > clear.sql
You can feed the generated script to mysql or python manage.py dbshell. Once that's done, you can re-run python manage.py syncdb.
If you have actual data in your database tables that can't be dropped or deleted: Slap yourself and repeat 100 times "I will never do development against a production database again. I will always back up my databases before changing them." Now you're going to have to hand-code SQL to change the affected table name, as well as anything else that references it and any references in the auth_permissions table and any other Django system tables. Your actual steps will depend entirely upon the state of your database and tables.
I also got error like this one using postgresql django 1.2, but the problem was not the length, but using ugettext_lazy for translating the names. ('can_purge', _("Can purge")) is evidently unacceptable, since the name is stored in the database as text