Related
Django version 1.11, sqlite3 version 3.11.
I'm using WAL mode and a long timeout:
from django.apps import AppConfig
from django.db.backends.signals import connection_created
class SQLite3Config(AppConfig):
name = 'sqlite3_config'
def ready(self):
connection_created.connect(configure_sqlite)
# noinspection PyUnusedLocal
def configure_sqlite(sender, connection, **_):
if connection.vendor == 'sqlite':
cursor = connection.cursor()
cursor.execute('PRAGMA journal_mode=WAL;')
cursor.execute('PRAGMA busy_timeout=5000;')
I want to retain sqlite3 and not move to mysql or postgres because the application is small and is installed by users on multiple servers.
I believe WAL should allow "concurrent" writes by serializing them. The "Database is locked" problem was observed when small bursts (half a dozen or so) were received together.
I can reproduce the problem in the shell with threads. The django model method simply sets a flag and saves the model:
def activate(self):
self.activate = True
self.save()
When I use threads I find it fails if I launch a few threads that attempt it at the same time. There is no wait so the timeout is not involved. The error occurs before the 5 second busy timeout has elapsed (in less than two seconds):
In [2]: [NGThread(notifier_group.id).start() for notifier_group in NotifierGroup.objects.all()[:2]]
Out[2]: [None, None]
In [3]: [NGThread(notifier_group.id).start() for notifier_group in NotifierGroup.objects.all()[:3]]
Out[3]: [None, None, None]
In [4]: [NGThread(notifier_group.id).start() for notifier_group in NotifierGroup.objects.all()[:4]]
Out[4]: [None, None, None, None]
In [5]: Exception in thread Thread-97:
Traceback (most recent call last):
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/home/paul/wk/cliosoft/sosadmin/scratch.py", line 41, in run
toggle_active(notifier_group)
File "/home/paul/wk/cliosoft/sosadmin/scratch.py", line 30, in toggle_active
model.activate()
File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 67, in activate
self.save()
File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 33, in save
self.verify()
File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 46, in verify
self.create_notifier(base_spec, model_set, group_event_condition)
File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 57, in create_notifier
notifier.users = self.users.all()
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 534, in __set__
manager.set(value)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1004, in set
self.add(*new_objs)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 931, in add
self._add_items(self.source_field_name, self.target_field_name, *objs)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1100, in _add_is
for obj_id in new_ids
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 442, in bulk_create
ids = self._batched_insert(objs_without_pk, fields, batch_size)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 1083, in _batched_insert
self._insert(item, fields=fields, using=self.db)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 1060, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1099, in execute_sql
cursor.execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 80, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked
Exception in thread Thread-98:
Traceback (most recent call last):
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/home/paul/wk/cliosoft/sosadmin/scratch.py", line 41, in run
toggle_active(notifier_group)
File "/home/paul/wk/cliosoft/sosadmin/scratch.py", line 28, in toggle_active
model.deactivate()
File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 72, in deactivate
self.save()
File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 33, in save
self.verify()
File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 46, in verify
self.create_notifier(base_spec, model_set, group_event_condition)
File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 57, in create_notifier
notifier.users = self.users.all()
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 534, in __set__
manager.set(value)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1004, in set
self.add(*new_objs)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 931, in add
self._add_items(self.source_field_name, self.target_field_name, *objs)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1100, in _add_is
for obj_id in new_ids
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 442, in bulk_create
ids = self._batched_insert(objs_without_pk, fields, batch_size)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 1083, in _batched_insert
self._insert(item, fields=fields, using=self.db)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 1060, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1099, in execute_sql
cursor.execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 80, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked
I can't see anything in the release notes but I've upgraded to Django 2.0 and this problem has now disappeared. With the WAL set up in sqlite3 its all working very well now.
As the system has scaled, this error has re-occurred. I'm required to use sqlite3 for reasons outside the scope of this problem.
The timeout option in Django seems to be ignored. I tried implementing it myself at the db cursor level with a wrapper around sqlite and had the same problem (at that level you can't back out far enough to abandon the transaction and try a new connection).
I'd have liked a context manager e.g. with db_locked_retries():... but it is not possible to iterate within the with context so I created a db_retry function thus:
import time
import logging
import random
from django.conf import settings
import django.db
logger = logging.getLogger('django.db.backends')
def db_retry(fn, timeout=None):
"""Call fn with no arguments. If OperationalError exception, make retries until timeout has passed"""
timeout = timeout or settings.DATABASES['default'].get('OPTIONS', dict()).get('timeout', 5)
now = time.time()
give_up_time = now + timeout
retries = 0
while now < give_up_time:
now = time.time()
try:
result = fn()
if retries:
logger.warning(f'db_retry: Succeeded after {retries} retries')
return result
except django.db.OperationalError as exception:
msg = str(exception)
if 'locked' in msg: # pragma: no cover
retries += 1
wait_time = random.uniform(1, timeout / 10)
logger.warning(f'db_retry: {msg}: Retrying after {wait_time} seconds')
django.db.close_old_connections()
time.sleep(wait_time)
else: # pragma: no cover
logger.exception(f'db_retry: {msg}: Giving up')
raise
As it is very easy to define the function to pass to this function within the model methods etc. this addresses the problem and successfully uses the timeout to retry the access. This is currently handling some hundreds of concurrent uses and retries once every thirty minutes or so in my particular use case.
I've only needed to 'wrap' model code that potentially makes several writes so far.
I am trying to migrate the database of one my django projects from sqlite to mysql. I first dumped the whole thing with ./manage.py dumpdata > dump.json and prepared the database with ./manage.py migrate and deleted any created data in the tables (This was necessary, as the dump holds all the data).
When I wanted to import the data into the new database with ./manage.py loaddata, there were a lot of errors I was able to resolve, but I can't find the source of this error:
Processed 330984 object(s).Traceback (most recent call last):
File "/app/manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 355, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.5/site-packages/django/core/management/commands/loaddata.py", line 69, in handle
self.loaddata(fixture_labels)
File "/usr/local/lib/python3.5/site-packages/django/core/management/commands/loaddata.py", line 109, in loaddata
self.load_label(fixture_label)
File "/usr/local/lib/python3.5/site-packages/django/core/management/commands/loaddata.py", line 175, in load_label
obj.save(using=self.using)
File "/usr/local/lib/python3.5/site-packages/django/core/serializers/base.py", line 205, in save
models.Model.save_base(self.object, using=using, raw=True, **kwargs)
File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 837, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 904, in _save_table
forced_update)
File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 954, in _do_update
return filtered._update(values) > 0
File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 664, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1199, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
raise original_exception
File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 884, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 60, in execute
self.db.validate_no_broken_transaction()
File "/usr/local/lib/python3.5/site-packages/django/db/backends/base/base.py", line 448, in validate_no_broken_transaction
"An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: Problem installing fixture '/path/to/django/dump.json': Could not load auth.User(pk=1): An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
I already tried to remove all signal receivers, so that none of my own code is run when executing loaddata.
Has anyone else experienced similar behaviour with django's loaddata and managed to get it working?
Context:
python v3.5
django v1.11.4
only stdlib django model fields used
This is more of a workaround than a solution, but it did solve my problem.
After further investigation I realized that the auth.user model somehow triggered the query and broke the import.
The solution was to write a simple script to split the dump into multiple parts (I did not have access to the database anymore) and then import them in the correct order: contenttypes, auth, and the rest.
You can find the script here.
This is how I split up the dump:
./investigate-django-dump.py dump.json extract contenttypes. > contenttypes.json
./investigate-django-dump.py dump.json extract auth. > auth.json
./investigate-django-dump.py dump.json extractnot auth.,contenttypes. > data.json
I'm running into a strange issue when resetting my database. A multiple choice field in a form is throwing exception
Unhandled exception in thread started by .wrapper at 0x7f59df160510>
Traceback (most recent call last):
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 337, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: app1_semester
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
......
File "/home/abdullah/PycharmProjects/langs-dept/project1/app1/urls.py", line 3, in
from . import views
File "/home/abdullah/PycharmProjects/langs-dept/project1/app1/views.py", line 10, in
from . import models, forms
File "/home/abdullah/PycharmProjects/langs-dept/project1/app1/forms.py", line 11, in
class AddAssignmentForm(forms.Form):
File "/home/abdullah/PycharmProjects/langs-dept/project1/app1/forms.py", line 14, in AddAssignmentForm
queryset=models.AssignmentType.get_assignment_types(),
File "/home/abdullah/PycharmProjects/langs-dept/project1/app1/models.py", line 180, in get_assignment_types
return AssignmentType.objects.filter(semester=Semester.get_current_semester())
File "/home/abdullah/PycharmProjects/langs-dept/project1/app1/models.py", line 157, in get_current_semester
return semester.first() if semester else None
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/models/query.py", line 260, in __bool__
self._fetch_all()
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/models/query.py", line 1087, in _fetch_all
self._result_cache = list(self.iterator())
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/models/query.py", line 54, in __iter__
results = compiler.execute_sql()
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
cursor.execute(sql, params)
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/home/abdullah/PycharmVirtualEnvs/venv/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 337, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: app1_semester
If I comment below piece of code in forms.py, everything is working fine.
class AddAssignmentForm(forms.Form):
assignments = forms.ModelMultipleChoiceField(widget=forms.CheckboxSelectMultiple(),queryset=models.AssignmentType.get_assignment_types(),required=False)
The method get_assignment_types() is a static method returning some list of values
models.py
#staticmethod
def get_assignment_types():
return AssignmentType.objects.filter()
I can comment that line and do python manage.py makemigrations, but I wonder what is wrong ? Can someone shed some light?
Error is in this line
AssignmentType.objects.filter(semester=Semester.get_current_semester())
You have to do proper migrations for model Semester
UPDATE for close - I think I go through it and it was a bit of stupidity on my behalf.
python manage.py inspectdb > models.py
I was forgetting to pipe the output to a file!
I've dumped that into my app/models.py and now I'm just trying to get it running from there but going further.
I'm trying to get django running against an SQL Server 2008 R2 server.
I've figured out the FreeTDS/unixODBC connections. I can successfully do an isql and tsql query against the database.
I've been able to do a connection directly in python like so -
import pyodbc
cnxn = pyodbc.connect('DRIVER=FreeTDS;SERVER=servername;DATABASE=dbname;UID=user;PWD=pass;TDS_Version=8.0;')
cursor = cnxn.cursor()
result = cursor.execute("select name from tablea where name = 'hello'")
print result
If I do an inspectdb in django shell it's fine. It returns 70+ tables and their layout perfectly ok.
once I try to do a syndb it's spilling back the following error -
Creating tables ...
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 415, in handle
return self.handle_noargs(**options)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/commands/syncdb.py", line 112, in handle_noargs
emit_post_sync_signal(created_models, verbosity, interactive, db)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/sql.py", line 216, in emit_post_sync_signal
interactive=interactive, db=db)
File "/usr/local/lib/python2.6/dist-packages/django/dispatch/dispatcher.py", line 185, in send
response = receiver(signal=self, sender=sender, **named)
File "/usr/local/lib/python2.6/dist-packages/django/contrib/auth/management/__init__.py", line 82, in create_permissions
ctype = ContentType.objects.db_manager(db).get_for_model(klass)
File "/usr/local/lib/python2.6/dist-packages/django/contrib/contenttypes/models.py", line 47, in get_for_model
defaults = {'name': smart_text(opts.verbose_name_raw)},
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 154, in get_or_create
return self.get_queryset().get_or_create(**kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 373, in get_or_create
return self.get(**lookup), False
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 301, in get
num = len(clone)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 77, in __len__
self._fetch_all()
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 854, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 771, in execute_sql
sql, params = self.as_sql()
File "/usr/local/lib/python2.6/dist-packages/sql_server/pyodbc/compiler.py", line 48, in as_sql
self._get_ordering(out_cols, supports_offset_clause or not do_offset)
File "/usr/local/lib/python2.6/dist-packages/sql_server/pyodbc/compiler.py", line 154, in _get_ordering
ordering, ordering_group_by = self.get_ordering()
ValueError: too many values to unpack
similarily if I try to do a query in django like so -
from myapp.models import tablea
tablea.objects.filter(id=5)
It fails with the following error message -
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 71, in __repr__
data = list(self[:REPR_OUTPUT_SIZE + 1])
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 96, in __iter__
self._fetch_all()
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 854, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 771, in execute_sql
sql, params = self.as_sql()
File "/usr/local/lib/python2.6/dist-packages/sql_server/pyodbc/compiler.py", line 48, in as_sql
self._get_ordering(out_cols, supports_offset_clause or not do_offset)
File "/usr/local/lib/python2.6/dist-packages/sql_server/pyodbc/compiler.py", line 154, in _get_ordering
ordering, ordering_group_by = self.get_ordering()
ValueError: too many values to unpack
I'm not convinced these errors are related to the other thread found here -
ValueError: Too many values to unpack Django
But I'm no expert so I could be wrong. I know I did try to change my password and it won't let me connect similar error message -
>>> u = User.objects.get(username="user")
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 151, in get
return self.get_queryset().get(*args, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 301, in get
num = len(clone)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 77, in __len__
self._fetch_all()
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 854, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 771, in execute_sql
sql, params = self.as_sql()
File "/usr/local/lib/python2.6/dist-packages/sql_server/pyodbc/compiler.py", line 48, in as_sql
self._get_ordering(out_cols, supports_offset_clause or not do_offset)
File "/usr/local/lib/python2.6/dist-packages/sql_server/pyodbc/compiler.py", line 154, in _get_ordering
ordering, ordering_group_by = self.get_ordering()
ValueError: too many values to unpack
edit to add - when doing a query my model definition is as follows -
from django.db import models
# Create your models here.
class tablea(models.Model):
tablea_id = models.IntegerField()
tablea_name = models.CharField(max_length=32)
tablea_duration = models.IntegerField()
class Meta:
db_table = u'tablea'
def __unicode__(self):
return self.tablea_name, tablea_duration
edit to add that I've done a complete fresh reinstall. Wiped the VM and brought it back to life. Here's a breakdown of the steps done
apt-get install gcc g++ make python-dev
download unixODBC
tar xzvf unixODBC-2.3.2.tar.gz
./configure
make
makeinstall
download FreeTDS
tar xzvf freetds-stable.tgz
./configure --with-tdsver=8.0 --with-unixodbc=usr/local
make
make install
ldconfig -v
odbc.ini
[mssql]
Description = MSSQL Server
Driver = freetds
Database = database
Server = server
Port = 1433
TDS_Version = 8.0
odbcinst.ini
[freetds]
Description = MS SQL database access with Free TDS
Driver = /usr/local/lib/libtdsodbc.so
Setup = /usr/local/lib/libtdsS.so
UsageCount = 1
freetds.conf
[mssql]
host = server
port = 1433
tds version = 8.0
tsql -S mssql -U user -P password works
isql -v mssql user password works (for some reason simple isql -v mssql does not)
apt-get install python-setuptools
easy_install pip
pip install pyodbc
pip install Django==1.6
pip install django-pyodbc-azure
settings.py
DATABASES = {
'default': {
'ENGINE': 'sql_server.pyodbc',
'HOST': "server\instance",
'USER': "username",
'PASSWORD': "password",
'NAME': "database",
'OPTIONS': {
'host_is_server': True,
'driver': 'FreeTDS',
'dsn': 'MSSQL',
},
}
}
It's a version incompatibility issue.
Django 1.6 changed the kind of value returned by django.db.models.sql.compiler.SQLCompiler.get_ordering. It returns a 3-element tuple, but pyodbc unpacks just two elements.
If you look at Django 1.5 version you see that it used to return a 2-element tuple as pyodbc expects.
I am relatively new to django. I have defined my db schema and validated it with no errors (manage.py validate reports 0 errors found).
Yet when I run ./manage.py syncdb
I get the following stack trace:
Creating table demo_foobar_one
Creating table demo_foobar_two
<snip>...</snip>
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_manager(settings)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/usr/local/lib/python2.6/dist-packages/django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 218, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/base.py", line 347, in handle
return self.handle_noargs(**options)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/commands/syncdb.py", line 103, in handle_noargs
emit_post_sync_signal(created_models, verbosity, interactive, db)
File "/usr/local/lib/python2.6/dist-packages/django/core/management/sql.py", line 185, in emit_post_sync_signal
interactive=interactive, db=db)
File "/usr/local/lib/python2.6/dist-packages/django/dispatch/dispatcher.py", line 162, in send
response = receiver(signal=self, sender=sender, **named)
File "/usr/local/lib/python2.6/dist-packages/django/contrib/auth/management/__init__.py", line 28, in create_permissions
defaults={'name': name, 'content_type': ctype})
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 135, in get_or_create
return self.get_query_set().get_or_create(**kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 373, in get_or_create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/base.py", line 435, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/base.py", line 528, in save_base
result = manager._insert(values, return_id=update_pk, using=using)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 195, in _insert
return insert_query(self.model, values, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 1479, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 783, in execute_sql
cursor = super(SQLInsertCompiler, self).execute_sql(None)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 727, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.6/dist-packages/django/db/backends/util.py", line 15, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.6/dist-packages/django/db/backends/mysql/base.py", line 86, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/cursors.py", line 176, in execute
if not self._defer_warnings: self._warning_check()
File "/usr/local/lib/python2.6/dist-packages/MySQL_python-1.2.3-py2.6-linux-i686.egg/MySQLdb/cursors.py", line 92, in _warning_check
warn(w[-1], self.Warning, 3)
_mysql_exceptions.Warning: Data truncated for column 'name' at row 1
I have checked (and double checked) my table schema. All name field are CharField type with maximum length = 64. The backend db I am using is MySQL, so I am sure that indexes can be created for strings of length 64.
What could be causing this error (I suspect it is a misleading error message - even though its coming from the db itself)...
The traceback is happening during the creation of a django.contrib.auth.Permission: some of these get created for your models automatically as part of syncdb.
Permission.name has max_length=50, so you probably have an application and/or model class with a really long name?
Try the following query in manage.py dbshell:
SELECT * FROM auth_permission WHERE LENGTH(name) = 50;
If you cannot change your model name, then you can fix this problem by reducing the length of the generated Permission.name by specifying verbose_name in the model Meta class (see here for more details):
class MyVeryLongModelNameThatIsBreakingMyMigration(models.Model):
class Meta:
verbose_name = 'my long model'
Update
There's an open (as of 2013) Django ticket to fix this:
#8162 (Increase Permission.name max_length)
As Piet Delport noted, the problem is that your model name is too long.
You're certainly going to have to shorten your model name, and then clean up your database. How you do that depends upon where you are in the development process.
If this is a brand new application, with a dedicated database, and no actual data, the simple answer is to drop and recreate the database, and then re-run python manage.py syncdb.
If you have other tables in the database that need to be left alone, but the tables for your Django have no 'real' data, and can thus be dropped without damage, then you can use manage.py sqlclear to generate SQL DDL to drop all of the Django-generated tables, constraints, and indexes.
Do the following:
apps="auth contenttypes sessions sites messages admin <myapp1> <myapp2>"
python manage.py sqlclear ${apps} > clear.sql
You can feed the generated script to mysql or python manage.py dbshell. Once that's done, you can re-run python manage.py syncdb.
If you have actual data in your database tables that can't be dropped or deleted: Slap yourself and repeat 100 times "I will never do development against a production database again. I will always back up my databases before changing them." Now you're going to have to hand-code SQL to change the affected table name, as well as anything else that references it and any references in the auth_permissions table and any other Django system tables. Your actual steps will depend entirely upon the state of your database and tables.
I also got error like this one using postgresql django 1.2, but the problem was not the length, but using ugettext_lazy for translating the names. ('can_purge', _("Can purge")) is evidently unacceptable, since the name is stored in the database as text