Django test table is not created [duplicate] - django

When I run my test dealing with my Customer model, I get the following error:
DatabaseError: (1146, "Table 'test_mcif2.customer' doesn't exist")
I'm not entirely surprised because I have my Django project connected to a "legacy" database. Since my tables weren't created "the Django way," it's not shocking that Django wouldn't be able to talk to them without some finagling. Here's my model:
from django.db import models
from django.db import connection, transaction
from mcif.models.mcif_model import McifModel
class Customer(McifModel):
class Meta:
db_table = u'customer'
app_name = 'mcif'
id = models.BigIntegerField(primary_key=True)
customer_number = models.CharField(unique=True, max_length=255)
social_security_number = models.CharField(unique=True, max_length=33)
name = models.CharField(unique=True, max_length=255)
phone = models.CharField(unique=True, max_length=255)
deceased = models.IntegerField(unique=True, null=True, blank=True)
do_not_mail = models.IntegerField(null=True, blank=True)
created_at = models.DateTimeField()
updated_at = models.DateTimeField()
def distinguishing_column_names(self):
return ['name', 'customer_number', 'social_security_number', 'phone']
Any idea why exactly this isn't working?
Edit: Here's McifModel:
from django.db import models
from django.db import connection, transaction
class McifModel(models.Model):
class Meta:
abstract = True
def upsert(self):
cursor = connection.cursor()
cursor.execute(self.upsert_sql())
transaction.commit_unless_managed()
return self
def value_list(self):
return ','.join(map(lambda column_name: "'{c}'".format(c=getattr(self, column_name)), self.distinguishing_column_names()))
def upsert_sql(self):
column_names = ','.join(self.distinguishing_column_names())
return "INSERT IGNORE INTO {t} ({c}) VALUES ({v})".format(t=self._meta.db_table, c=column_names, v=self.value_list())
#classmethod
def save_from_row(cls, row):
object = cls()
map(lambda column_name: setattr(object, column_name, row.value(object._meta.db_table, column_name)), object.distinguishing_column_names())
return object.upsert()
Edit: I took tarequeh's advice and put the contents of the Caktus file in mcif/utils.py. I also set TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'. If I go on the console I can verify that Customer is unmanaged:
>>> [m for m in get_models() if not m._meta.managed]
[<class 'mcif.models.customer.Customer'>]
However, my test still complains that the table doesn't exist. What am I missing?
Here's my settings.py:
# Django settings for mcifdjango project.
DEBUG = True
TEMPLATE_DEBUG = DEBUG
ADMINS = (
('Jason Swett', 'jason.swett#gmail.com'),
)
MANAGERS = ADMINS
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'xxxxx', # Or path to database file if using sqlite3.
'USER': 'xxxxx', # Not used with sqlite3.
'PASSWORD': 'xxxxx', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
}
# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'America/Chicago'
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'
SITE_ID = 1
# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True
# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True
# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = ''
# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash if there is a path component (optional in other cases).
# Examples: "http://media.lawrence.com", "http://example.com/media/"
MEDIA_URL = ''
# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = '/media/'
# Make this unique, and don't share it with anybody.
SECRET_KEY = '#7+qm%hqfe+z8ul5#x_i&sqmu!n=4sa0&i0_#)m99*w$fbk3%#'
# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
# 'django.template.loaders.eggs.Loader',
)
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
)
ROOT_URLCONF = 'mcifdjango.urls'
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
)
INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.admin',
'django_extensions',
'mcif',
# Uncomment the next line to enable the admin:
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
)
TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'
import os
ROOTDIR = os.path.abspath(os.path.dirname(__file__))
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
ROOTDIR + '/mcif/templates',
)
Edit 2:
Here's my Customer class now:
from django.db import models
from django.db import connection, transaction
from mcif.models.mcif_model import McifModel
class Customer(McifModel):
class Meta:
db_table = u'customer'
managed = False
id = models.BigIntegerField(primary_key=True)
customer_number = models.CharField(unique=True, max_length=255)
social_security_number = models.CharField(unique=True, max_length=33)
name = models.CharField(unique=True, max_length=255)
phone = models.CharField(unique=True, max_length=255)
deceased = models.IntegerField(unique=True, null=True, blank=True)
do_not_mail = models.IntegerField(null=True, blank=True)
created_at = models.DateTimeField()
updated_at = models.DateTimeField()
def distinguishing_column_names(self):
return ['name', 'customer_number', 'social_security_number', 'phone']
Here's what I get when I run the test:
$ ./manage.py test mcif.CustomerUpsertTest
Creating test database 'default'...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table django_admin_log
Installing index for auth.Permission model
Installing index for auth.Group_permissions model
Installing index for auth.User_user_permissions model
Installing index for auth.User_groups model
Installing index for auth.Message model
Installing index for admin.LogEntry model
No fixtures found.
E
======================================================================
ERROR: test_upsert (mcif.tests.customer_upsert_test.CustomerUpsertTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jason/projects/mcifdjango/mcif/tests/customer_upsert_test.py", line 9, in test_upsert
customer.upsert()
File "/home/jason/projects/mcifdjango/mcif/models/mcif_model.py", line 11, in upsert
cursor.execute(self.upsert_sql())
File "/usr/lib/pymodules/python2.6/django/db/backends/mysql/base.py", line 86, in execute
return self.cursor.execute(query, args)
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
DatabaseError: (1146, "Table 'test_mcif_django.customer' doesn't exist")
----------------------------------------------------------------------
Ran 1 test in 3.724s
FAILED (errors=1)
Destroying test database 'default'...

Since you're using a legacy database, you are probably not adding the app name to INSTALLED_APPS. If an app is not included in INSTALLED_APPS, the tables for the apps' models will not get created on syncdb. This works for you in production since you already have a table, but not in test environment.
You can adopt any of the following:
The supermonkeypatch way: Take out app_name from Customer class Meta, put the model in a models.py file inside a python module name mcif, and add mcif to INSTALLED_APPS - just for the sake of testing
The nicer way: Extend DjangoTestSuiteRunner and override setup_test_environment to call super and then create your legacy table manually in the test DB.
The nicest way: Put your model in properly named app module. Remove app_name from model Meta but add managed=False docs. Include app name in INSTALLED_APPS. Now django will not create table for that model. Then use this nice snippet the Caktus group folks have compiled to run your tests.
Cheers!
Edit - How to use the overridden DjangoTestSuiteRunner
You will need at least Django 1.2 for this.
Copy the code from here. Put it in utils.py inside the mcif app.
Add/edit the following in settings.py:
TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'
Now when you run tests, all unmanaged tables will be treated as managed table only for the duration of the test. So the tables will be created prior to running tests.
Notice this part of the code, thats where the magic happens.
self.unmanaged_models = [m for m in get_models() if not m._meta.managed]
for m in self.unmanaged_models:
m._meta.managed = True
2nd Edit: Possible Gotchas
Make sure of the following:
The DB user has privilege to create databases and not only tables because django will try to create a test database
The test cases extend django.test.TransactionTestCase, since you have transactional behavior
If none of the above applies, put a pdb in ManagedModelTestRunner's setup_test_environment just to make sure the code is being reached. Because if that code is reached, the table should get created
3rd Edit: Debugging
Inside mcif.utils.ManagedModelTestRunner replace setup_test_environment function with the following and let me know if the output of your test changes:
def setup_test_environment(self, *args, **kwargs):
print "Loading ManagedModelTestRunner"
from django.db.models.loading import get_models
self.unmanaged_models = [m for m in get_models()
if not m._meta.managed]
for m in self.unmanaged_models:
print "Modifying model %s to be managed for testing" % m
m._meta.managed = True
super(ManagedModelTestRunner, self).setup_test_environment(*args, **kwargs)

The solutions presented by tarequeh worked for me after overriding DATABASE_ROUTERS.
I am using routers in order to prevent writes on the legacy database. In order to get around this I created a test_settings file with the following contents:
from settings import *
DEBUG = True
TEST_RUNNER = 'legacy.utils.ManagedModelTestRunner'
DATABASE_ROUTERS = []
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(HERE, 'test.db'),
},
}
Then when running tests:
python manage.py test [app_name] --settings=test_settings

There's not enough info above to answer your first question. However, once you get that issue resolved you'll probably want to install django-extensions for the following reason: It has an incredibly useful sqldiff command that will inform you if there's a mismatch between the legacy database and your application model.

Here's a more up-to-date solution, that also works with current versions of Django (I tested it on Django 3.2.11):
https://medium.com/an-idea/testing-with-the-legacy-database-in-django-3be84786daba
Also, in case you want to furthermore populate the Django test-database with your legacy database's data:
Check out fixtures

Related

How to configure sqlite of django project with pythonanywhere

I have deployed my project to pythonanywhere.
It is working locally.
But with pythonanywhere I am getting no such table exception.
I have configured sqllite as in this link
Just mentioned to generate the sqlite file using runmigrations.
I have changed the settings.py to use os.path.join at that Database section also but still same issue.
Exception Type: ProgrammingError
Exception Value:
(1146, "Table 'todo_todo' doesn't exist")
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
tried with os.path.join also but same error.
my models.py
from django.db import models
from django.contrib.auth.models import User
# Create your models here.
class Todo(models.Model):
title = models.CharField(max_length=100)
memo = models.TextField(blank=True)
created=models.DateTimeField(auto_now_add=True)
datecompleted=models.DateTimeField(null=True, blank=True)
important=models.BooleanField(default=False)
user = models.ForeignKey(User,on_delete=models.CASCADE)
def __str__(self):
return self.title
I migrated individual apps also.
python manage.py makemigrations appname

How to save extra fields on registration using custom user model in DRF + django-rest-auth

Using Django REST Framework (DRF), with django-rest-auth, I created a custom user model with one extra field. My aim is to use the django-rest-auth registration endpoint to register a new user in one request, and thus sending all the data to create a new user, including the data for the extra field.
I am using AbstractUser, since it seems recommended for beginners, where more advanced developers could use AbstractBaseUser. This is also why the following SO answers looks too complicated for what I want to achieve: link here.
I know this question has been asked multiple times, but the answers are not exactly what I am looking for. For a beginner like me this is complicated stuff.
So, my question is, can anyone explain how to achieve what I want?
I am using:
Django 2.1.4
django-allauth 0.38.0
django-rest-auth 0.9.3
djangorestframework 3.9.0
Here's the code that I have up until this point:
Used this tutorial to get to this code
settings.py:
import os
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.1/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '!gxred^*penrx*qlb=#p)p(vb!&6t78z4n!poz=zj+a0_9#sw1'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = []
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'rest_framework.authtoken',
'rest_auth',
'django.contrib.sites',
'allauth',
'allauth.account',
'rest_auth.registration',
'users',
]
SITE_ID = 1
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'DRF_custom_user.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'DRF_custom_user.wsgi.application'
# Database
# https://docs.djangoproject.com/en/2.1/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
# Password validation
# https://docs.djangoproject.com/en/2.1/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/2.1/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.1/howto/static-files/
STATIC_URL = '/static/'
AUTH_USER_MODEL = 'users.CustomUser'
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
users.models.py:
from django.contrib.auth.models import AbstractUser
from django.db import models
class CustomUser(AbstractUser):
preferred_locale = models.CharField(blank=True, null=True, max_length=2)
users.admin.py:
from django.contrib import admin
from django.contrib.auth import get_user_model
from django.contrib.auth.admin import UserAdmin
from .forms import CustomUserCreationForm, CustomUserChangeForm
from .models import CustomUser
class CustomUserAdmin(UserAdmin):
add_form = CustomUserCreationForm
form = CustomUserChangeForm
model = CustomUser
list_display = ['email', 'preferred_locale']
admin.site.register(CustomUser, CustomUserAdmin)
users.forms.py:
from django import forms
from django.contrib.auth.forms import UserCreationForm, UserChangeForm
from .models import CustomUser
class CustomUserCreationForm(UserCreationForm):
class Meta(UserCreationForm):
model = CustomUser
fields = ('email', )
class CustomUserChangeForm(UserChangeForm):
class Meta:
model = CustomUser
fields = UserChangeForm.Meta.fields
I went looking for an answer myself. Spend some time digging in the source code. I realize this solution may be missing the actual validation for the extra fields added to the custom user model, but I will look into that later.
What I have written below I wrote with a potential blog post in mind.
I am going to assume you know how to set up a DRF project and install the above packages. The django-rest-auth documentation is clear on how to install that package (https://django-rest-auth.readthedocs.io/en/latest/index.html), make sure to also follow the steps to install the part of django-rest-auth for user registration.
Create a new app ‘users’
This app will hold my custom code for implementing the custom user model. I also install it in the Django main settings file:
settings.py:
INSTALLED_APPS = [
...
'users',
]
Create my custom user model
Notice that I just added one custom field, but you can add whatever fields you want ofcourse.
users.models.py:
from django.contrib.auth.models import AbstractUser
from django.db import models
class CustomUser(AbstractUser):
preferred_locale = models.CharField(max_length=2, blank=True, null=True)
Tell django to use the CustomUser model
settings.py:
…
AUTH_USER_MODEL = 'users.CustomUser'
Register Custom user model at Django admin
users.admin.py:
from django.contrib import admin
from .models import CustomUser
admin.site.register(CustomUser)
Make migrations and run them
This is the first time I do this for this project.
In command line:
python manage.py makemigrations users
python manage.py migrate
Registering new users with extra fields
If you start the Django development server now, you’ll see in the admin that you can see the custom user model, with the extra fields.
But when you go to ‘http://127.0.0.1:8000/rest-auth/registration/’ you don’t see the extra fields yet.
In the process of user registration two important classes are used, namely:
a serializer ‘rest_auth.registration.RegisterSerializer’
an adapter ‘allauth.account.adapter.DefaultAccountAdapter’
We’ll make a custom version of both of these, inheriting all the functionality of it’s parent class.
Creating a custom RegisterSerializer
Create a new file ‘serializers.py’ in the users app/folder.
users.serializers.py:
from rest_framework import serializers
from allauth.account.adapter import get_adapter
from allauth.account.utils import setup_user_email
from rest_auth.registration.serializers import RegisterSerializer
class CustomRegisterSerializer(RegisterSerializer):
preferred_locale = serializers.CharField(
required=False,
max_length=2,
)
def get_cleaned_data(self):
data_dict = super().get_cleaned_data()
data_dict['preferred_locale'] = self.validated_data.get('preferred_locale', '')
return data_dict
Here I create a new field for each extra field on the custom user model. So in my case a added this:
preferred_locale = serializers.CharField(
required=False,
max_length=2,
)
Also, the get_cleaned_data method should return a dict which contains all the data for the fields that you want to have saved when registering a new user.
This is what the original method (of the default RegisterSerializer looks like):
def get_cleaned_data(self):
return {
'username': self.validated_data.get('username', ''),
'password1': self.validated_data.get('password1', ''),
'email': self.validated_data.get('email', '')
}
As you can see it returns a dictionary, containing all the data for the new user. You want to add a keyval entry to this dictionary for each extra field you have added to your custom user model.
In my case, needing only to add data for the field ‘preferred_locale’, this is the resulting method:
def get_cleaned_data(self):
data_dict = super().get_cleaned_data()
data_dict['preferred_locale'] = self.validated_data.get('preferred_locale', '')
return data_dict
Tell django to use this new serializer
settings.py:
REST_AUTH_REGISTER_SERIALIZERS = {
'REGISTER_SERIALIZER': 'users.serializers.CustomRegisterSerializer',
}
Preventing errors
If you will try to register a new user, you might get the following error in the console where your development server is running:
ConnectionRefusedError: [Errno 111] Connection refused
Although a user is still created, you can fix this error by adding the following line to your settings.py file:
settings.py:
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
Another error that will occur when you delete a user is:
django.db.utils.OperationalError: no such table: allauth_socialaccount
To solve this, add this to your settings.py:
settings.py:
INSTALLED_APPS = [
...
'allauth.socialaccount',
]
After that, you should apply migrations before you can continue:
python manage.py migrate
Creating a custom AccountAdapter
After the above steps, going to ‘http://127.0.0.1:8000/rest-auth/registration/’ will show you the extra fields. But when you register a new user, and send the data for the extra fields, the extra field’s data is not saved.
The last thing we need to do to solve this is to create a custom AccountAdapter
In our users app/folder create a new file named ‘adapter.py’:
users.adapter.py:
from allauth.account.adapter import DefaultAccountAdapter
class CustomAccountAdapter(DefaultAccountAdapter):
def save_user(self, request, user, form, commit=False):
user = super().save_user(request, user, form, commit)
data = form.cleaned_data
user.preferred_locale = data.get('preferred_locale')
user.save()
return user
Here, if you have followed the above steps correctly, you can access the data of the extra added fields in the form.cleaned_data dictionary. This is the dictionary that is returned by the get_cleaned_data method from our custom RegisterSerializer.
In the save_user method above, we can then use this data and save it to the appropriate fields, like so:
user.preferred_locale = data.get('preferred_locale')
Tell Django to use this new adapters
settings.py:
ACCOUNT_ADAPTER = 'users.adapter.CustomAccountAdapter'
Now you can register your user, using the django-rest-auth registration endpoint '/rest-auth/registration/', and send the data for the extra fields you added. This will all be saved in one request.
Again, I realize that custom validation for each field needs to be added. But that's another topic that I will dive into later, and update the post when I found out how that works precisely.
Let's break your question. Please note that I am explaining the basics of Django REST Framework to you.
Overriding User's Model
[x] Step 1: You override User model: You did this. Alternative? Yes. Create a model that has OneToOneForeignKey pointed to User model.
[x] Step 2: Use this CustomUserModel. To do this, you need to set AUTH_USER_MODEL in settings.py Link to official documentation You did this.
[ ] Step 3: Create a UserManager to handle user's registration and other information. You haven't done this.
Registration from API
[ ] Create a serializer that clearly mentions all the required fields that you're expecting from an end user. You can even use serializer.ModelSerializer if there are no custom fields.
[ ] Handle explicit verifications in serializer itself. Use def validate(self, attrs) if required. Here is the official document link.
[ ] Finally, create a view and use APIView as you will want to register a user using UserManager that you created above.
I can also refer to you an app that I built myself. Here's the link: DRF-USER. I customized User model to some extent and followed the same process.
Hope this helps.

Django testing on separate database giving "OperationalError: no such table: auth_user"

My problem is that I wanted to put up a separate testing database, separate from my development database. App itself is almost identical to Django-Rest-Framework quick-start tutorial, just that I use LDAP backend. My development database uses MySQL. I have separate settings file for testing.
Full error traceback I put in here: http://dpaste.com/1W3TX1E, but interesting part is:
sqlite3.OperationalError: no such table: auth_user
Full test-settings are here: http://dpaste.com/1K7KHK4. My related settings are (pyldap and django-ldap-auth are missing from installed apps, they are installed manually with pip):
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'njord',
'permissions',
]
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ':memory:',
}
}
Right now the app does basically only authentication with LDAP. I think that is why I have an error - when I do tests I have nothing in my database - no users, no groups. If a user is authenticated against LDAP server, user is created in database alongside with all the groups he/she is in, so it happens on-demand.
What works: Development version of my app works well against external LDAP and also testing works with development database.
What does not work: Testing with sqllite3 in-memory database(I repeat, testing with development database and mocked ldap works).
The code I test with is here:
from django.test import TestCase, Client
import django
import ldap
from mockldap import MockLdap
class AuthenticationTests(TestCase):
"""
Set Up the structure ( Accounts, Groups ) and try authentication
"""
# Top level hierarchy
top = ('dc=ee', {'dc': ['ee']})
test = ('dc=test,dc=ee', {'dc': ['test']})
# Top level groups
people = ('ou=people,dc=test,dc=ee', {'ou': ['people'], 'objectClass': ['organizationalUnit', 'top']})
groups = ('ou=groups,dc=test,dc=ee', {'ou': ['groups'], 'objectClass': ['organizationalUnit', 'top']})
# Groups
admins = ('cn=admins,ou=groups,dc=test,dc=ee', {'cn': ['admins'], 'memberUid': ['admin'],
'objectClass': ['sambaGroupMapping', 'posixGroup', 'top'],
'gidNumber': ['1']})
group1 = ('cn=group1,ou=groups,dc=test,dc=ee', {'cn': ['group1'],
'memberUid': ['alice', 'bob'],
'objectClass': ['sambaGroupMapping', 'posixGroup', 'top']})
group2 = ('cn=group2,ou=groups,dc=test,dc=ee', {'cn': ['group2'], 'memberUid': ['admin', 'bob', 'karl'],
'objectClass': ['sambaGroupMapping', 'posixGroup', 'top']})
# Users
admin = ('uid=admin,ou=people,dc=test,dc=ee', {'uid': ['admin'], 'userPassword': ['ldaptest'], 'sn': ['Joe'],
'cn': ['Admin Joe'], 'mail': ['admin.joe#test.ee'],
'givenName': ['Admin'], 'objectClass':
['top', 'person', 'posixAccount', 'shadowAccount',
'inetOrgPerson', 'sambaSamAccount']})
alice = ('uid=alice,ou=people,dc=test,dc=ee', {'uid': ['alice'], 'userPassword': ['ldaptest'], 'sn': ['Cooper'],
'cn': ['Alice Cooper'], 'mail': ['alice.cooper#test.ee'],
'givenName': ['Alice'], 'objectClass':
['top', 'person', 'posixAccount', 'shadowAccount',
'inetOrgPerson', 'sambaSamAccount']})
bob = ('uid=bob,ou=people,dc=test,dc=ee', {'uid': ['bob'], 'userPassword': ['ldaptest'], 'sn': ['Marley'],
'cn': ['Bob Marley'], 'mail': ['bob.marley#test.ee'],
'givenName': ['Bob'], 'objectClass':
['top', 'person', 'posixAccount', 'shadowAccount',
'inetOrgPerson', 'sambaSamAccount']})
karl = ('uid=karl,ou=people,dc=test,dc=ee', {'uid': ['karl'], 'userPassword': ['ldaptest'], 'sn': ['Suur'],
'cn': ['Karl Suur'], 'mail': ['karl.suur#test.ee'],
'givenName': ['Karl'], 'objectClass':
['top', 'person', 'posixAccount', 'shadowAccount',
'inetOrgPerson', 'sambaSamAccount']})
# This is the content of our mock LDAP directory. It takes the form
# {dn: {attr: [value, ...], ...}, ...}.
directory = dict([top, test, people, groups, admins, group1, group2, admin, alice, bob, karl])
#classmethod
def setUpTestData(cls):
# We only need to create the MockLdap instance once. The content we
# pass in will be used for all LDAP connections.
cls.mockldap = MockLdap(cls.directory)
#classmethod
def tearDownClass(cls):
del cls.mockldap
def setUp(self):
# Patch ldap.initialize
django.setup()
self.mockldap.start()
self.ldapobj = self.mockldap['ldap://localhost/']
def tearDown(self):
# Stop patching ldap.initialize and reset state.
self.mockldap.stop()
del self.ldapobj
def test_some_basic_mockldap_auth(self):
searchStr = 'uid=alice,ou=people,dc=test,dc=ee'
results = _do_simple_ldap_search(searchStr)
ldapName = results[0][0]
ldapPropDict = results[0][1]
self.assertEqual(searchStr, ldapName)
self.assertEqual(len(ldapPropDict), 7)
def test_index_visible_for_all(self):
c = Client()
response = c.get("/")
self.assertContains(response, "/users/", 1)
self.assertContains(response, "/groups/", 1)
def test_login(self):
c = Client()
response = c.post("/api-auth/login/", {'username': 'bob', 'password': 'ldaptest'})
print(response.status_code)
response = c.get("/users/")
print(response._container)
def _do_simple_ldap_search(searchStr):
conn = ldap.initialize('ldap://localhost/')
conn.simple_bind_s(searchStr, 'ldaptest')
results = conn.search_s(searchStr, ldap.SCOPE_SUBTREE, )
return results
Code also available here: http://dpaste.com/3D2H4NK (syntax highlight).
I am not sure, what the problem is. Only thing I can think of is that because there are no users in database at the creation, database is not created, but I am not sure. Any help is much appreciated.
I have done all the migrations.
(venv)user#box:~/workspace/fileshare/njord$ python manage.py showmigrationsadmin
[X] 0001_initial
[X] 0002_logentry_remove_auto_add
auth
[X] 0001_initial
[X] 0002_alter_permission_name_max_length
[X] 0003_alter_user_email_max_length
[X] 0004_alter_user_username_opts
[X] 0005_alter_user_last_login_null
[X] 0006_require_contenttypes_0002
[X] 0007_alter_validators_add_error_messages
contenttypes
[X] 0001_initial
[X] 0002_remove_content_type_name
sessions
[X] 0001_initial
(venv)user#box:~/workspace/fileshare/njord$ python manage.py makemigrationsNo changes detected
(venv)user#box:~/workspace/fileshare/njord$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, sessions, auth, contenttypes
Running migrations:
No migrations to apply.
run
python manage.py makemigrations <appname>
for each applications in INSTALLED_APPS.(Specially the apps that has a ForeignKey field on auth_user)
I got different error to get me here, but think cause is the same.
django.db.utils.OperationalError: (1005, "Can't create table '<test_db>.#sql-3821_1c9' (errno: 150)")
In both cases all tables related to auth module are only not created when testing, without executing makemigrations command.
I found the solution from here
What had to be done, was add these lines to manage.py:
if __name__ == "__main__":
if 'test' in sys.argv:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.test_settings")
else:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")
After adding that if 'test' thing to main method of manage.py, the app started working correctly, after that python manage.py test and python manage.py runserver both worked with correct settings.
However, I have not got this to work with PyCharm and testing it now, how to get it to work.

Django - can't run tests while the application works normally

Problem:
The problem is that I have an application which works ok, but when I try to run tests with the command:
coverage run manage.py test --settings=crm.settings.test
there occurs an error in the very beggining:
Creating test database for alias 'default'...
IntegrityError: profiles_usermodel.current_project_id may not be NULL
Previously I ran tests and everything worked nicely. Then I significantly changed the models and the application and tried to run tests -- as a result I got the mentioned above problem. What I did wrong?
My settings:
I have separate settings for tests:
My_application
|___My_application
|___urls.py
|___ __init__.py
|___wsgi.py
|___settings
|___base.py
|___test.py
|___local.py
|___profiles
|___models.py
|___views.py
|___tests
|___ __init__.py
|___models.py
The test settings are as follows:
"""Local test settings and globals which allows us to run our
test suite locally."""
from .base import *
########## TEST SETTINGS
TEST_RUNNER = 'discover_runner.DiscoverRunner'
TEST_DISCOVER_TOP_LEVEL = PROJECT_ROOT
TEST_DISCOVER_ROOT = PROJECT_ROOT
TEST_DISCOVER_PATTERN = "*"
########## IN-MEMORY TEST DATABASE
DATABASES = {
"default": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": ":memory:",
"USER": "",
"PASSWORD": "",
"HOST": "",
"PORT": "",
},
}
INSTALLED_APPS += ('coverage',)
And there is the models from My_application/profiles/models.py (ommited several fields and methods):
from django.db import models
from django.contrib.auth.models import (
BaseUserManager, AbstractBaseUser, PermissionsMixin )
from django.contrib.auth.models import Group
class Project(models.Model):
name = models.CharField(max_length=255)
class UserModel(AbstractBaseUser, PermissionsMixin):
email = models.EmailField(max_length=255, unique=True, db_index=True,)
current_project = models.ForeignKey(Group, related_name='usermodel_current')
USERNAME_FIELD = 'email'
Also I use South for db migrations and django 1.5c2.
The error is being pretty explicit.
The test runner is attempting to create the database, and insert a record into 'profiles_usermodel'. This insertion is failing because the field 'current_project' cannot be null, but the creation process isn't setting anything into that field.
We'd have to see your tests to narrow down exactly where this is occurring.
If you don't care about getting it working "right" and just want it to work, update your user model, and add null=True to the kwargs of UserModel.current_project.

Django tests complain of missing tables

When I run my test dealing with my Customer model, I get the following error:
DatabaseError: (1146, "Table 'test_mcif2.customer' doesn't exist")
I'm not entirely surprised because I have my Django project connected to a "legacy" database. Since my tables weren't created "the Django way," it's not shocking that Django wouldn't be able to talk to them without some finagling. Here's my model:
from django.db import models
from django.db import connection, transaction
from mcif.models.mcif_model import McifModel
class Customer(McifModel):
class Meta:
db_table = u'customer'
app_name = 'mcif'
id = models.BigIntegerField(primary_key=True)
customer_number = models.CharField(unique=True, max_length=255)
social_security_number = models.CharField(unique=True, max_length=33)
name = models.CharField(unique=True, max_length=255)
phone = models.CharField(unique=True, max_length=255)
deceased = models.IntegerField(unique=True, null=True, blank=True)
do_not_mail = models.IntegerField(null=True, blank=True)
created_at = models.DateTimeField()
updated_at = models.DateTimeField()
def distinguishing_column_names(self):
return ['name', 'customer_number', 'social_security_number', 'phone']
Any idea why exactly this isn't working?
Edit: Here's McifModel:
from django.db import models
from django.db import connection, transaction
class McifModel(models.Model):
class Meta:
abstract = True
def upsert(self):
cursor = connection.cursor()
cursor.execute(self.upsert_sql())
transaction.commit_unless_managed()
return self
def value_list(self):
return ','.join(map(lambda column_name: "'{c}'".format(c=getattr(self, column_name)), self.distinguishing_column_names()))
def upsert_sql(self):
column_names = ','.join(self.distinguishing_column_names())
return "INSERT IGNORE INTO {t} ({c}) VALUES ({v})".format(t=self._meta.db_table, c=column_names, v=self.value_list())
#classmethod
def save_from_row(cls, row):
object = cls()
map(lambda column_name: setattr(object, column_name, row.value(object._meta.db_table, column_name)), object.distinguishing_column_names())
return object.upsert()
Edit: I took tarequeh's advice and put the contents of the Caktus file in mcif/utils.py. I also set TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'. If I go on the console I can verify that Customer is unmanaged:
>>> [m for m in get_models() if not m._meta.managed]
[<class 'mcif.models.customer.Customer'>]
However, my test still complains that the table doesn't exist. What am I missing?
Here's my settings.py:
# Django settings for mcifdjango project.
DEBUG = True
TEMPLATE_DEBUG = DEBUG
ADMINS = (
('Jason Swett', 'jason.swett#gmail.com'),
)
MANAGERS = ADMINS
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
'NAME': 'xxxxx', # Or path to database file if using sqlite3.
'USER': 'xxxxx', # Not used with sqlite3.
'PASSWORD': 'xxxxx', # Not used with sqlite3.
'HOST': '', # Set to empty string for localhost. Not used with sqlite3.
'PORT': '', # Set to empty string for default. Not used with sqlite3.
}
}
# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# On Unix systems, a value of None will cause Django to use the same
# timezone as the operating system.
# If running in a Windows environment this must be set to the same as your
# system time zone.
TIME_ZONE = 'America/Chicago'
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'
SITE_ID = 1
# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True
# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale
USE_L10N = True
# Absolute path to the directory that holds media.
# Example: "/home/media/media.lawrence.com/"
MEDIA_ROOT = ''
# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash if there is a path component (optional in other cases).
# Examples: "http://media.lawrence.com", "http://example.com/media/"
MEDIA_URL = ''
# URL prefix for admin media -- CSS, JavaScript and images. Make sure to use a
# trailing slash.
# Examples: "http://foo.com/media/", "/media/".
ADMIN_MEDIA_PREFIX = '/media/'
# Make this unique, and don't share it with anybody.
SECRET_KEY = '#7+qm%hqfe+z8ul5#x_i&sqmu!n=4sa0&i0_#)m99*w$fbk3%#'
# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
'django.template.loaders.filesystem.Loader',
'django.template.loaders.app_directories.Loader',
# 'django.template.loaders.eggs.Loader',
)
MIDDLEWARE_CLASSES = (
'django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
)
ROOT_URLCONF = 'mcifdjango.urls'
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
)
INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.admin',
'django_extensions',
'mcif',
# Uncomment the next line to enable the admin:
# 'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
)
TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'
import os
ROOTDIR = os.path.abspath(os.path.dirname(__file__))
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
ROOTDIR + '/mcif/templates',
)
Edit 2:
Here's my Customer class now:
from django.db import models
from django.db import connection, transaction
from mcif.models.mcif_model import McifModel
class Customer(McifModel):
class Meta:
db_table = u'customer'
managed = False
id = models.BigIntegerField(primary_key=True)
customer_number = models.CharField(unique=True, max_length=255)
social_security_number = models.CharField(unique=True, max_length=33)
name = models.CharField(unique=True, max_length=255)
phone = models.CharField(unique=True, max_length=255)
deceased = models.IntegerField(unique=True, null=True, blank=True)
do_not_mail = models.IntegerField(null=True, blank=True)
created_at = models.DateTimeField()
updated_at = models.DateTimeField()
def distinguishing_column_names(self):
return ['name', 'customer_number', 'social_security_number', 'phone']
Here's what I get when I run the test:
$ ./manage.py test mcif.CustomerUpsertTest
Creating test database 'default'...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table django_admin_log
Installing index for auth.Permission model
Installing index for auth.Group_permissions model
Installing index for auth.User_user_permissions model
Installing index for auth.User_groups model
Installing index for auth.Message model
Installing index for admin.LogEntry model
No fixtures found.
E
======================================================================
ERROR: test_upsert (mcif.tests.customer_upsert_test.CustomerUpsertTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jason/projects/mcifdjango/mcif/tests/customer_upsert_test.py", line 9, in test_upsert
customer.upsert()
File "/home/jason/projects/mcifdjango/mcif/models/mcif_model.py", line 11, in upsert
cursor.execute(self.upsert_sql())
File "/usr/lib/pymodules/python2.6/django/db/backends/mysql/base.py", line 86, in execute
return self.cursor.execute(query, args)
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
DatabaseError: (1146, "Table 'test_mcif_django.customer' doesn't exist")
----------------------------------------------------------------------
Ran 1 test in 3.724s
FAILED (errors=1)
Destroying test database 'default'...
Since you're using a legacy database, you are probably not adding the app name to INSTALLED_APPS. If an app is not included in INSTALLED_APPS, the tables for the apps' models will not get created on syncdb. This works for you in production since you already have a table, but not in test environment.
You can adopt any of the following:
The supermonkeypatch way: Take out app_name from Customer class Meta, put the model in a models.py file inside a python module name mcif, and add mcif to INSTALLED_APPS - just for the sake of testing
The nicer way: Extend DjangoTestSuiteRunner and override setup_test_environment to call super and then create your legacy table manually in the test DB.
The nicest way: Put your model in properly named app module. Remove app_name from model Meta but add managed=False docs. Include app name in INSTALLED_APPS. Now django will not create table for that model. Then use this nice snippet the Caktus group folks have compiled to run your tests.
Cheers!
Edit - How to use the overridden DjangoTestSuiteRunner
You will need at least Django 1.2 for this.
Copy the code from here. Put it in utils.py inside the mcif app.
Add/edit the following in settings.py:
TEST_RUNNER = 'mcif.utils.ManagedModelTestRunner'
Now when you run tests, all unmanaged tables will be treated as managed table only for the duration of the test. So the tables will be created prior to running tests.
Notice this part of the code, thats where the magic happens.
self.unmanaged_models = [m for m in get_models() if not m._meta.managed]
for m in self.unmanaged_models:
m._meta.managed = True
2nd Edit: Possible Gotchas
Make sure of the following:
The DB user has privilege to create databases and not only tables because django will try to create a test database
The test cases extend django.test.TransactionTestCase, since you have transactional behavior
If none of the above applies, put a pdb in ManagedModelTestRunner's setup_test_environment just to make sure the code is being reached. Because if that code is reached, the table should get created
3rd Edit: Debugging
Inside mcif.utils.ManagedModelTestRunner replace setup_test_environment function with the following and let me know if the output of your test changes:
def setup_test_environment(self, *args, **kwargs):
print "Loading ManagedModelTestRunner"
from django.db.models.loading import get_models
self.unmanaged_models = [m for m in get_models()
if not m._meta.managed]
for m in self.unmanaged_models:
print "Modifying model %s to be managed for testing" % m
m._meta.managed = True
super(ManagedModelTestRunner, self).setup_test_environment(*args, **kwargs)
The solutions presented by tarequeh worked for me after overriding DATABASE_ROUTERS.
I am using routers in order to prevent writes on the legacy database. In order to get around this I created a test_settings file with the following contents:
from settings import *
DEBUG = True
TEST_RUNNER = 'legacy.utils.ManagedModelTestRunner'
DATABASE_ROUTERS = []
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(HERE, 'test.db'),
},
}
Then when running tests:
python manage.py test [app_name] --settings=test_settings
There's not enough info above to answer your first question. However, once you get that issue resolved you'll probably want to install django-extensions for the following reason: It has an incredibly useful sqldiff command that will inform you if there's a mismatch between the legacy database and your application model.
Here's a more up-to-date solution, that also works with current versions of Django (I tested it on Django 3.2.11):
https://medium.com/an-idea/testing-with-the-legacy-database-in-django-3be84786daba
Also, in case you want to furthermore populate the Django test-database with your legacy database's data:
Check out fixtures