In my django project, I have 5 applications, with total 15 models,and all of them are unmanaged. I've written some tests in pytest-django, and when I run them, they fail due to not being able to find tables.
How can I create database entries for all these models so that the tests don't fail?
I was trying to get this to work on Django==4.0.4, pytest-django==4.5.2 and none of the results I could find out there worked for me. This is what I could figure out:
# add this to conftest.py
#pytest.hookimpl(tryfirst=True)
def pytest_runtestloop():
from django.apps import apps
unmanaged_models = []
for app in apps.get_app_configs():
unmanaged_models += [m for m in app.get_models()
if not m._meta.managed]
for m in unmanaged_models:
m._meta.managed = True
It seems intuitive to think that we can achieve what we need by overriding django_db_setup, and that is the solution provided in other answers on SO. However I was not able to get this to work, that might have something to do with changes to the order of execution of these fixtures over the years, I am not sure.
Excerpt from the current pytest-django docs:
pytest-django calls django.setup() automatically. If you want to do anything before this, you have to create a pytest plugin and use the pytest_load_initial_conftests() hook
Read more
You can override the django_db_setup fixture in conftest.py file:
#pytest.fixture(scope="session")
def django_db_setup(django_db_blocker):
with django_db_blocker.unblock():
from django.apps import apps
models_list = apps.get_models()
for model in models_list:
with connection.schema_editor() as schema_editor:
schema_editor.create_model(model)
if model._meta.db_table not in connection.introspection.table_names():
raise ValueError(
"Table `{table_name}` is missing in test database.".format(
table_name=model._meta.db_table
)
)
yield
for model in models_list:
with connection.schema_editor() as schema_editor:
schema_editor.delete_model(model)
This will create the tables for unmanaged models before running tests, and delete those tables after test.
Related
I have the following inside conftest.py
#pytest.mark.django_db
#pytest.fixture(scope='module')
def thing():
print('sleeping') # represents a very expensive function that i want to only ever once once per module
Thing.objects.create(thing='hello')
Thing.objects.create(thing='hello')
Thing.objects.create(thing='hello')
Inside tests.py
#pytest.mark.django_db
def test_thing(thing):
assert models.Thing.objects.count() > 1
#pytest.mark.django_db
def test_thing2(thing):
assert models.Thing.objects.count() > 1
#pytest.mark.django_db
#pytest.mark.usefixtures('thing')
def test_thing3():
assert models.Thing.objects.count() > 1
All three tests throw the same error: RuntimeError: Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it.
I've tried using scope='session' / scope='class' / scope='package' / scope='module' -- the only one that works is `scope='function' which defeats the purpose of what I'm trying to accomplish. I want to be able to create all these items ONCE per module, not once per test.
Note: I ran into this issue with a large code base and created a new django project with a single app to test and see if the problem was the existing test code, and it failed on a standalone test also. Tested it with both postgres and sqlite; doesn't seem like a database issue.
Not that it matters, but the models.py
class Thing(models.Model):
thing = models.CharField(max_length=100)
Ok, turns out this is a known limitation, and it's somewhat documented here. If you want to solve this issue, and get away from this bug:
#pytest.mark.django_db
#pytest.fixture(scope='module')
def thing(django_db_setup, django_db_blocker):
del django_db_setup # Cannot be used with usefixtures(..) it won't work
with django_db_blocker.unblock():
print('sleeping')
Thing.objects.create(thing='hello')
Thing.objects.create(thing='hello')
Thing.objects.create(thing='hello')
Thing.objects.create(thing='hello')
yield
I've created a Configuration model in django so that the site admin can change some settings on the fly, however some of the models are reliant on these configurations. I'm using Django 2.0.2 and Python 3.6.4.
I created a config.py file in the same directory as models.py.
Let me paracode (paraphase the code? Real Enum has many more options):
# models.py
from .config import *
class Configuration(models.Model):
starting_money = models.IntegerField(default=1000)
class Person(models.Model):
funds = models.IntegarField(default=getConfig(ConfigData.STARTING_MONEY))
# config.py
from .models import Configuration
class ConfigData(Enum):
STARTING_MONEY = 1
def getConfig(data):
if not isinstance(data, ConfigData):
raise TypeError(f"{data} is not a valid configuration type")
try:
config = Configuration.objects.get_or_create()
except Configuration.MultipleObjectsReturned:
# Cleans database in case multiple configurations exist.
Configuration.objects.exclude(Configuration.objects.first()).delete()
return getConfig(data)
if data is ConfigData.MAXIMUM_STAKE:
return config.max_stake
How can I do this without an import error? I've tried absolute imports
You can postpone loading the models.py by loading it in the getConfig(data) function, as a result we no longer need models.py at the time we load config.py:
# config.py (no import in the head)
class ConfigData(Enum):
STARTING_MONEY = 1
def getConfig(data):
from .models import Configuration
if not isinstance(data, ConfigData):
raise TypeError(f"{data} is not a valid configuration type")
try:
config = Configuration.objects.get_or_create()
except Configuration.MultipleObjectsReturned:
# Cleans database in case multiple configurations exist.
Configuration.objects.exclude(Configuration.objects.first()).delete()
return getConfig(data)
if data is ConfigData.MAXIMUM_STAKE:
return config.max_stake
We thus do not load models.py in the config.py. We only check if it is loaded (and load it if not) when we actually execute the getConfig function, which is later in the process.
Willem Van Onsem's solution is a good one. I have a different approach which I have used for circular model dependencies using django's Applications registry. I post it here as an alternate solution, in part because I'd like feedback from more experienced python coders as to whether or not there are problems with this approach.
In a utility module, define the following method:
from django.apps import apps as django_apps
def model_by_name(app_name, model_name):
return django_apps.get_app_config(app_name).get_model(model_name)
Then in your getConfig, omit the import and replace the line
config = Configuration.objects.get_or_create()
with the following:
config_class = model_by_name(APP_NAME, 'Configuration')
config = config_class.objects.get_or_create()
I'm using django with neo4j as database and noemodel as OGM. How do I test it?
When I run python3 manage.py test all the changes, my tests make are left.
And also how do I make two databases, one for testing, another for working in production and specify which one to use how?
I assume the reason all of your changes are being retained is due to using the same neo4j database for testing as you are using in development. Since neomodel isn't integrated tightly with Django it doesn't act the same way Django's ORM does when testing. Django will do some helpful things when you run tests using its ORM, such as creating a test database that will be destroyed upon completion.
With neo4j and neomodel I'd recommend doing the following:
Create a Custom Test Runner
Django enables you to define a custom test runner by setting the TEST_RUNNER settings variable. An extremely simple version of this to get you going would be:
from time import sleep
from subprocess import call
from django.test.runner import DiscoverRunner
class MyTestRunner(DiscoverRunner):
def setup_databases(self, *args, **kwargs):
# Stop your development instance
call("sudo service neo4j-service stop", shell=True)
# Sleep to ensure the service has completely stopped
sleep(1)
# Start your test instance (see section below for more details)
success = call("/path/to/test/db/neo4j-community-2.2.2/bin/neo4j"
" start-no-wait", shell=True)
# Need to sleep to wait for the test instance to completely come up
sleep(10)
if success != 0:
return False
try:
# For neo4j 2.2.x you'll need to set a password or deactivate auth
# Nigel Small's py2neo gives us an easy way to accomplish this
call("source /path/to/virtualenv/bin/activate && "
"/path/to/virtualenv/bin/neoauth "
"neo4j neo4j my-p4ssword")
except OSError:
pass
# Don't import neomodel until we get here because we need to wait
# for the new db to be spawned
from neomodel import db
# Delete all previous entries in the db prior to running tests
query = "match (n)-[r]-() delete n,r"
db.cypher_query(query)
super(MyTestRunner, self).__init__(*args, **kwargs)
def teardown_databases(self, old_config, **kwargs):
from neomodel import db
# Delete all previous entries in the db after running tests
query = "match (n)-[r]-() delete n,r"
db.cypher_query(query)
sleep(1)
# Shut down test neo4j instance
success = call("/path/to/test/db/neo4j-community-2.2.2/bin/neo4j"
" stop", shell=True)
if success != 0:
return False
sleep(1)
# start back up development instance
call("sudo service neo4j-service start", shell=True)
Add a secondary neo4j database
This can be done in a couple ways but to follow along with the test runner above you can download a community distribution from neo4j's website. With this secondary instance you can now swap between which database you'd like to use utilizing the command line statements used in the calls within the test runner.
Wrap Up
This solution assume's you're on a linux box but should be portable to a different OS with minor modifications. Also I'd recommend checking out the Django's Test Runner Docs to expand upon what the test runner can do.
There currently isn't mechanism for working with test databases in neomodel as neo4j only has 1 schema per instance.
However you can override the environment variable NEO4J_REST_URL when running the tests like so
export NEO4J_REST_URL=http://localhost:7473/db/data python3 manage.py test
The way I went about this was to give in and use the existing database, but mark all test-related nodes and detach/delete them when finished. It's obviously not ideal; all your node classes must inherit from NodeBase or risk polluting the db with test data, and if you have unique constraints, those will still be enforced across both live/test data. But it works for my purposes, and I thought I'd share in case it helps someone else.
in myproject/base.py:
from neomodel.properties import Property, validator
from django.conf import settings
class TestModeProperty(Property):
"""
Boolean property that is only set during unit testing.
"""
#validator
def inflate(self, value):
return bool(value)
#validator
def deflate(self, value):
return bool(value)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.default = True
self.has_default = settings.UNIT_TESTING
class NodeBase(StructuredNode):
__abstract_node__ = True
test_mode = TestModeProperty()
in myproject/test_runner.py:
from django.test.runner import DiscoverRunner
from neomodel import db
class NeoDiscoverRunner(DiscoverRunner):
def teardown_databases(self, old_config, **kwargs):
db.cypher_query(
"""
MATCH (node {test_mode: true})
DETACH DELETE node
"""
)
return super().teardown_databases(old_config, **kwargs)
in settings.py:
UNIT_TESTING = sys.argv[1:2] == ["test"]
TEST_RUNNER = "myproject.test_runner.NeoDiscoverRunner"
I'm trying to pre-populate the database with some test data for my Django project. Is there some easy way to do this with a script that's "outside" of Django?
Let's say I want to do this very simple task, creating 5 test users using the following code,
N = 10
i = 0
while i < N:
c = 'user' + str(i) + '#gmail.com'
u = lancer.models.CustomUser.objects.create_user(email=c, password="12345")
i = i + 1
The questions are,
WHERE do I put this test script file?
WHAT IMPORTS / COMMANDS do I need to put at the beginning of the file so it has access to all the Django environment & resources as if I were writing this inside the app?
I'm thinking you'd have to import and set up the settings file, and import the app's models, etc... but all my attempts have failed one way or another, so would appreciate some help =)
Thanks!
Providing another answer
The respondes below are excellent answers. I fiddled around and found an alternative way. I added the following to the top of the test data script,
from django.core.management import setup_environ
from project_lancer import settings
setup_environ(settings)
import lancer.models
Now my code above works.
I recommend you to use fixtures for these purposes:
https://docs.djangoproject.com/en/dev/howto/initial-data/
If you still want to use this initial code then read:
If you use south you can create migration and put this code there:
python manage.py schemamigration --empty my_data_migration
class Migration(SchemaMigration):
no_dry_run = False
def forwards(self, orm):
# more pythonic, you can also use bulk_insert here
for i in xrange(10):
email = "user{}#gmail.com".format(i)
u = orm.CustomUser.objects.create_user(email=email, password='12345)
You can put it to setUp method of your TestCase:
class MyTestCase(TestCase):
def setUp(self):
# more pythonic, you can also use bulk_insert here
for i in xrange(10):
email = "user{}#gmail.com".format(i)
u = lancer.models.CustomUser.objects.create_user(email=email,
password='12345')
def test_foo(self):
pass
Also you can define your BaseTestCase in which you override setUp method then you create your TestCase classes that inherit from BaseTestCase:
class BaseTestCase(TestCase):
def setUp(self):
'your initial logic here'
class MyFirstTestCase(BaseTestCase):
pase
class MySecondTestCase(BaseTestCase):
pase
But I think that fixtures is the best way:
class BaseTestCase(TestCase):
fixtures = ['users_for_test.json']
class MyFirstTestCase(BaseTestCase):
pase
class MySecondTestCase(BaseTestCase):
fixtures = ['special_users_for_only_this_test_case.json']
Updated:
python manage.py shell
from django.contrib.auth.hashers import make_password
make_password('12312312')
'pbkdf2_sha256$10000$9KQ15rVsxZ0t$xMEKUicxtRjfxHobZ7I9Lh56B6Pkw7K8cO0ow2qCKdc='
You can also use something like this or this to auto-populate your models for testing purposes.
With Django's new multi-db functionality in the development version, I've been trying to work on creating a management command that let's me synchronize the data from the live site down to a developer machine for extended testing. (Having actual data, particularly user-entered data, allows me to test a broader range of inputs.)
Right now I've got a "mostly" working command. It can sync "simple" model data but the problem I'm having is that it ignores ManyToMany fields which I don't see any reason for it do so. Anyone have any ideas of either how to fix that or a better want to handle this? Should I be exporting that first query to a fixture first and then re-importing it?
from django.core.management.base import LabelCommand
from django.db.utils import IntegrityError
from django.db import models
from django.conf import settings
LIVE_DATABASE_KEY = 'live'
class Command(LabelCommand):
help = ("Synchronizes the data between the local machine and the live server")
args = "APP_NAME"
label = 'application name'
requires_model_validation = False
can_import_settings = True
def handle_label(self, label, **options):
# Make sure we're running the command on a developer machine and that we've got the right settings
db_settings = getattr(settings, 'DATABASES', {})
if not LIVE_DATABASE_KEY in db_settings:
print 'Could not find "%s" in database settings.' % LIVE_DATABASE_KEY
return
if db_settings.get('default') == db_settings.get(LIVE_DATABASE_KEY):
print 'Data cannot synchronize with self. This command must be run on a non-production server.'
return
# Fetch all models for the given app
try:
app = models.get_app(label)
app_models = models.get_models(app)
except:
print "The app '%s' could not be found or models could not be loaded for it." % label
for model in app_models:
print 'Syncing %s.%s ...' % (model._meta.app_label, model._meta.object_name)
# Query each model from the live site
qs = model.objects.all().using(LIVE_DATABASE_KEY)
# ...and save it to the local database
for record in qs:
try:
record.save(using='default')
except IntegrityError:
# Skip as the record probably already exists
pass
Django command extension's Dumpscript should help a lot.
This doesn't answer your question exactly but why not just do a db dump and a db restore?