Django test database requires Postgres Extension enabled - django

I have a Django app which requires Postgres' fuzzystrmatch extension to be enabled on the database.
Django's unittest framework creates and destroys a new database. I need this new database to have the extension turned on for testing.
I can use './manage.py test --keepdb' to keep the database and then manually turn on the extension, but this is an ineloquent solution.
Any idea how I can enable this extension programmatically?

Create an empty migration and use the CreateExtension operation:
from django.contrib.postgres.operations import CreateExtension
class Migration(migrations.Migration):
...
operations = [
CreateExtension(name='fuzzystrmatch'),
...
]
Relevant docs
Edit:
If that doesn't work, reviewing how Django actually handles those classes internally would be my next suggestion.

In case you've disabled the migrations in the test settings, the migrations will not be run, so you can't use django CreateExtension.
Example:
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "test-database",
"USER": "username",
"PASSWORD": "password",
"HOST": "postgresql-host",
"PORT": "5432",
"TEST": {
"MIGRATE": False,
},
},
}
In this case you can manually call SQL code to create the extension.
If you use unitest, you can for example create a custom TestCase class inherited by all your tests like so:
from django.test import TestCase from django.db import connection
class PostgresExtensionTestCase(TestCase):
#classmethod
def setUpClass(cls):
with connection.cursor() as cursor:
cursor.execute("CREATE EXTENSION IF NOT EXISTS unaccent;")
super().setUpClass()
If you use pytest, you can create a fixture in your conftest.py like so (for unaccent in the example):
from django.db import connection
#pytest.fixture(autouse=True)
def activate_postgresql_unaccent():
with connection.cursor() as cursor:
cursor.execute("CREATE EXTENSION IF NOT EXISTS unaccent;")

Related

How to insert(Add) JSON data in MySql database through Postman or Rest API Framework using Django?

I have started coding in Python using Django and MySql recently.So, I am facing issue in simple inserting data(JSON format) in a table in MySql database.I have tried few things which I am mentioning below.But it did not work.And I am using the following - Python, Django Framework, MySql Database, mysqlclient connector, Rest API Framework, Postman. Please help me out with suggestions.
from django.shortcuts import render, redirect
from rest_framework.decorators import api_view
from rest_framework.response import Response
from .models import fetchdata
from .serializers import fetchdataSerializers
from rest_framework import status,response
from rest_framework import viewsets
from django.http import HttpResponse
#api_view(['GET','POST'])
def index(request):
if request.method=='POST':
try:
_json = request.json
_name = _json['name']
_email = _json['email']
_password = _json['pwd']
sql = "INSERT INTO tbl_user(user_name,user_email,user_password)
VALUES(%s,%s,%s)"
data = (_name, _email, _password,)
conn = mysql.connect()
cursor = conn.cursor()
cursor.execute(sql, data)
conn.commit()
resp = jsonify('User added successfully!')
serializer=fetchdataSerializers(data=request.data)
if serializer.is_valid():
serializer.save()
return resp
return Response(serializer.data,status=status.HTTP_201_CREATED)
return Response(serializer.errors,status=status.HTTP_400_BAD_REQUEST)
You can add a fixtures folder in your app and place the Json file in it.
The Json file might be something like this category.json
[
{
"model": "category",
"pk": 1,
"fields": {
"name": "person",
"created_at": "2019-05-06T09:27:51.386383",
"updated_at": "2019-05-06T09:27:51.386383"
}
},
]
Run python manage.py loadata category.json. Depending on how you plan to implement this, you can run this command after your migration.
Check the Doc to understand more

How install a extension in postgresql before creating the models in django?

Using django 1.8 + Postgres 9+, I have models with custom PG datatypes (like ltree). Creating de database from zero fail because
CREATE EXTENSION ltree;
is not executed. I try with a empty migration, but that run after the models creation. Exist a way to run sql before the models creation?
I know this is unanswered for long, and maybe now you already have figured out the answer. But I'm posting just in case someone gets a little bit of help from this.
For extensions that are available in Postgres
If the extension is one of the defaults that are available with Postgres, then you can simply create first migration as this, and then load other migrations.
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.contrib.postgres.operations import HStoreExtension
from django.db import migrations
class Migration(migrations.Migration):
run_before = [
('some_app_that_requires_hstore', '0001_initial'),
]
operations = [
HStoreExtension(),
]
Notice the use of run_before. It is the exact opposite of dependencies. Alternatively, you can make this migration as say 1st one and load all others after this using dependencies.
In case this migration fails to create an extension because of privileges issues, then you can simply use the superuser and nosuperuser in Postgres to temporarily provide privileges for the current user to run migrations like:
ALTER ROLE user_name superuser;
# create the extension and then remove the superuser privileges.
ALTER ROLE user_name nosuperuser;
For third-party extensions that are not available in Postgres
For third-party extensions, you can use run_python, to load the extension for you like:
from django.db import migrations
def create_third_party_extension(apps, schema_editor):
schema_editor.execute("CREATE EXTENSION my_custom_extension;")
def drop_third_party_extension(apps, schema_editor):
schema_editor.execute("DROP EXTENSION IF EXISTS my_custom_extension;")
class Migration(migrations.Migration):
dependencies = [
('venues', '0001_auto_20180607_1014'),
]
operations = [
migrations.RunPython(create_third_party_extension, reverse_code=drop_third_party_extension, atomic=True)
]
Alternatively, you can just have them as a part of your deployment scripts instead of migrations.
I hope this helps.
You can also install the extension on the template database, and when new databases are created (like when running tests), the new database will copy that template and include the extension.
psql -d template1 -c 'create extension if not exists hstore;'

Django - populate table on startup with known values

I have a Dog Model that have a "dog_type" field. i want the dog_type to be chosen from a list of pre-defined dog types, i DO NOT want to use a textfield with choices but a ForeignKey to a "DogType" Model. How could I populate the DogType Model with types on server startup? is this a good practice or a hack?
thanks.
code:
class Dog(Model):
name = CharField(...)
dog_type = ForeignKey(DogType)
class DogType(Model):
type_name = CharField(...)
type_max_hight = IntegerField(...)
etc....
You'll probably want to write a data migration that will add your choices in database.
Advantages of using this approach is that the data will be loaded in all your databases (production, dev, etc.)
(If you're not using migrations yet, you should consider it, it's a clean and well-supported way to manage your database)
In your django project, just run python manage.py shell makemigrations myapp --empty. This will create an empty migration file under myapp/migrations.
You can then edit it:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models, migrations
DEFAULT_DOG_TYPES = (
('Labrador', 90),
('Berger Allemand', 66),
('Chihuaha', -2),
)
def add_dog_types(apps, schema_editor):
DogType = apps.get_model('myapp', 'DogType')
for name, max_height in DEFAULT_DOG_TYPES:
dog_type = DogType(name=name, max_height=max_height)
dog_type.save()
def remove_dog_types(apps, schema_editor):
# This is used to migrate backward
# You can remove your dog types, or just pass
# pass
for name, max_height in DEFAULT_DOG_TYPES:
dog_type = DogType.objects.get(name=name, max_height=max_height)
dog_type.delete()
class Migration(migrations.Migration):
dependencies = [
# if you're already using migrations, this line will be different
('myapp', '0001_initial'),
]
operations = [
migrations.RunPython(add_dog_types, remove_dog_types),
]
After that, all you need to do is to run python manage.py syncdb.
not so far after, i found this it's called "fixtures"...
basically what you need to do is to place a "fixture" file in a format of your choice (JSON\YAML...) "myapp/fixtures/" for example, a JSON file would look like this:
[
{
"model": "myapp.person",
"pk": 1,
"fields": {
"first_name": "John",
"last_name": "Lennon"
}
},
{
"model": "myapp.person",
"pk": 2,
"fields": {
"first_name": "Paul",
"last_name": "McCartney"
}
}
]
then simply run from the command line:
python manage.py loaddata <filename> # file with no path!

Django and postgresql testing schema

I'm using PostgreSQL 9.3 and Django 1.7.4 with psycopg2 2.5.4
The DBA, asked us to create a schema for our application instead of using public.
We defined the schema, and we had to add the
'OPTIONS': {
'options': '-c search_path=custom-schema-name'
},
to the settings.
During testing, Django is creating the test database with the corresponding name, but we can't set the custom-schema name
I tried to find a way to set up the custom schema name (I've read the docs) but I can't find a way to force the creation of the schema name during testing.
The error that I obtain is
django.db.utils.ProgrammingError: no schema has been selected to create in
When I see the created database , it has the schema public created by default.
I partially solved this issue and added to the search path the schema name public
'OPTIONS': {
'options': '-c search_path=custom-schema-name,public'
},
but I'd like to create the test database with a custom schema name.
Does anybody knows how to set the testing schema name ?
I ended up with writing a custom test runner to solve this problem (using django 1.9.x):
myapp/test/runner.py
from types import MethodType
from django.test.runner import DiscoverRunner
from django.db import connections
def prepare_database(self):
self.connect()
self.connection.cursor().execute("""
CREATE SCHEMA foobar_schema AUTHORIZATION your_user;
GRANT ALL ON SCHEMA foobar_schema TO your_user;
""")
class PostgresSchemaTestRunner(DiscoverRunner):
def setup_databases(self, **kwargs):
for connection_name in connections:
connection = connections[connection_name]
connection.prepare_database = MethodType(prepare_database, connection)
return super().setup_databases(**kwargs)
settings.py
TEST_RUNNER = 'myapp.test.runner.PostgresSchemaTestRunner'

Django - can't run tests while the application works normally

Problem:
The problem is that I have an application which works ok, but when I try to run tests with the command:
coverage run manage.py test --settings=crm.settings.test
there occurs an error in the very beggining:
Creating test database for alias 'default'...
IntegrityError: profiles_usermodel.current_project_id may not be NULL
Previously I ran tests and everything worked nicely. Then I significantly changed the models and the application and tried to run tests -- as a result I got the mentioned above problem. What I did wrong?
My settings:
I have separate settings for tests:
My_application
|___My_application
|___urls.py
|___ __init__.py
|___wsgi.py
|___settings
|___base.py
|___test.py
|___local.py
|___profiles
|___models.py
|___views.py
|___tests
|___ __init__.py
|___models.py
The test settings are as follows:
"""Local test settings and globals which allows us to run our
test suite locally."""
from .base import *
########## TEST SETTINGS
TEST_RUNNER = 'discover_runner.DiscoverRunner'
TEST_DISCOVER_TOP_LEVEL = PROJECT_ROOT
TEST_DISCOVER_ROOT = PROJECT_ROOT
TEST_DISCOVER_PATTERN = "*"
########## IN-MEMORY TEST DATABASE
DATABASES = {
"default": {
"ENGINE": "django.db.backends.sqlite3",
"NAME": ":memory:",
"USER": "",
"PASSWORD": "",
"HOST": "",
"PORT": "",
},
}
INSTALLED_APPS += ('coverage',)
And there is the models from My_application/profiles/models.py (ommited several fields and methods):
from django.db import models
from django.contrib.auth.models import (
BaseUserManager, AbstractBaseUser, PermissionsMixin )
from django.contrib.auth.models import Group
class Project(models.Model):
name = models.CharField(max_length=255)
class UserModel(AbstractBaseUser, PermissionsMixin):
email = models.EmailField(max_length=255, unique=True, db_index=True,)
current_project = models.ForeignKey(Group, related_name='usermodel_current')
USERNAME_FIELD = 'email'
Also I use South for db migrations and django 1.5c2.
The error is being pretty explicit.
The test runner is attempting to create the database, and insert a record into 'profiles_usermodel'. This insertion is failing because the field 'current_project' cannot be null, but the creation process isn't setting anything into that field.
We'd have to see your tests to narrow down exactly where this is occurring.
If you don't care about getting it working "right" and just want it to work, update your user model, and add null=True to the kwargs of UserModel.current_project.