I have a Django application that executes a full-text-search on a database. The view that executes this query is my search_view (I'm ommiting some parts for the sake of simplicity). It just retrieve the results of the search on my Post model and send to the template:
def search_view(request):
posts = m.Post.objects.all()
query = request.GET.get('q')
search_query = SearchQuery(query, config='english')
qs = Post.objects.annotate(
rank=SearchRank(F('vector_column'), search_query) + TrigramSimilarity('post_title', query)
).filter(rank__gte=0.15).order_by('-rank'), 15
)
context = {
results = qs
}
return render(request, 'core/search.html', context)
The application is working just fine. The problem is with a test I created. Here is my tests.py:
class SearchViewTests(TestCase):
def test_search_without_results(self):
"""
If the user's query did not retrieve anything
show him a message informing that
"""
response = self.client.get(reverse('core:search') + '?q=eksjeispowjskdjies')
self.assertEqual(response.status_code, 200)
self.assertContains(response.content, "We didn\'t find anything on our database. We\'re sorry")
This test raises an ProgrammingError exception:
django.db.utils.ProgrammingError: function similarity(character varying, unknown) does not exist
LINE 1: ...plainto_tsquery('english'::regconfig, 'eksjeispowjskdjies')) + SIMILARITY...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
I understand very well this exception, 'cause I got it sometimes. The SIMILARITY function in Postgres accepts two arguments, and both need to be of type TEXT. The exception is raising because the second argument (my query term) is of type UNKNOWN, therefore the function won't work and Django raises the exception. And I don't understand why, because the actual search is working! Even in the shell it works perfectly:
In [1]: from django.test import Client
In [2]: c = Client()
In [3]: response = c.get(reverse('core:search') + '?page=1&q=eksjeispowjskdjies')
In [4]: response
Out[4]: <HttpResponse status_code=200, "text/html; charset=utf-8">
Any ideas about why test doesn't work, but the actual execution of the app works and console test works too?
I had the same problem and this how I solved it in my case:
First of all, the problem was that when Django creates the test database that it is going to use for tests it does not actually run all of your migrations. It simply creates the tables based on your models.
This means that migrations that create some extension in your database, like pg_trgm do not run when creating the test database.
One way to overcome this is to use a fixture in your conftest.py file which will make create said extensions before any tests run.
So, in your conftest.py file add the following:
# the following fixture is used to add the pg_trgm extension to the test database
#pytest.fixture(scope="session", autouse=True)
def django_db_setup(django_db_setup, django_db_blocker):
"""Test session DB setup."""
with django_db_blocker.unblock():
with connection.cursor() as cursor:
cursor.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm;")
You can of course replace pg_trgm with any other extension you require.
PS: You must make sure the extension you are trying to use works for the test database you have chosen. In order to change the database used by Django you can change the value of
DATABASES = {'default': env.db('your_database_connection_uri')}
in your application's settings.py.
I've got a wagtail site powered by Postgres and would like to implement a fuzzy search on all documents. However, according to wagtail docs "SearchField(partial_match=True) is not handled." Does anyone know of a way I can implement my own partial matching search?
I'm leaving this question intentionally open-ended because I'm open to pretty much any solution that works well and is fairly scalable.
We’re currently rebuilding the Wagtail search API in order to make autocomplete usable roughly the same way across backends.
For now, you can use directly the IndexEntry model that stores search data. Unfortunately, django.contrib.postgres.search does not contain a way to do an autocomplete query, so we have to do it ourselves for now. Here is how to do that:
from django.contrib.postgres.search import SearchQuery
from wagtail.contrib.postgres_search.models import IndexEntry
class SearchAutocomplete(SearchQuery):
def as_sql(self, compiler, connection):
return "to_tsquery(''%s':*')", [self.value]
query = SearchAutocomplete('postg')
print(IndexEntry.objects.filter(body_search=query).rank(query))
# All results containing words starting with “postg”
# should be displayed, sorted by relevance.
It doesn't seem to be documented yet, but the gist of autocomplete filtering with Postgres, using a request object, is something like
from django.conf import settings
from wagtail.search.backends import get_search_backend
from wagtail.search.backends.base import FilterFieldError, OrderByFieldError
def filter_queryset(queryset, request):
search_query = request.GET.get("search", "").strip()
search_enabled = getattr(settings, 'WAGTAILAPI_SEARCH_ENABLED', True)
if 'search' in request.GET and search_query:
if not search_enabled:
raise BadRequestError("search is disabled")
search_operator = request.GET.get('search_operator', None)
order_by_relevance = 'order' not in request.GET
sb = get_search_backend()
try:
queryset = sb.autocomplete(search_query, queryset, operator=search_operator, order_by_relevance=order_by_relevance)
except FilterFieldError as e:
raise BadRequestError("cannot filter by '{}' while searching (field is not indexed)".format(e.field_name))
except OrderByFieldError as e:
raise BadRequestError("cannot order by '{}' while searching (field is not indexed)".format(e.field_name))
The line to note is the call to sb.autocomplete.
If you want to use custom fields with autocomplete, you'll also need to add them into search_fields as an AutocompleteField in addition to a SearchField -- for example
search_fields = Page.search_fields + [
index.SearchField("field_to_index", partial_match=True)
index.AutocompleteField("field_to_index", partial_match=True),
...
This solution is working for Wagtail 2.3. If you using an older version, it is unlikely to work, and if you are using a future version, hopefully the details will be incorporated into the official documents, which currently state that autocomplete with Postgres is NOT possible. Thankfully, that has turned out to not be true, due to the work of Bertrand Bordage in the time since he wrote the other answer.
Is it possible to get object with comments related to it? Right now django comment framework creates query for every object which has related comments and another queries for comments owners. Can I somehow avoid this? I use django 1.4 so prefetch_related is allowed.
You could create a function that caches the count:
from django.contrib.contenttypes.models import ContentType
from django.contrib import comments
def get_comment_count_key(model):
content_type = ContentType.objects.get_for_model(model)
return 'comment_count_%s_%s' % (content_type.pk, model.pk)
def get_comment_count(model):
key = get_comment_count_key(model)
value = cache.get(key)
if value is None:
value = comments.get_model().objects.filter(
content_type = ContentType.objects.get_for_model(model),
object_pk = model.pk,
site__pk = settings.SITE_ID
).count()
cache.set(key, value)
return value
You could extend the Comment model and add get_comment_count there. Or put get_comment_count as a template filter. It doesn't matter.
Of course, you would also need cache invalidation when a new comment is posted:
from django.db.models import signals
from django.contrib import comments
def refresh_comment_count(sender, instance, **kwargs):
cache.delete(get_comment_count_key(instance.content_object))
get_comment_count(instance.content_object)
post_save.connect(refresh_comment_count, sender=comments.get_model())
post_delete.connect(refresh_comment_count, sender=comments.get_model())
You could improve this last snippet, by using cache.incr() on comment_was_posted, and cache.decr() on post_delete but that's left as an exercise for you :)
With Django's new multi-db functionality in the development version, I've been trying to work on creating a management command that let's me synchronize the data from the live site down to a developer machine for extended testing. (Having actual data, particularly user-entered data, allows me to test a broader range of inputs.)
Right now I've got a "mostly" working command. It can sync "simple" model data but the problem I'm having is that it ignores ManyToMany fields which I don't see any reason for it do so. Anyone have any ideas of either how to fix that or a better want to handle this? Should I be exporting that first query to a fixture first and then re-importing it?
from django.core.management.base import LabelCommand
from django.db.utils import IntegrityError
from django.db import models
from django.conf import settings
LIVE_DATABASE_KEY = 'live'
class Command(LabelCommand):
help = ("Synchronizes the data between the local machine and the live server")
args = "APP_NAME"
label = 'application name'
requires_model_validation = False
can_import_settings = True
def handle_label(self, label, **options):
# Make sure we're running the command on a developer machine and that we've got the right settings
db_settings = getattr(settings, 'DATABASES', {})
if not LIVE_DATABASE_KEY in db_settings:
print 'Could not find "%s" in database settings.' % LIVE_DATABASE_KEY
return
if db_settings.get('default') == db_settings.get(LIVE_DATABASE_KEY):
print 'Data cannot synchronize with self. This command must be run on a non-production server.'
return
# Fetch all models for the given app
try:
app = models.get_app(label)
app_models = models.get_models(app)
except:
print "The app '%s' could not be found or models could not be loaded for it." % label
for model in app_models:
print 'Syncing %s.%s ...' % (model._meta.app_label, model._meta.object_name)
# Query each model from the live site
qs = model.objects.all().using(LIVE_DATABASE_KEY)
# ...and save it to the local database
for record in qs:
try:
record.save(using='default')
except IntegrityError:
# Skip as the record probably already exists
pass
Django command extension's Dumpscript should help a lot.
This doesn't answer your question exactly but why not just do a db dump and a db restore?
There is a lot of documentation on how to serialize a Model QuerySet but how do you just serialize to JSON the fields of a Model Instance?
You can easily use a list to wrap the required object and that's all what django serializers need to correctly serialize it, eg.:
from django.core import serializers
# assuming obj is a model instance
serialized_obj = serializers.serialize('json', [ obj, ])
If you're dealing with a list of model instances the best you can do is using serializers.serialize(), it gonna fit your need perfectly.
However, you are to face an issue with trying to serialize a single object, not a list of objects. That way, in order to get rid of different hacks, just use Django's model_to_dict (if I'm not mistaken, serializers.serialize() relies on it, too):
from django.forms.models import model_to_dict
# assuming obj is your model instance
dict_obj = model_to_dict( obj )
You now just need one straight json.dumps call to serialize it to json:
import json
serialized = json.dumps(dict_obj)
That's it! :)
To avoid the array wrapper, remove it before you return the response:
import json
from django.core import serializers
def getObject(request, id):
obj = MyModel.objects.get(pk=id)
data = serializers.serialize('json', [obj,])
struct = json.loads(data)
data = json.dumps(struct[0])
return HttpResponse(data, mimetype='application/json')
I found this interesting post on the subject too:
http://timsaylor.com/convert-django-model-instances-to-dictionaries
It uses django.forms.models.model_to_dict, which looks like the perfect tool for the job.
There is a good answer for this and I'm surprised it hasn't been mentioned. With a few lines you can handle dates, models, and everything else.
Make a custom encoder that can handle models:
from django.forms import model_to_dict
from django.core.serializers.json import DjangoJSONEncoder
from django.db.models import Model
class ExtendedEncoder(DjangoJSONEncoder):
def default(self, o):
if isinstance(o, Model):
return model_to_dict(o)
return super().default(o)
Now use it when you use json.dumps
json.dumps(data, cls=ExtendedEncoder)
Now models, dates and everything can be serialized and it doesn't have to be in an array or serialized and unserialized. Anything you have that is custom can just be added to the default method.
You can even use Django's native JsonResponse this way:
from django.http import JsonResponse
JsonResponse(data, encoder=ExtendedEncoder)
It sounds like what you're asking about involves serializing the data structure of a Django model instance for interoperability. The other posters are correct: if you wanted the serialized form to be used with a python application that can query the database via Django's api, then you would wan to serialize a queryset with one object. If, on the other hand, what you need is a way to re-inflate the model instance somewhere else without touching the database or without using Django, then you have a little bit of work to do.
Here's what I do:
First, I use demjson for the conversion. It happened to be what I found first, but it might not be the best. My implementation depends on one of its features, but there should be similar ways with other converters.
Second, implement a json_equivalent method on all models that you might need serialized. This is a magic method for demjson, but it's probably something you're going to want to think about no matter what implementation you choose. The idea is that you return an object that is directly convertible to json (i.e. an array or dictionary). If you really want to do this automatically:
def json_equivalent(self):
dictionary = {}
for field in self._meta.get_all_field_names()
dictionary[field] = self.__getattribute__(field)
return dictionary
This will not be helpful to you unless you have a completely flat data structure (no ForeignKeys, only numbers and strings in the database, etc.). Otherwise, you should seriously think about the right way to implement this method.
Third, call demjson.JSON.encode(instance) and you have what you want.
If you want to return the single model object as a json response to a client, you can do this simple solution:
from django.forms.models import model_to_dict
from django.http import JsonResponse
movie = Movie.objects.get(pk=1)
return JsonResponse(model_to_dict(movie))
If you're asking how to serialize a single object from a model and you know you're only going to get one object in the queryset (for instance, using objects.get), then use something like:
import django.core.serializers
import django.http
import models
def jsonExample(request,poll_id):
s = django.core.serializers.serialize('json',[models.Poll.objects.get(id=poll_id)])
# s is a string with [] around it, so strip them off
o=s.strip("[]")
return django.http.HttpResponse(o, mimetype="application/json")
which would get you something of the form:
{"pk": 1, "model": "polls.poll", "fields": {"pub_date": "2013-06-27T02:29:38.284Z", "question": "What's up?"}}
.values() is what I needed to convert a model instance to JSON.
.values() documentation: https://docs.djangoproject.com/en/3.0/ref/models/querysets/#values
Example usage with a model called Project.
Note: I'm using Django Rest Framework
from django.http import JsonResponse
#csrf_exempt
#api_view(["GET"])
def get_project(request):
id = request.query_params['id']
data = Project.objects.filter(id=id).values()
if len(data) == 0:
return JsonResponse(status=404, data={'message': 'Project with id {} not found.'.format(id)})
return JsonResponse(data[0])
Result from a valid id:
{
"id": 47,
"title": "Project Name",
"description": "",
"created_at": "2020-01-21T18:13:49.693Z",
}
I solved this problem by adding a serialization method to my model:
def toJSON(self):
import simplejson
return simplejson.dumps(dict([(attr, getattr(self, attr)) for attr in [f.name for f in self._meta.fields]]))
Here's the verbose equivalent for those averse to one-liners:
def toJSON(self):
fields = []
for field in self._meta.fields:
fields.append(field.name)
d = {}
for attr in fields:
d[attr] = getattr(self, attr)
import simplejson
return simplejson.dumps(d)
_meta.fields is an ordered list of model fields which can be accessed from instances and from the model itself.
Here's my solution for this, which allows you to easily customize the JSON as well as organize related records
Firstly implement a method on the model. I call is json but you can call it whatever you like, e.g.:
class Car(Model):
...
def json(self):
return {
'manufacturer': self.manufacturer.name,
'model': self.model,
'colors': [color.json for color in self.colors.all()],
}
Then in the view I do:
data = [car.json for car in Car.objects.all()]
return HttpResponse(json.dumps(data), content_type='application/json; charset=UTF-8', status=status)
Use list, it will solve problem
Step1:
result=YOUR_MODELE_NAME.objects.values('PROP1','PROP2').all();
Step2:
result=list(result) #after getting data from model convert result to list
Step3:
return HttpResponse(json.dumps(result), content_type = "application/json")
Use Django Serializer with python format,
from django.core import serializers
qs = SomeModel.objects.all()
serialized_obj = serializers.serialize('python', qs)
What's difference between json and python format?
The json format will return the result as str whereas python will return the result in either list or OrderedDict
To serialize and deserialze, use the following:
from django.core import serializers
serial = serializers.serialize("json", [obj])
...
# .next() pulls the first object out of the generator
# .object retrieves django object the object from the DeserializedObject
obj = next(serializers.deserialize("json", serial)).object
All of these answers were a little hacky compared to what I would expect from a framework, the simplest method, I think by far, if you are using the rest framework:
rep = YourSerializerClass().to_representation(your_instance)
json.dumps(rep)
This uses the Serializer directly, respecting the fields you've defined on it, as well as any associations, etc.
It doesn't seem you can serialize an instance, you'd have to serialize a QuerySet of one object.
from django.core import serializers
from models import *
def getUser(request):
return HttpResponse(json(Users.objects.filter(id=88)))
I run out of the svn release of django, so this may not be in earlier versions.
ville = UneVille.objects.get(nom='lihlihlihlih')
....
blablablab
.......
return HttpResponse(simplejson.dumps(ville.__dict__))
I return the dict of my instance
so it return something like {'field1':value,"field2":value,....}
how about this way:
def ins2dic(obj):
SubDic = obj.__dict__
del SubDic['id']
del SubDic['_state']
return SubDic
or exclude anything you don't want.
This is a project that it can serialize(JSON base now) all data in your model and put them to a specific directory automatically and then it can deserialize it whenever you want... I've personally serialized thousand records with this script and then load all of them back to another database without any losing data.
Anyone that would be interested in opensource projects can contribute this project and add more feature to it.
serializer_deserializer_model
Let this is a serializers for CROPS, Do like below. It works for me, Definitely It will work for you also.
First import serializers
from django.core import serializers
Then you can write like this
class CropVarietySerializer(serializers.Serializer):
crop_variety_info = serializers.serialize('json', [ obj, ])
OR you can write like this
class CropVarietySerializer(serializers.Serializer):
crop_variety_info = serializers.JSONField()
Then Call this serializer inside your views.py
For more details, Please visit https://docs.djangoproject.com/en/4.1/topics/serialization/
serializers.JSONField(*args, **kwargs) and serializers.JSONField() are same. you can also visit https://www.django-rest-framework.org/api-guide/fields/ for JSONField() details.