I want to save the python dictionary inside the Django model as JSON and I want to convert that JSON back into a python dictionary when that data is fetched.
I know I can do it inside view but I want to implement it in the model so it can return dictionary object when queried.
is there any signal or any post_fetch method that I can use to achieve it, I couldn't find anything googling it...
You might want to look into simply using the Python JSON package, unless you are using Postgres as your database - in which case JSONField is the way to go. The json package is explained here and you could use it in a model like so if I understand what you are saying:
import json
class MyModel(models.Model):
json_field = models.TextField() # or you can use JSONField if using Postgres
#property
def get_json_field_as_dictionary(self):
return json.loads(self.json_field)
def set_json_field(self, mydict):
self.json_field = json.dumps(mydict)
#classmethod
def get_json_from_dictionary(cls, mydict):
return json.dumps(mydict)
When you are saving to the database, you can use json.dumps(myDictionary) or convert the dictionary to JSON by calling MyModelObject.set_json_field(myDictionary) to convert a Python dictionary to JSON and then store it in the json_field of the model. To retrieve the JSON data as a dictionary, you simply call MyModel.objects.last().get_json_field_as_dictionary (or whatever you prefer to call it, json_dictionary perhaps so it would be MyModel.objects.last().json_dictionary) and it will return the value of that property as if it were an element in the model without having to do the conversion each time.
Or, if you are using Postgres as your backend, this is a lot easier with JSONField:
class MyModel(models.Model):
json_field = models.JSONField(null=True)
And to save:
myObject = myModel.objects.create(json_field=myDictionary)
If you clarify I can update my answer to explain better.
You may want to use JSONField. It allows storing data encoded as JSON and retrieve them as the corresponding Python data type, including dictionary.
JSONField only works on PostgreSQL with Django < 3.1, but works on any database with Django >= 3.1.
Related
Complete DRF beginner here... I'm confused about the following concepts:
Let's say I POST some data, including a complex JSON blob for one of the fields, in order to create an object. Where should I actually create this object? Looking at the 3.1 docs, it seems like two places are equally valid for this: Serializer.create() and ViewSet.create(). How do I decide where to create my object and which way is considered "canonical"?
I understand that I need to run Serializer.is_valid() in order to validate the POSTed data. However, what is the difference between .data and .validated_data? They appear to be the same.
Finally, what is the "canonical" way to use a JSONField (e.g. django-jsonfield, but I'm not married to this package/implementation)? I have a model with several JSONFields and would like to use it "correctly" in DRF. I am aware of https://stackoverflow.com/a/28200902/585783, but it doesn't seem enough.
EDIT: My use case is an API POST that includes a complex JSON blob in one of the fields. I need to parse the JSON field, validate it, get/create several objects based on it, link new and existing objects, and finally store the JSON field in one of the new objects. So, I need to do custom validation for this JSON field by parsing it to python:
from django.utils.six import BytesIO
from rest_framework.parsers import JSONParser
class MySerializer(serializers.ModelSerializer):
my_json_field = JSONSerializerField()
def validate_my_json_field(self, value):
stream = BytesIO(value)
list_of_dicts = JSONParser().parse(stream)
# do lots of validation to list_of_dicts
# render list_of_dicts back to a JSON string
return validated_list_of_dicts_as_json
Now, depending on which way I choose in Concept 1, I have to parse the validated JSON again to create my objects in create(), which doesn't feel right.
Thanks in advance!
The contents of HTTP requests (POST, GET, PUT, DELETE) will always be processed by the views (View, APIView, generic views, viewsets). The serializers are just part of how these views process the requests. Serializers are the "means" to connect the View layer with the Model layer. For what serializers do specifically, please read the first paragraph of the this page of the official docs.
To answer #1: you almost always do not need to touch either unless you have a very specific use case. In those extraordinary cases:
You override Serializer.create() if you have to customize how model
instances are converted into native Python objects and vice versa. (e.g. create multiple objects)
You override ViewSet.create() if you need to customize how the actual request itself will be processed. (e.g. if there is an additional query parameter in the request, add some response headers)
To answer #2, you almost never need to use is_valid() when using generic views or ViewSets. They already do it under the hood for you. The serializer's .data and .validated_data are a bit tricky to explain. The former contains the Python datatype representation of the queryset/model instances you want to serialize, while the latter is the result of the validation process involved in checking if a Python object conforms to that particular Python datatype representation mentioned earlier, which in turn can be converted into a model instance. If that did not make sense, refer to Serializing objects and Deserializing objects.
As for #3, what do you mean by JSON field? As far as I know, Django does not have a model field called JSONField. Is this from a third party package or your own custom written model field? If so, then you will probably have to find or write a package that will let you integrate it with DRF smoothly and "correctly" whatever that means.
EDIT
Your use case is too complicated. I can only give you rough code for this one.
class MyModelSerializer(serializers.ModelSerializer):
my_json_field = JSONSerializerField()
class Meta:
model = MyModel
def validate(self, data):
# Get JSON blob
json_blob = data['my_json_field']
# Implement your own JSON blob cleanup method
# Return None if invalid
json_blob = clean_json_blob(json_blob)
# Raise HTTP 400 with your custom error message if not valid
if not json_blob:
raise serializers.ValidationError({'error': 'Your error message'})
# Reassign if you made changes and return
data['my_json_field'] = json_blob
return data
def create(self, validated_data):
json_blob = validated_data['my_json_field']
# Implement your object creation here
create_all_other_objects_from_json(json_blob)
# ...
# Then return a MyModel instance
return my_model
Given the following Django model:
from django.db import models
class SomeModel(models.Model):
date = models.DateField()
When I create an object with following code, type of date field is unicode
obj = SomeModel.objects.create(date=u'2015-05-18')
But when I get the same object from database, type of date field will be datetime.date
obj = SomeModel.objects.get(pk=obj.pk)
I know Django will transform u'2015-05-18' to a datetime.date object automatically, but why it returns the original unicode string with Model.objects.create()? is there a way to always get datetime.date from Model.objects.create() and Model.objects.get()?
Simply pass an datetime object to the create method:
from datetime import date
obj = SomeModel.objects.create(date=date(2015, 5, 18))
or
obj = SomeModel.objects.create(date=date.strftime('2015-05-18', '%y-%m-%d'))
There are two ways Django transforms the data in a model. The first, when saving the model the data is transformed to the correct data type and send to the backend. The data on the model itself is not changed. The second, when the data is loaded from the database, it is converted to the datatype that Django thinks is most appropriate.
This is by design: Django does not want to do any magic on the models itself. Models are just python class instances with attributes, so you are free to do whatever you want. This includes using the string u'2015-05-18' as a date or the string 'False' to store as True (yeah that's right).
The database cannot store dates as arbitrary data types, so in the database it is just the equivalent of a Python date object. The information that it used to be a string is lost, and when loading the data directly from the database with get(), the data is consistently converted to the most appropriate Python data type, a date.
I want to implement JsonField in my django application running postgresql.Can I also have indexing on that Json Field so that I can have Mongo like features? Do I have to make use of sqlalchemy for this or django built-in ORM is suitable for this purpose?
Thanks.
You can easily install django-jsonfield
pip install jsonfield
and use it on your field
class MyModel(models.Model):
my_json_field = JSONField()
It's just a TextField that serializes the json object to a python dictionary so no you can't have an index on it nor can you make queries against your json field.
If you are using Postgres (and don't care about compatibility with other DB engines) you should consider django's postgres fields
https://docs.djangoproject.com/en/1.9/ref/contrib/postgres/fields/
this should have much better performance than the ordinary jsonfield.
If DB compatibility is an issue and/or you want your field be readable/editable through django admin you might want to consider KeyValueField instead https://github.com/rewardz/django_model_helpers
Data is stored in DB like this
Name = abc
Age = 123
but returned to you like this
{"Name": "abc", "Age": "123"}
So if you make db_index = True you can do field__contains="Age = 123" but despite using db_index, its not fool proof because Age=1234 will also be returned by that query plus indexing text field is not usually recommended
I have a table into mysql that is the type TextField (django) by using the JSONField.
This is how my model looks
from django.db import models
from json_field import JSONField
class Model(models.Model):
obj = JSONField()
The value I send via tastypie is
json_string = '{"data":"value"}'
Into the database I can see
{"data":"value"}
But when retrive the data with curl I get something like this
"{u'data': u'value'}"
What I can do to not have the python u'field' representation into the tastypie's output ?
thanks!
I fixed this issue like so:
def dehydrate_user_inputs(self, bundle):
requirement = Requirement.objects.get(pk = bundle.obj.pk)
user_inputs = json.dumps(requirement.user_inputs)
return user_inputs
My JSONField is named user_inputs. Requirement is the model that it belongs to.
I feel weird doing a Query here when Tastypie has already done so for me, but, this works. I'd love if there are better solutions.
The error you're seeing is caused by Tastypie treating the JSONField like a TextArea and calling str() on the object JSONField returns before returning it to the caller.
Another approach is to use fields.ApiFields for the JSONField. This works because fields.ApiFields does not perform any conversions on either the way in (hydrate()) or way out (convert()). This is exactly what we want - the underlying JSONField will convert the JSON object to a string for persistence on the way in and recreate the object from the string on the way out. Thus, tastypie does not need to do anything. My code looks a bit like this (class/variable names based on OP's example) -
class JSONField(fields.apiField):
""" Wrapper over fields.apiField to make what we're doing here clear """
pass
class MyModelResource(ModelResource):
obj = JSONField('obj')
Use DictField:
obj = fields.DictField(attribute='obj')
I was running into similar problems where my unicode strings would be returned in a weird format in the API (I think the raw encoded strings were returned as opposed to the actual utf-8 characters).
Anyway instead of using the dehydrate method and re-doing the query, you are better off with a custom serializer in your resources.
This is what I used:
class JSONSerializer(Serializer):
'''using the standard json library for better unicode support,
also note django.utils.simplejson, used in the standard Tastypie serializer,
is set for depreciation'''
def to_json(self, data, options=None):
options = options or {}
data = self.to_simple(data, options)
return json.dumps(data)
then in your resources:
class PlaceResource(ModelResource):
class Meta:
queryset = Place.objects.all()
resource_name = 'place'
serializer = JSONSerializer()
I'm using JSONField in some of my Django models and would like to migrate this data from Oracle to Postgres.
So far I haven't had any luck keeping this JSON data intact when using Django's dumpdata and loaddata commands, the data is transformed into string representations of the JSON. I've yet to find a good solution to this... Ideas?
I ended up solving this problem by overriding Django's included JSON serializer, specifically the handle_field method, in a custom serializer file called custom_json_serializer.py. By doing this I can ensure that specific JSONFields stay as is, without being converted to string.
On the chance anyone else runs into this issue, these are the steps I took. I had to add this custom serializer to the settings.py file:
SERIALIZATION_MODULES = {
'custom_json': 'myapp.utils.custom_json_serializer',
}
and then call it when serializing the data from Django:
python manage.py dumpdata mymodel --format=custom_json --indent=2 --traceback > mymodel_data.json
The custom serializer looks like:
from django.core.serializers.json import Serializer as JSONSerializer
from django.utils.encoding import is_protected_type
# JSONFields that are normally incorrectly serialized as strings
json_fields = ['problem_field1', 'problem_field2']
class Serializer(JSONSerializer):
"""
A fix on JSONSerializer in order to prevent stringifying JSONField data.
"""
def handle_field(self, obj, field):
value = field._get_val_from_obj(obj)
# Protected types (i.e., primitives like None, numbers, dates,
# and Decimals) are passed through as is. All other values are
# converted to string first.
if is_protected_type(value) or field.name in json_fields:
self._current[field.name] = value
else:
self._current[field.name] = field.value_to_string(obj)
The really strange part is that before this fix some JSONFields were serializing just fine, while others were not. That is why I took the approach of specifying the fields to be handled. Now all data is serializing correctly.
I haven't used the JSONField before, but what I do is:
import json
data_structure = json.loads(myData)
Maybe that will work for what you need as well. There's likely a better way to deal with this.
EDIT: If you end up using the package json - only then is the following solution applicable.
If you are using Python 2.6 and above you can use:
import json
otherwise, you can use the simplejson that is bundled with django.utils (for Python < 2.6).
from django.utils import simplejson as json
That way you can continue to use the same package name, and take your code to Google App Engine as it supports Python 2.5.2 at the moment.