I'm saving the following simple, valid JSON object to a model in my Django app:
{
"start_date": 1311471044.24338
"post_count": 25
}
The model looks like this:
from django.db import models
from django_extensions.db.fields import json as json
class UserProfile(models.Model):
data = json.JSONField()
To read the posted data, I basically do
posted_data = request.FILES.get('posted_data').read()
json_data = simplejson.loads(posted_data)
The data then contains the expect type (float)
logging.debug( "start_date type: " + str(type(json_data.get('start_date'))))
logging.debug( "post_count type: " + str(type(json_data.get('post_count'))))
>> 2011-07-24 10:03:01,636 DEBUG start_date type: <type 'float'>
>> 2011-07-24 10:03:01,636 DEBUG post_count type: <type 'int'>
I then save the data like this:
user_profile.data = json_data
user_profile.save()
and then I read the data back, integers are fine, but floating point numbers are quoted, for example:
print user_profile.data
{
"post_count : 25
"start_date": "1311471044.24338"
}
How can I prevent floating point numbers from being turned to strings unnecessarily?
Edit:
May have found an explanation here: Rails JSON Serialization of Decimal adds Quotes
I still would be interested in other explanations though.
This answer seems to be the best explanation:
The only "safe" way to hand decimals from language A to language B is to use a String. If your json contains "rating": 98.79999999999999 it will probably be converted to 98.79999999999998 by your JavaScript runtime.
Related
I'm using JsonResponse(status=200, data=<python_dict_with_data>) to return from my Django backend API. However, upon inspecting the result of this, all number values get converted to strings (these values are the value portion of the dict, not key). This creates a problem in the frontend receiving this response because now I have to parse them as integers to do formatting and lightweight calculations. Is there a way to prevent this conversion when returned from Django?
Or is there a way the response is parsed correctly in the frontend? I'm using Axios library in React in the frontend.
Is there a way to prevent this conversion when returned from Django?
The keys will indeed be transformed into strings, because ints as keys are illegal in JSON. Indeed, if you use a validator like JSONLint, you will see that {1: 1} is invalid JSON, whereas { "1": 1 } is valid JSON. The Python JSON encoder will thus fallback on converting the integers to strings, to still produce valid content.
If you have to do lightweight calculations, likely using these as keys is not a good idea. For example if you have data that looks like:
{ 1: 4, 2: 5 }
you might consider restructuring the data, for example to:
{ "data": [ {"key": 1, "value": 4}, {"key": 2, "value": 5} ] }
You can also return it as HTTP response, and do parsing at the JavaScript end, but likely that will only result in more trouble.
For Decimal numbers, it will also use a string. Django uses by default the DjangoJSONEncoder [Django-doc] which:
Decimal, Promise (django.utils.functional.lazy() objects), UUID:
A string representation of the object.
If we for example encode a Decimal, we see:
>>> djenc = DjangoJSONEncoder()
>>> djenc.encode({'a': Decimal('0.25')})
'{"a": "0.25"}'
You can subclass the encoder, and resolve the Decimal for example to a float, but note that this can result in loss of precision. This is exactly why a string is used: to ensure that no digits are lossed:
from django.core.serializers.json import DjangoJSONEncoder
from decimal import Decimal
class MyDjangoJSONEncoder(DjangoJSONEncoder):
def default(self, o):
if isinstance(o, Decimal):
return float(o)
return super().default(o)
this then produces:
>>> mydjenc = MyDjangoJSONEncoder()
>>> mydjenc.encode({'a': Decimal('0.25')})
'{"a": 0.25}'
You can then use this encoder in your JsonResponse:
from decimal import Decimal
def myview(request):
# …
JsonResponse(encoder=MyDjangoJSONEncoder, data={'a': Decimal('0.25')})
I am building an API using flask-restful. I am also using flask-resfulplus for generating swagger documentation. I want to return a dictionary of items where the key is going to vary depending on the item. My model looks like this:
item = api.model('Item',{
'item':fields.Integer()}) <- This is wrong
ItemModel = api.model('ItemsList', {
'_header': fields.Nested(_header),
'items':fields.Nested(item)
})
Note that I tried a few variations of this, but nothing seems to work; this is just the latest interation.
And the response I am looking for is something like this.
{
'_header':{} <-This works fine
'items': {
'item1':5,
'item2':2
}
}
Item in the items dictionary will have a different key for the item and the count as the value.
I've tried setting the item field to field.Raw() and it works fine, but then it doesn't get displayed in the swagger documentation.
Note that I do not want to return a list of dictionaries, which I have working.
Thanks
Sometimes you have your own custom formatting needs. You can subclass the fields.Raw class and implement the format function. This is especially useful when an attribute stores multiple pieces of information. e.g. a bit-field whose individual bits represent distinct values. You can use fields to multiplex a single attribute to multiple output values.
If you don’t know the name(s) of the field(s) you want to unmarshall, you can use Wildcard.
>>> from flask_restplus import fields, marshal
>>> import json
>>>
>>> wild = fields.Wildcard(fields.String)
>>> wildcard_fields = {'*': wild}
>>> data = {'John': 12, 'bob': 42, 'Jane': '68'}
>>> json.dumps(marshal(data, wildcard_fields))
>>> '{"Jane": "68", "bob": "42", "John": "12"}'
The name you give to your Wildcard acts as a real glob as shown below.
>>> from flask_restplus import fields, marshal
>>> import json
>>>
>>> wild = fields.Wildcard(fields.String)
>>> wildcard_fields = {'j*': wild}
>>> data = {'John': 12, 'bob': 42, 'Jane': '68'}
>>> json.dumps(marshal(data, wildcard_fields))
>>> '{"Jane": "68", "John": "12"}'
Note
It is important you define your Wildcard outside your model (ie. you cannot use it like this: res_fields = {'': fields.Wildcard(fields.String)})* because it has to be stateful to keep a track of what fields it has already treated.
I have a Django model containing DecimalField. The resulting json should contain only data (no keys) so I'm using values_list() to convert queryset to list of Tuples:
MyModel.objects.filter(...).values_list('my_date_field','my_decimal_field').order_by('my_date_field')
Then, I need to serialize it to json... but json.dumps does not seems to be able to process the Decimal field... A lot of SO answers about that suggest to make your own encoder to use with json.dumps but those custom encoders are not recursive and seems not to work with a list of Tuple...
What I need is returning json with this format:
[[1162512000000,78.29],
[1162771200000,79.71],
[1162857600000,80.51],
[1162944000000,82.45],
[1163030400000,83.34],
[1163116800000,83.12],
[1163376000000,84.35]]
It seems to me that this should be a simple task but can't find a simple way to do it without having to parse and process everything manually...
Any suggestions?
Thanks a lot
Etienne
This should work:
import json
from decimal import Decimal as D
class DecimalJSONEncoder(json.JSONEncoder):
def default(self, o):
if type(o) == D:
# Here You can decide if You want decimal to be converted
# to string or float.
return float(o)
return super(DecimalJSONEncoder, self).default(o)
data = [[1162512000000, D(78.29)],
[1162771200000, D(79.71)],
[1162857600000, D(80.51)],
[1162944000000, D(82.45)],
[1163030400000, D(83.34)],
[1163116800000, D(83.12)],
[1163376000000, D(84.35)]]
encoder = DecimalJSONEncoder()
encoder.encode(data)
# Result:
# '[[1162512000000, 78.29], [1162771200000, 79.71], [1162857600000, 80.51], ...'
I have a django application with tastypie. One of the models in my app has a DecimalField. When I get a response from the API in JSON format, all the decimal fields appear as strings instead of numbers:
For example I get:
objects: [
{
id: "1",
my_decimal_field: "84.54"
}
instead of
objects: [
{
id: "1"
my_decimal_field: 84.54
}
This also happens with the id field.
¿Any thoughts?
In JavaScript, JSON decodes to double-precision floating point format, which causes a loss of precision. Decimal objects are encoded to string to maintain precision.
If you want to encode to JSON number format, you can use a FloatField.
I have a json field in my database which is like
jsonfield = {'username':'chingo','reputation':'5'}
how can i write a query so that i can find if a user name exists. something like
username = 'chingo'
query = User.objects.get(jsonfield['username']=username)
I know the above query is a wrong but I wanted to know if there is a way to access it?
If you are using the django-jsonfield package, then this is simple. Say you have a model like this:
from jsonfield import JSONField
class User(models.Model):
jsonfield = JSONField()
Then to search for records with a specific username, you can just do this:
User.objects.get(jsonfield__contains={'username':username})
Since Django 1.9, you have been able to use PostgreSQL's native JSONField. This makes search JSON very simple. In your example, this query would work:
User.objects.get(jsonfield__username='chingo')
If you have an older version of Django, or you are using the Django JSONField library for compatibility with MySQL or something similar, you can still perform your query.
In the latter situation, jsonfield will be stored as a text field and mapped to a dict when brought into Django. In the database, your data will be stored like this
{"username":"chingo","reputation":"5"}
Therefore, you can simply search the text. Your query in this siutation would be:
User.objects.get(jsonfield__contains='"username":"chingo"')
2019: As #freethebees points out it's now as simple as:
User.objects.get(jsonfield__username='chingo')
But as the doc examples mention you can query deeply, and if the json is an array you can use an integer to index it:
https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/fields/#querying-jsonfield
>>> Dog.objects.create(name='Rufus', data={
... 'breed': 'labrador',
... 'owner': {
... 'name': 'Bob',
... 'other_pets': [{
... 'name': 'Fishy',
... }],
... },
... })
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': None})
>>> Dog.objects.filter(data__breed='collie')
<QuerySet [<Dog: Meg>]>
>>> Dog.objects.filter(data__owner__name='Bob')
<QuerySet [<Dog: Rufus>]>
>>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
<QuerySet [<Dog: Rufus>]>
Although this is for postgres, I believe it works the same in other DBs like MySQL
Postgres: https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/fields/#querying-jsonfield
MySQL: https://django-mysql.readthedocs.io/en/latest/model_fields/json_field.html#querying-jsonfield
This usage is somewhat anti-pattern. Also, its implementation is not going to have regular performance, and perhaps is error-prone.
Normally don't use jsonfield when you need to look up through fields. Use the way the RDBMS provides or MongoDB(which internally operates on faster BSON), as Daniel pointed out.
Due to the deterministic of JSON format,
you could achieve it by using contains (regex has issue when dealing w/ multiple '\' and even slower), I don't think it's good to use username in this way, so use name instead:
def make_cond(name, value):
from django.utils import simplejson
cond = simplejson.dumps({name:value})[1:-1] # remove '{' and '}'
return ' ' + cond # avoid '\"'
User.objects.get(jsonfield__contains=make_cond(name, value))
It works as long as
the jsonfield using the same dump utility (the simplejson here)
name and value are not too special (I don't know any egde-case so far, maybe someone could point it out)
your jsonfield data is not corrupt (unlikely though)
Actually I'm working on a editable jsonfield and thinking about whether to support such operations. The negative proof is as said above, it feels like some black-magic, well.
If you use PostgreSQL you can use raw sql to solve problem.
username = 'chingo'
SQL_QUERY = "SELECT true FROM you_table WHERE jsonfield::json->>'username' = '%s'"
User.objects.extra(where=[SQL_EXCLUDE % username]).get()
where you_table is name of table in your database.
Any methods when you work with JSON like with plain text - looking like very bad way.
So, also I think that you need a better schema of database.
Here is the way I have found out that will solve your problem:
search_filter = '"username":{0}'.format(username)
query = User.objects.get(jsonfield__contains=search_filter)
Hope this helps.
You can't do that. Use normal database fields for structured data, not JSON blobs.
If you need to search on JSON data, consider using a noSQL database like MongoDB.