marshalling dictionary with variable keys in flask restful /plus - flask

I am building an API using flask-restful. I am also using flask-resfulplus for generating swagger documentation. I want to return a dictionary of items where the key is going to vary depending on the item. My model looks like this:
item = api.model('Item',{
'item':fields.Integer()}) <- This is wrong
ItemModel = api.model('ItemsList', {
'_header': fields.Nested(_header),
'items':fields.Nested(item)
})
Note that I tried a few variations of this, but nothing seems to work; this is just the latest interation.
And the response I am looking for is something like this.
{
'_header':{} <-This works fine
'items': {
'item1':5,
'item2':2
}
}
Item in the items dictionary will have a different key for the item and the count as the value.
I've tried setting the item field to field.Raw() and it works fine, but then it doesn't get displayed in the swagger documentation.
Note that I do not want to return a list of dictionaries, which I have working.
Thanks

Sometimes you have your own custom formatting needs. You can subclass the fields.Raw class and implement the format function. This is especially useful when an attribute stores multiple pieces of information. e.g. a bit-field whose individual bits represent distinct values. You can use fields to multiplex a single attribute to multiple output values.
If you don’t know the name(s) of the field(s) you want to unmarshall, you can use Wildcard.
>>> from flask_restplus import fields, marshal
>>> import json
>>>
>>> wild = fields.Wildcard(fields.String)
>>> wildcard_fields = {'*': wild}
>>> data = {'John': 12, 'bob': 42, 'Jane': '68'}
>>> json.dumps(marshal(data, wildcard_fields))
>>> '{"Jane": "68", "bob": "42", "John": "12"}'
The name you give to your Wildcard acts as a real glob as shown below.
>>> from flask_restplus import fields, marshal
>>> import json
>>>
>>> wild = fields.Wildcard(fields.String)
>>> wildcard_fields = {'j*': wild}
>>> data = {'John': 12, 'bob': 42, 'Jane': '68'}
>>> json.dumps(marshal(data, wildcard_fields))
>>> '{"Jane": "68", "John": "12"}'
Note
It is important you define your Wildcard outside your model (ie. you cannot use it like this: res_fields = {'': fields.Wildcard(fields.String)})* because it has to be stateful to keep a track of what fields it has already treated.

Related

Django JsonResponse converts numbers to string

I'm using JsonResponse(status=200, data=<python_dict_with_data>) to return from my Django backend API. However, upon inspecting the result of this, all number values get converted to strings (these values are the value portion of the dict, not key). This creates a problem in the frontend receiving this response because now I have to parse them as integers to do formatting and lightweight calculations. Is there a way to prevent this conversion when returned from Django?
Or is there a way the response is parsed correctly in the frontend? I'm using Axios library in React in the frontend.
Is there a way to prevent this conversion when returned from Django?
The keys will indeed be transformed into strings, because ints as keys are illegal in JSON. Indeed, if you use a validator like JSONLint, you will see that {1: 1} is invalid JSON, whereas { "1": 1 } is valid JSON. The Python JSON encoder will thus fallback on converting the integers to strings, to still produce valid content.
If you have to do lightweight calculations, likely using these as keys is not a good idea. For example if you have data that looks like:
{ 1: 4, 2: 5 }
you might consider restructuring the data, for example to:
{ "data": [ {"key": 1, "value": 4}, {"key": 2, "value": 5} ] }
You can also return it as HTTP response, and do parsing at the JavaScript end, but likely that will only result in more trouble.
For Decimal numbers, it will also use a string. Django uses by default the DjangoJSONEncoder [Django-doc] which:
Decimal, Promise (django.utils.functional.lazy() objects), UUID:
A string representation of the object.
If we for example encode a Decimal, we see:
>>> djenc = DjangoJSONEncoder()
>>> djenc.encode({'a': Decimal('0.25')})
'{"a": "0.25"}'
You can subclass the encoder, and resolve the Decimal for example to a float, but note that this can result in loss of precision. This is exactly why a string is used: to ensure that no digits are lossed:
from django.core.serializers.json import DjangoJSONEncoder
from decimal import Decimal
class MyDjangoJSONEncoder(DjangoJSONEncoder):
def default(self, o):
if isinstance(o, Decimal):
return float(o)
return super().default(o)
this then produces:
>>> mydjenc = MyDjangoJSONEncoder()
>>> mydjenc.encode({'a': Decimal('0.25')})
'{"a": 0.25}'
You can then use this encoder in your JsonResponse:
from decimal import Decimal
def myview(request):
# …
JsonResponse(encoder=MyDjangoJSONEncoder, data={'a': Decimal('0.25')})

Using Python to retrieve website table data after filtering a specific date

I am trying to build a python script to retrieve historic wind power data from this site
I have done sort of a similar thing before. In that case the date and relevant parameters were entered explicitly in the url address.
As you can see in the previous link, e.g. the date is selected from a calendar and it is not displayed as part of the web address.
How can I use python to select an specific date and type in an Id for the fields Settlement Dateand NGC BM Unit Id respectively?
For example:
Settlement Date = 2017-08-01
NGC BM Unit Id = ANSUW-1
I don't have a MWE because I've no clue how to proceed. I was trying to reuse code from other script I'd used to get weather data:
from lxml import html
from lxml import etree
import urllib
def gettabledata():
web= urllib.urlopen("https://www.bmreports.com/bmrs/?q=actgenration/actualgeneration")
s = web.read()
html = etree.HTML(s)
but in this case it's no that simple since the filter parameters are not passed through the url.
Thanks.
I think the below script will fetch you the desired response:
import requests
payload = {"flowid":"b1610","start_date":"2017-08-01","period":"*","bmu_id":"ANSUW-1"}
headers={
"User-Agent":"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36",
"X-Requested-With":"XMLHttpRequest"
}
page = requests.get("https://www.bmreports.com/bmrs/?", params=payload, headers=headers).text
print(page)
Yes, the advice I offered in the comment was genuinely awful. Shahin is right. What I would add is that you can get the result in json which is relatively easy to process. It has taken me this long to get to this point.
>>> import requests
>>> parameter={"flowid":"b1610","start_date":"2017-08-02","period":"*","bmu_id":"ANSUW-1"}
>>> arg = 'https://www.bmreports.com/bmrs/?q=tablegen&parameter=%s' % str(parameter).replace("'",'"').replace(' ','')
>>> r = requests.get(arg)
>>> r
<Response [200]>
The result in r is json which admittedly looks horrible. However, on inspection it proves to be a series of nested dictionaries. Eventually, if you burrow in you find that 'item' is a list of 48 dictionaries from which you can easily extract whatever you might want.
>>> r.json()['responseBody']['responseList']['item'][0]
{'quantity': '1.414', 'marketGenerationBMUId': 'T_ANSUW-1', 'timeSeriesID': 'ELX-EMFIP-AGOG-TS-14842', 'powerSystemResourceType': 'Generation', 'resolution': 'PT30M', 'documentRevNum': '1', 'bMUnitID': 'T_ANSUW-1', 'registeredResourceEICCode': '48W00000ANSUW-1E', 'businessType': 'Production', 'settlementPeriod': '48', 'curveType': 'Sequential fixed size block', 'marketGenerationUnitEICCode': '48W00000ANSUW-1E', 'activeFlag': 'Y', 'nGCBMUnitID': 'ANSUW-1', 'processType': 'Realised', 'documentID': 'ELX-EMFIP-AGOG-17134615', 'marketGenerationNGCBMUId': 'ANSUW-1', 'settlementDate': '2017-08-02', 'documentType': 'Actual generation'}
>>> r.json()['responseBody']['responseList']['item'][47]
{'quantity': '1.088', 'marketGenerationBMUId': 'T_ANSUW-1', 'timeSeriesID': 'ELX-EMFIP-AGOG-TS-172', 'powerSystemResourceType': 'Generation', 'resolution': 'PT30M', 'documentRevNum': '1', 'bMUnitID': 'T_ANSUW-1', 'registeredResourceEICCode': '48W00000ANSUW-1E', 'businessType': 'Production', 'settlementPeriod': '1', 'curveType': 'Sequential fixed size block', 'marketGenerationUnitEICCode': '48W00000ANSUW-1E', 'activeFlag': 'Y', 'nGCBMUnitID': 'ANSUW-1', 'processType': 'Realised', 'documentID': 'ELX-EMFIP-AGOG-17134615', 'marketGenerationNGCBMUId': 'ANSUW-1', 'settlementDate': '2017-08-02', 'documentType': 'Actual generation'}
You can set items to the 'item' dictionary and then go from there.
>>> items = r.json()['responseBody']['responseList']['item']
>>> items[0]['settlementPeriod']
'48'
>>> items[47]['quantity']
'1.088'
Addendum: In case you don't know how I was able to get that url this is it. I used the Chrome browser. I right-clicked on any element and then on 'Inspect'. Then I clicked on the 'Network' tab in the right-hand pane, then on 'XHR'. Now I clicked on the 'View' button. As you see in the small screen view below I could just about see '?q=tablegen' in the table. I right-clicked and copied that into an editor for study.

django retrieve specific data from a dictionary database field

I have a table that contains values saved as a dictionary.
FIELD_NAME: extra_data
VALUE:
{"code": null, "user_id": "103713616419757182414", "access_token": "ya29.IwBloLKFALsddhsAAADlliOoDeE-PD_--yz1i_BZvujw8ixGPh4zH-teMNgkIA", "expires": 3599}
I need to retrieve the user_id value from the field "extra_data" only not the dictionnary like below.
event_list = Event.objects.filter(season_id=season_id, event_status_id=2).value('extra_data')
If you are storing a dictionary as text in the code you can easily convert it to a python dictionary using eval - although I don't know why you'd want to as it opens you to all sorts of potential malicious code injections.
event_list = eval(Event.objects.filter(season_id=season_id, event_status_id=2).value('extra_data'))
user_id = event_list['user_id']
print user_id
Would give:
"103713616419757182414"
Edit:
On deeper inspection , thats not a Python dictionary, you could import a JSON library to import this, or declare what null is like so:
null = None
event_list = eval(Event.objects.filter(season_id=season_id, event_status_id=2).value('extra_data'))
user_id = event_list['user_id']
Either way, the idea of storing any structured data in a django textfield is fraught with danger that will come back to bite you. The best solution is to rethink your data structures.
This method worked for me. However, this works with a json compliant string
import json
json_obj = json.loads(event_list)
dict1 = dict(json_obj)
print dict1['user_id']

How to serialize to JSON list of tuples containing Decimal in Django?

I have a Django model containing DecimalField. The resulting json should contain only data (no keys) so I'm using values_list() to convert queryset to list of Tuples:
MyModel.objects.filter(...).values_list('my_date_field','my_decimal_field').order_by('my_date_field')
Then, I need to serialize it to json... but json.dumps does not seems to be able to process the Decimal field... A lot of SO answers about that suggest to make your own encoder to use with json.dumps but those custom encoders are not recursive and seems not to work with a list of Tuple...
What I need is returning json with this format:
[[1162512000000,78.29],
[1162771200000,79.71],
[1162857600000,80.51],
[1162944000000,82.45],
[1163030400000,83.34],
[1163116800000,83.12],
[1163376000000,84.35]]
It seems to me that this should be a simple task but can't find a simple way to do it without having to parse and process everything manually...
Any suggestions?
Thanks a lot
Etienne
This should work:
import json
from decimal import Decimal as D
class DecimalJSONEncoder(json.JSONEncoder):
def default(self, o):
if type(o) == D:
# Here You can decide if You want decimal to be converted
# to string or float.
return float(o)
return super(DecimalJSONEncoder, self).default(o)
data = [[1162512000000, D(78.29)],
[1162771200000, D(79.71)],
[1162857600000, D(80.51)],
[1162944000000, D(82.45)],
[1163030400000, D(83.34)],
[1163116800000, D(83.12)],
[1163376000000, D(84.35)]]
encoder = DecimalJSONEncoder()
encoder.encode(data)
# Result:
# '[[1162512000000, 78.29], [1162771200000, 79.71], [1162857600000, 80.51], ...'

how to write a query to get find value in a json field in django

I have a json field in my database which is like
jsonfield = {'username':'chingo','reputation':'5'}
how can i write a query so that i can find if a user name exists. something like
username = 'chingo'
query = User.objects.get(jsonfield['username']=username)
I know the above query is a wrong but I wanted to know if there is a way to access it?
If you are using the django-jsonfield package, then this is simple. Say you have a model like this:
from jsonfield import JSONField
class User(models.Model):
jsonfield = JSONField()
Then to search for records with a specific username, you can just do this:
User.objects.get(jsonfield__contains={'username':username})
Since Django 1.9, you have been able to use PostgreSQL's native JSONField. This makes search JSON very simple. In your example, this query would work:
User.objects.get(jsonfield__username='chingo')
If you have an older version of Django, or you are using the Django JSONField library for compatibility with MySQL or something similar, you can still perform your query.
In the latter situation, jsonfield will be stored as a text field and mapped to a dict when brought into Django. In the database, your data will be stored like this
{"username":"chingo","reputation":"5"}
Therefore, you can simply search the text. Your query in this siutation would be:
User.objects.get(jsonfield__contains='"username":"chingo"')
2019: As #freethebees points out it's now as simple as:
User.objects.get(jsonfield__username='chingo')
But as the doc examples mention you can query deeply, and if the json is an array you can use an integer to index it:
https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/fields/#querying-jsonfield
>>> Dog.objects.create(name='Rufus', data={
... 'breed': 'labrador',
... 'owner': {
... 'name': 'Bob',
... 'other_pets': [{
... 'name': 'Fishy',
... }],
... },
... })
>>> Dog.objects.create(name='Meg', data={'breed': 'collie', 'owner': None})
>>> Dog.objects.filter(data__breed='collie')
<QuerySet [<Dog: Meg>]>
>>> Dog.objects.filter(data__owner__name='Bob')
<QuerySet [<Dog: Rufus>]>
>>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
<QuerySet [<Dog: Rufus>]>
Although this is for postgres, I believe it works the same in other DBs like MySQL
Postgres: https://docs.djangoproject.com/en/2.2/ref/contrib/postgres/fields/#querying-jsonfield
MySQL: https://django-mysql.readthedocs.io/en/latest/model_fields/json_field.html#querying-jsonfield
This usage is somewhat anti-pattern. Also, its implementation is not going to have regular performance, and perhaps is error-prone.
Normally don't use jsonfield when you need to look up through fields. Use the way the RDBMS provides or MongoDB(which internally operates on faster BSON), as Daniel pointed out.
Due to the deterministic of JSON format,
you could achieve it by using contains (regex has issue when dealing w/ multiple '\' and even slower), I don't think it's good to use username in this way, so use name instead:
def make_cond(name, value):
from django.utils import simplejson
cond = simplejson.dumps({name:value})[1:-1] # remove '{' and '}'
return ' ' + cond # avoid '\"'
User.objects.get(jsonfield__contains=make_cond(name, value))
It works as long as
the jsonfield using the same dump utility (the simplejson here)
name and value are not too special (I don't know any egde-case so far, maybe someone could point it out)
your jsonfield data is not corrupt (unlikely though)
Actually I'm working on a editable jsonfield and thinking about whether to support such operations. The negative proof is as said above, it feels like some black-magic, well.
If you use PostgreSQL you can use raw sql to solve problem.
username = 'chingo'
SQL_QUERY = "SELECT true FROM you_table WHERE jsonfield::json->>'username' = '%s'"
User.objects.extra(where=[SQL_EXCLUDE % username]).get()
where you_table is name of table in your database.
Any methods when you work with JSON like with plain text - looking like very bad way.
So, also I think that you need a better schema of database.
Here is the way I have found out that will solve your problem:
search_filter = '"username":{0}'.format(username)
query = User.objects.get(jsonfield__contains=search_filter)
Hope this helps.
You can't do that. Use normal database fields for structured data, not JSON blobs.
If you need to search on JSON data, consider using a noSQL database like MongoDB.