Django POST multipart/form-data unexpectedly parsing field as array - django

I am testing an endpoint in the following manner:
from rest_framework.test import APIClient
from django.urls import reverse
import json
client = APIClient()
response = client.post(list_url, {'name': 'Zagyg Co'})
I find that the model object is being created with a name of [Zagyg Co] instead of Zagyg Co.
Inspecting the request object reveals the following:
self._request.META['CONTENT_TYPE']
#=> 'multipart/form-data; boundary=BoUnDaRyStRiNg; charset=utf-8'
self._request.body
#=> b'--BoUnDaRyStRiNg\r\nContent-Disposition: form-data; name="name"\r\n\r\nZagyg Co\r\n--BoUnDaRyStRiNg--\r\n'
self._request.POST
#=> <QueryDict: {'name': ['Zagyg Co']}>
Using JSON like so:
response = client.post(
list_url,
json.dumps({'name': 'Zagyg Co'}),
content_type='application/json',
)
sets the name correctly. Why is this so?

request.data is a Django QueryDict. When the data is sent as a multipart form it handles potential multiple values of the same field by putting it in a list.
Using its dict method or using dictionary access syntax returns the last value stored in the relevant key(s):
request.data['name']
#=> 'Zagyg Co'
request.dict()
#=> {'name': 'Zagyg Co'}
Which is great if it's guaranteed that each key has a single value. For list values there's getlist:
request.data.getlist('name')
#=> ['Zagyg Co']
For mixes of keys with single and multiple values it seems like manual parsing is required.

Related

how to access values in python from json

I'm new to Python but required to write a script for API so trying to read the response from API and put it in a file, version is python2.7
Following is the code
import requests
import json
#URL = "someurl"
# sending get request and saving the response as response object
#response = requests.get(url = URL)
#print(response.status_code)
#print(response.content)
items = json.loads('{"batch_id":"5d83a2d317cb4","names":
{"19202":"text1","19203":"text2"}}')
print(items['names'])
for item in items['names']:
print(item)
Current output is
19202
19203
But I would like to pick text1,text2 and write to a file, can anyone help how to get those values
items is a dictionary. items['names'] is also a dictionary. for item in items['names']: will iterate over keys not values. The item will hold key in the dictionary.
To access the value in that key-value pair, you have to use print items['names'][item] instead of print item. Your code should look something like below.
import requests
import json
#URL = "someurl"
# sending get request and saving the response as response object
#response = requests.get(url = URL)
#print(response.status_code)
#print(response.content)
items = json.loads('{"batch_id":"5d83a2d317cb4","names":
{"19202":"text1","19203":"text2"}}')
print(items['names'])
for item in items['names']:
print(items['names'][item])
>>> print(list(items['names'].values()))
['text1', 'text2']
So you can do like this:
for item in items['names'].values():
print(item)

Setting a value for when value isn't present

Using request.POST.get you can do this:
token = request.POST.get('token', False)
Is there a way to do the same thing using
request.data['token']
Figured it out.
Axios was serializing the object to JSON by default while I needed to send the data in the `application/x-www-form-urlencoded' format. I used the 'qs' library to achieve this.
var qs = require('qs');
axios.post('/foo', qs.stringify({ 'bar': 123 });
Thought it was a Django issue but it was actually Axios.
The HttpRequest object doesn't have a 'data' attribute, but depending on what you are looking for there are other attributes, such as 'body', 'FILES', 'META'. Full details in the documentation:
https://docs.djangoproject.com/en/2.2/ref/request-response/

Tweepy location on Twitter API filter always throws 406 error

I'm using the following code (from django management commands) to listen to the Twitter stream - I've used the same code on a seperate command to track keywords successfully - I've branched this out to use location, and (apparently rightly) wanted to test this out without disrupting my existing analysis that's running.
I've followed the docs and have made sure the box is in Long/Lat format (in fact, I'm using the example long/lat from the Twitter docs now). It looks broadly the same as the question here, and I tried using their version of the code from the answer - same error. If I switch back to using 'track=...', the same code works, so it's a problem with the location filter.
Adding a print debug inside streaming.py in tweepy so I can see what's happening, I print out the self.parameters self.url and self.headers from _run, and get:
{'track': 't,w,i,t,t,e,r', 'delimited': 'length', 'locations': '-121.7500,36.8000,-122.7500,37.8000'}
/1.1/statuses/filter.json?delimited=length and
{'Content-type': 'application/x-www-form-urlencoded'}
respectively - seems to me to be missing the search for location in some way shape or form. I don't believe I'm/I'm obviously not the only one using tweepy location search, so think it's more likely a problem in my use of it than a bug in tweepy (I'm on 2.3.0), but my implementation looks right afaict.
My stream handling code is here:
consumer_key = 'stuff'
consumer_secret = 'stuff'
access_token='stuff'
access_token_secret_var='stuff'
import tweepy
import json
# This is the listener, resposible for receiving data
class StdOutListener(tweepy.StreamListener):
def on_data(self, data):
# Twitter returns data in JSON format - we need to decode it first
decoded = json.loads(data)
#print type(decoded), decoded
# Also, we convert UTF-8 to ASCII ignoring all bad characters sent by users
try:
user, created = read_user(decoded)
print "DEBUG USER", user, created
if decoded['lang'] == 'en':
tweet, created = read_tweet(decoded, user)
print "DEBUG TWEET", tweet, created
else:
pass
except KeyError,e:
print "Error on Key", e
pass
except DataError, e:
print "DataError", e
pass
#print user, created
print ''
return True
def on_error(self, status):
print status
l = StdOutListener()
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret_var)
stream = tweepy.Stream(auth, l)
#locations must be long, lat
stream.filter(locations=[-121.75,36.8,-122.75,37.8], track='twitter')
The issue here was the order of the coordinates.
Correct format is:
SouthWest Corner(Long, Lat), NorthEast Corner(Long, Lat). I had them transposed. :(
The streaming API doesn't allow to filter by location AND keyword simultaneously.
you must refer to this answer i had the same problem earlier
https://stackoverflow.com/a/22889470/4432830

How to URL encode all my data

My gateway say I need to URL encode all my data, in Django how is this possible?
requests.post(default_gateway.keyword_forwarding_url, data=raw_data,
stream=True, verify=True)
I have tried
import urllib
requests.post(default_gateway.keyword_forwarding_url, data=urllib.urlencode(raw_data),
stream=True, verify=True)
Your data needs to be passed to urlencode() as a dict or in a sequence of paired tuples:
import urllib
formatted_raw_data = {raw_data[0]: raw_data[1], raw_data[2]: raw_data[3]} // or however it needs to be formatted
requests.post(default_gateway.keyword_forwarding_url, data=urllib.urlencode(formatted_raw_data),
stream=True, verify=True)

Decoding utf-8 in Django while using unicode_literals in Python 2.7

I'm using Django to manage a Postgres database. I have a value stored in the database representing a city in Spain (Málaga). My Django project uses unicode strings for everything by putting from __future__ import unicode_literals at the beginning of each of the files I created.
I need to pull the city information from the database and send it to another server using an XML request. There is logging in place along the way so that I can observe the flow of data. When I try and log the value for the city I get the following traceback:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 1: ordinal not in range(128)
Here is the code I use to log the values I'm passing.
def createXML(self, dict):
"""
.. method:: createXML()
Create a single-depth XML string based on a set of tuples
:param dict: Set of tuples (simple dictionary)
"""
xml_string = ''
for key in dict:
self.logfile.write('\nkey = {0}\n'.format(key))
if (isinstance(dict[key], basestring)):
self.logfile.write('basestring\n')
self.logfile.write('value = {0}\n\n'.format(dict[key].decode('utf-8')))
else:
self.logfile.write('value = {0}\n\n'.format(dict[key]))
xml_string += '<{0}>{1}</{0}>'.format(key, dict[key])
return xml_string
I'm basically saving all the information I have in a simple dictionary and using this function to generate an XML formatted string - this is beyond the scope of this question.
The error I am getting had me wondering what was actually being saved in the database. I have verified the value is utf-8 encoded. I created a simple script to extract the value from the database, decode it and print it to the screen.
from __future__ import unicode_literals
import psycopg2
# Establish the database connection
try:
db = psycopg2.connect("dbname = 'dbname' \
user = 'user' \
host = 'IP Address' \
password = 'password'")
cur = db.cursor()
except:
print "Unable to connect to the database."
# Get database info if any is available
command = "SELECT state FROM table WHERE id = 'my_id'"
cur.execute(command)
results = cur.fetchall()
state = results[0][0]
print "my state is {0}".format(state.decode('utf-8'))
Result: my state is Málaga
In Django I'm doing the following to create the HTTP request:
## Create the header
http_header = "POST {0} HTTP/1.0\nHost: {1}\nContent-Type: text/xml\nAuthorization: Basic {2}\nContent-Length: {3}\n\n"
req = http_header.format(service, host, auth, len(self.xml_string)) + self.xml_string
Can anyone help me correct the problem so that I can write this information to the database and be able to create the req string to send to the other server?
Am I getting this error as a result of how Django is handling this? If so, what is Django doing? Or, what am I telling Django to do that is causing this?
EDIT1:
I've tried to use Django's django.utils.encoding on this state value as well. I read a little from saltycrane about a possible hiccup Djano might have with unicode/utf-8 stuff.
I tried to modify my logging to use the smart_str functionality.
def createXML(self, dict):
"""
.. method:: createXML()
Create a single-depth XML string based on a set of tuples
:param dict: Set of tuples (simple dictionary)
"""
xml_string = ''
for key in dict:
if (isinstance(dict[key], basestring)):
if (key == 'v1:State'):
var_str = smart_str(dict[key])
for index in range(0, len(var_str)):
var = bin(ord(var_str[index]))
self.logfile.write(var)
self.logfile.write('\n')
self.logfile.write('{0}\n'.format(var_str))
xml_string += '<{0}>{1}</{0}>'.format(key, dict[key])
return xml_string
I'm able to write the correct value to the log doing this but I narrowed down another possible problem with the .format() string functionality in Python. Of course my Google search of python format unicode had the first result as Issue 7300, which states that this is a known "issue" with Python 2.7.
Now, from another stackoverflow post I found a "solution" that does not work in Django with the smart_str functionality (or at least I've been unable to get them to work together).
I'm going to continue digging around and see if I can't find the underlying problem - or at least a work-around.
EDIT2:
I found a work-around by simply concatenating strings rather than using the .format() functionality. I don't like this "solution" - it's ugly, but it got the job done.
def createXML(self, dict):
"""
.. method:: createXML()
Create a single-depth XML string based on a set of tuples
:param dict: Set of tuples (simple dictionary)
"""
xml_string = ''
for key in dict:
xml_string += '<{0}>'.format(key)
if (isinstance(dict[key], basestring)):
xml_string += smart_str(dict[key])
else:
xml_string += str(dict[key])
xml_string += '<{0}>'.format(key)
return xml_string
I'm going to leave this question unanswered as I'd love to find a solution that lets me use .format() the way it was intended.
This is correct approach (problem was with opening file. With UTF-8 You MUST use codecs.open() :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import codecs
class Writer(object):
logfile = codecs.open("test.log", "w", 'utf-8')
def createXML(self, dict):
xml_string = ''
for key, value in dict.iteritems():
self.logfile.write(u'\nkey = {0}\n'.format(key))
if (isinstance(value, basestring)):
self.logfile.write(u'basestring\n')
self.logfile.write(u'value = {0}\n\n'.format( value))
else:
self.logfile.write(u'value = {0}\n\n'.format( value ))
xml_string += u'<{0}>{1}</{0}>'.format(key, value )
return xml_string
And this is from python console:
In [1]: from test import Writer
In [2]: d = { 'a' : u'Zażółć gęślą jaźń', 'b' : u'Och ja Ci zażółcę' }
In [3]: w = Writer()
In [4]: w.createXML(d)
Out[4]: u'<a>Za\u017c\xf3\u0142\u0107 g\u0119\u015bl\u0105 ja\u017a\u0144</a><b>Och ja Ci za\u017c\xf3\u0142c\u0119</b>'
And this is test.log file:
key = a
basestring
value = Zażółć gęślą jaźń
key = b
basestring
value = Och ja Ci zażółcę