I have a Django view which accepts an uploaded file over PUT. I've created my own upload handler and am processing the data in chunks like so:
handler = MD5ChecksumUploadHandler()
handler.new_file(field_name="file", file_name="unknown",
content_type=request.META.get('CONTENT_TYPE', 'application/octet-stream'),
content_length=int(request.META.get('CONTENT_LENGTH', 0)))
upload_size = 0
while True:
# read the request body in chunks
chunk = request.read(handler.chunk_size)
if chunk:
handler.receive_data_chunk(chunk, start=None)
upload_size += len(chunk)
else:
break
# return the MD5ChecksumUploadedFile
return handler.file_complete(upload_size)
As I've found out, the request.read method starts reading at the beginning of the actual request and not the request body. This causes my MD5 checksums to fail, which is incidentally good, as I know that something's going wrong.
Is there a way for me to read the actual request body rather than just the raw request?
The request I was making was a bad one, that's what my problem was:
Content-MD5\n: XXXXXXXXXXXXXXXXXXXXX
Check your request if you're having this problem.
Related
Here is my Django view:
def sendMail(request):
url = 'https://example.com/example'
dictwithlist = request.FILES['dictwithlist']
parsed = json.load(dictwithlist)
// the log file shows that the lists are intact after json.loads
logip(request, json.loads(parsed))
x = requests.post(url, json.loads(clockoutJSON))
return HttpResponse(status=204)
If I just send parsed data my express server receives an empty dict, {}. When I log the json.loads(parsed) I find good data, with the lists intact. When the data gets to the other side though, the dictionaries inside the nested list are all removed, replaced by only strings of their keys.
I tried using headers as described here: Sending list of dicts as value of dict with requests.post going wrong but I just get 500 errors. I don't know if I'm formatting the headers wrong or not. (because the code has line spacing and I'm copying it)
Can anyone help me understand why this is failing? I need that list to get through with its dictionaries intact.
I believe you may need to use dumps rather than loads when sending the request:
x = requests.post(url, json.dumps(parsed))
Alternatively if you're using the python requests library you can send json as a dict as below:
response = requests.post(url=url, json=parsed)
I want to upload large files using flask. Rather than try to load the entire file into memory, I've implemented the request.stream.read() method to stream the file to disk in chunks, as per the following code, which is very similar to answers given to many similar questions I have found:
#app.route("/uploadData", methods=["POST"])
def uploadData():
filename = uuid.uuid4().hex + '.nc'
filePath = os.path.join("/tmp", filename)
with open(filePath, "wb+") as f:
chunk_size = 4096
while True:
chunk = flask.request.stream.read(chunk_size)
if len(chunk) == 0:
break
f.write(chunk)
return flask.jsonify({'success': True, 'filename': filename})
This works well, except that it "wraps" the file in post data, like the following:
------WebKitFormBoundaryoQ8GPdNkcfUNrKBd
Content-Disposition: form-data; name="inputFile"; filename="some_file_upload.nc"
Content-Type: application/x-netcdf
<Actual File content here>
------WebKitFormBoundaryoQ8GPdNkcfUNrKBd--
How can I stream the file to disk without getting the form boundary stuff?
In theory, I could call flask.request.file or the like to get the file correctly, but as that loads the entire file into memory (or more likely a temporary file), and is quite slow relative to the stream method, I don't like it as a solution.
If it makes a difference, I'm initiating the file upload using the following javascript:
var formData=new FormData($('#fileform')[0])
$.ajax({
url:'/uploadData',
data:formData,
processData:false,
contentType:false,
type:'POST'
})
EDIT: I've managed to work around the issue by using readline() rather than read(), discarding the first four lines, and then checking for chunk starting with "---" to discard the last line, which works. However, this feels both kludgy and fragile, so if there is a better solution, I would love to hear it.
My server is uWSGI and Python. I send myself an image from a file upload on the web page. How do I parse that file on the server?
I was able to handle a CSV because it's just text and I sent it by itself, but I have no idea how to handle images, or if I send the text file with other data. I'll add sample POST data to clarify when I'm back at my computer.
Part of my problem is the previous developer did some weird things with parsing POST data, so instead of being able to let uWSGI turn it into usable data, I have to do that myself in Python.
I assume you were handeling url encoded data by doing read on environ['wigs.imput'], something like this.
try:
request_body_size = int(environ.get('CONTENT_LENGTH', 0))
except (ValueError):
request_body_size = 0
request_body = environ['wsgi.input'].read(request_body_size)
dP = parse_qs(request_body)
For multipart/form-data data you need to use cgi.FieldStorage.
d = cgi.FieldStorage(environ=environ, fp=environ['wsgi.input'], keep_blank_values=True)
For Normal values in your form you can do
firstName = d.getvalue("firstName")
For the file you can get it by
file_data = d['imageFile'].file.read()
filename = d['imageFile'].filename
I have a Django App which contains a view that returns a .zip file using HttpResponse
resp = HttpResponse(s.getvalue(), content_type="application/x-zip-compressed")
resp['Content-Disposition'] = 'attachment; filename=%s' % zip_filename
return resp
The .zip is created inside the view where I also calculate its checksum.
On the client-side I use a requests.get() to get the zip file.
How can I send the checksum in the same HttpResponse along with the zip.
I tried adding the checksum to the header by
resp['hash-key'] = sha_checksum
but on the client requests.headers['hash-key'] seems to be None
How can I do this?
Edit:
As it seems my problem is located in the calculation of the hash which results in None.
The strange thing is that the same function is used on the client-side and works fine, but I guess this is another question.
Since hash-key is in the response, it sounds like this line is working.
resp['hash-key'] = sha_checksum
Try printing the value of sha_checksum before you assign it, to make sure it is not None.
Below is my python snippet code to upload a file on certain server:
import requests
url = "http://<my_url>"
files = {'file':open("<file to be uploaded>", "rb"), "name":"<name of the file>"}
r = requests.post(url, files=files)
Status code is 200, so it's OK.
My question is, on the server side, how do I save this file?
I can access 'name' on the dict thru this:
request.POST.get('name')
I can access the 'file' as well but I can't save it. I am trying this:
ufile = request.POST.get('file')
dest = open("<file on the server side>", "wb+")
for chunk in ufile.chunks():
dest.write(chunk)
dest.close()
But this is not working and it throws this exception:
unicode object has no attribute 'chunks'
I also tried:
ufile.save(<"filepath on the server side">)
But I encountered the same exception.
Hoping for any feedback! Thanks in advance!
Files are not in request.POST, they are in request.FILES.
But you probably want to read the file uploads documentation.
Try the following instead of writing chunk by chunk:
dest.write(ufile)
The ufile object is a string, not a file handle.
Had the same issue..
You should delete "Content-type" from request header, and your file will appear in request.FILES, then you can work with <TemporaryUploadedFile>
I think this answer not relevant for #jaysonpryde, but maybe will help someone else.. :)