Django REST Framework POST data with file error (using ModelResource) - django

I have a really big problem trying to post data with file in a Django REST Framework app. I've created a simple application by the example at djangorestframework website. So I have the urls file:
class MyImageResource(ModelResource):
model = Image
and in urlpatters:
url(r'^image/$', ListOrCreateModelView.as_view(resource=MyImageResource)),
url(r'^image/(?P<pk>[^/]+)/$', InstanceModelView.as_view(resource=MyImageResource)),
The Image model is simple:
class Image(models.Model):
image = models.ImageField(upload_to=get_file_path)
name = models.CharField(max_length=256, blank=True)
description = models.TextField(blank=True)
Testing the REST page in browser, works perfect. Even posting data with files.
My problem is that I want to create a simple python application to post data. I've used simple urllib2, but I get 500 Internal Error or 400 Bad Request:
poza = open('poza.jpg', 'rb')
initial_data = (
{'name', 'Imagine de test REST'},
{'description', 'Dude, this is awesome'},
{'image', poza},
)
d = urllib.urlencode(initial_data)
r = urllib2.Request('http://localhost:8000/api/image/', data=d,
headers={'Content-Type':'multipart/form-data'})
resp = urllib2.urlopen(r)
code = resp.getcode()
data = resp.read()
I've tried also with MultipartPostHandler:
import MultipartPostHandler, urllib2, cookielib
cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies),
MultipartPostHandler.MultipartPostHandler)
params = {
"name":"bob",
"description":"riviera",
"content" : open("poza.jpg", "rb")
}
opener.open("http://localhost:8000/api/image/", params)
but same: 500 or 400 errors and the server (python manage.py runserver) stops with the following errors:
Exception happened during processing of request from ('127.0.0.1', 64879)
Traceback (most recent call last):
File "C:\Python27\lib\SocketServer.py", line 284, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\lib\site-packages\django\core\servers\basehttp.py", line 570
, in __init__
BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
File "C:\Python27\lib\SocketServer.py", line 641, in __init__
self.finish()
File "C:\Python27\lib\SocketServer.py", line 694, in finish
self.wfile.flush()
File "C:\Python27\lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in yo
ur host machine
If somebody have, please give me an example of posting data with files or tell me what's wrong in my posting python code. I couldn't find any more examples.
The server looks ok, I can POST data in browser. Thank you very much.

It doesn't look as if you are reading in the file, and are instead passing the file pointer?
Try:
initial_data = (
{'name', 'Imagine de test REST'},
{'description', 'Dude, this is awesome'},
{'image', poza.read()},
)

Related

Flask-Security Login Functional testing

I'm trying to do some functional testing on Flask view functions.
Currently I'm using login, logout from Flask Security module and when I try to follow the login and logout guide from flask's documentation(http://flask.pocoo.org/docs/0.12/testing/#logging-in-and-out), the 'post' of login seems to not working. I've been getting this same error when I try to post using requests module too.
My Flask-Security's login endpoint is /login_test/
Below are piece of my unit test code.
class TestUser(unittest.TestCase):
#run before each test
def setUp(self):
self.client = app.test_client()
db.create_all()
def tearDown(self):
#db.session.remove()
#DropEverything().drop_db()
pass
def login(self, email, password):
return self.client.post('/login_test/', data=dict(
email=email,
password=password
), follow_redirects=False)
def logout(self):
return self.client.get('/logout', follow_redirects=True)
def test_login_logout(self):
response = self.client.post('/login_test', data=dict(
email='admin',
password='admin'
), follow_redirects=False)
self.assertIn(b'You logged in', response.data)
The error message that I got after hitting test_login_logout is like below. The below is when I hit the url with '/login_test'
Ran 1 test in 0.187s
FAILED (failures=1)
Failure
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
yield
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/case.py", line 600, in run
testMethod()
File "/Users/genom003dm/PycharmProjects/sample_accessioning_dev/app/tests/user_management_testing.py", line 38, in test_login_logout
), follow_redirects=False)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/werkzeug/test.py", line 801, in post
return self.open(*args, **kw)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/testing.py", line 127, in open
follow_redirects=follow_redirects)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/werkzeug/test.py", line 764, in open
response = self.run_wsgi_app(environ, buffered=buffered)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/werkzeug/test.py", line 677, in run_wsgi_app
rv = run_wsgi_app(self.application, environ, buffered=buffered)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/werkzeug/test.py", line 884, in run_wsgi_app
app_rv = app(environ, start_response)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1590, in dispatch_request
self.raise_routing_exception(req)
File "/Users/genom003dm/sample_accessioning_dev_virtual_env/lib/python3.5/site-packages/flask/app.py", line 1576, in raise_routing_exception
raise FormDataRoutingRedirect(request)
flask.debughelpers.FormDataRoutingRedirect: b'A request was sent to this URL (http://localhost/login_test) but a redirect was issued automatically by the routing system to "http://localhost/login_test/". The URL was defined with a trailing slash so Flask will automatically redirect to the URL with the trailing slash if it was accessed without one. Make sure to directly send your POST-request to this URL since we can\'t make browsers or HTTP clients redirect with form data reliably or without user interaction.\n\nNote: this exception is only raised in debug mode'
If I change the URL to /login_test/ then I get HTTP 400 errors. I'm assuming that this is happening due to the fact that I'm missing form object for login? (but in this case I don't have form object because I'm trying just trying to login with post api).
I want to know is there a way to login using flask-security's /login_test/ url.
Thanks
Ok, I found an answer. The reason why I was only seeing HTTP 400 errors instead of the specifics of HTTP 400 errors are because I put the error handling on Flask app and it just showed me 400 rather than what the actual error was. Once I removed the HTTP 400 error handling, it was saying that the CSRF token was missing. So what I did was to WTF_CSRF_ENABLED = False in app config file.

"working outside of application context" in Google App Engine with Python remote API

I deployed one simple project on Google App Engine and there the project is working. Before I was able to get list of users from datastore through the remote API console , but now this no longer works. To start the remote API console I'm connecting this way (running from the project directory):
../../+/google_appengine_PYTHON_SDK/remote_api_shell.py -s project.appspot.com
I found that the easiest way to load all modules is to run:
s~project> help()
help>modules
help>quit
After that I try to get the list of users:
s~project> from app.models import User
s~project> User.query().fetch()
The last row results in this:
WARNING:root:suspended generator _run_to_list(query.py:964) raised RuntimeError(working outside of application context)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/ext/ndb/utils.py", line 142, in positional_wrapper
return wrapped(*args, **kwds)
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/ext/ndb/query.py", line 1187, in fetch
return self.fetch_async(limit, **q_options).get_result()
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/ext/ndb/tasklets.py", line 325, in get_result
self.check_success()
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/ext/ndb/tasklets.py", line 368, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/ext/ndb/query.py", line 964, in _run_to_list
batch = yield rpc
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/ext/ndb/tasklets.py", line 454, in _on_rpc_completion
result = rpc.get_result()
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/datastore/datastore_query.py", line 3014, in __query_result_hook
self.__results = self._process_results(query_result.result_list())
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/datastore/datastore_query.py", line 3047, in _process_results
for result in results]
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/datastore/datastore_rpc.py", line 185, in pb_to_query_result
return self.pb_to_entity(pb)
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/ext/ndb/model.py", line 662, in pb_to_entity
entity = modelclass._from_pb(pb, key=key, set_key=False)
File "/home/krasen/Programs/+/google_appengine_PYTHON_SDK/google/appengine/ext/ndb/model.py", line 3119, in _from_pb
ent = cls()
File "app/models.py", line 68, in __init__
if self.email == current_app.config['MAIL_ADMIN']:
File "/home/krasen/Programs/workspacePycharm/0010_1user_profile/venv/local/lib/python2.7/site-packages/werkzeug/local.py", line 338, in __getattr__
return getattr(self._get_current_object(), name)
File "/home/krasen/Programs/workspacePycharm/0010_1user_profile/venv/local/lib/python2.7/site-packages/werkzeug/local.py", line 297, in _get_current_object
return self.__local()
File "/home/krasen/Programs/workspacePycharm/0010_1user_profile/venv/local/lib/python2.7/site-packages/flask/globals.py", line 34, in _find_app
raise RuntimeError('working outside of application context')
RuntimeError: working outside of application context
I'm very new to python. I found http://kronosapiens.github.io/blog/2014/08/14/understanding-contexts-in-flask.html but couldn't understand from this how to start working inside the app context.
I have tried on linux with python versions:
2.7.6(on clean installed linux)
2.7.9
I have tried with google app engine SDK versions:
1.9.17
1.9.23
1.9.24
The problem was in the User class itself. There I have:
def __init__(self, **kwargs):
super(User, self).__init__(**kwargs)
# FLASKY_ADMIN configuration variable, so as soon as that email address appears in a registration request it can be given the correct role
if self.role is None:
print ("there is no role for this user")
if self.email == current_app.config['MAIL_ADMIN']:
self.role = ndb.Key('Role', 'administrator')
the problematic line is this line:
if self.email == current_app.config['MAIL_ADMIN']:
I don't know how to manage to make it work with this line so I've removed it and now the user is retrieved.

error: [Errno 32] Broken pipe django

Sometimes when i look at my terminal, i am seeing the below error, can anyone please let me know y it is displaying and how to avoid it ?
Exception happened during processing of request from ('127.0.0.1', 39444)
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 582, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/home/comp/Envs/proj/local/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 150, in __init__
super(WSGIRequestHandler, self).__init__(*args, **kwargs)
File "/usr/lib/python2.7/SocketServer.py", line 640, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 693, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
You get that error due two of the following reasons. You might see the same issue due to other reasons as well
1-You're missing / at the end of your url and you can fix it by fixed by added "/" to the end of the URL you request
2-You make some requests then quickly stop it. Like calling a url then cancelling the call and call another url. Check where do you make your calls (JavaScript or backend) and make sure you call the url without cancelling it.
This might be because you are using two method for inserting data into database and this cause the site to slow down.
def add_subscriber(request, email=None):
if request.method == 'POST':
email = request.POST['email_field']
e = Subscriber.objects.create(email=email).save() <====
return HttpResponseRedirect('/')
else:
return HttpResponseRedirect('/')
eg. in above function error is where arrow is pointing
the correct way to implement above is
def add_subscriber(request, email=None):
if request.method == 'POST':
email = request.POST['email_field']
e = Subscriber.objects.create(email=email)
return HttpResponseRedirect('/')
else:
return HttpResponseRedirect('/')

Python SSL web page scraping

I am trying to scrape web page using Python 2.7 and BeautifulSoup but I can't get past a protocol error which doesn't make much sense to me. This only happens on the specific website that I need to do this for: https://edd.telstra.com/telstra
The code I use just for basic test:
#! /usr/bin/python
from urllib import urlopen
from BeautifulSoup import BeautifulSoup
import re
# Copy all of the content from the provided web page
webpage = urlopen("https://edd.telstra.com/telstra/").read()
And I get the following error (running on Ubuntu 12.10):
Traceback (most recent call last):
File "e.py", line 8, in <module>
webpage = urlopen("https://edd.telstra.com/telstra/").read()
File "/usr/lib/python2.7/urllib.py", line 86, in urlopen
return opener.open(url)
File "/usr/lib/python2.7/urllib.py", line 207, in open
return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 436, in open_https
h.endheaders(data)
File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 780, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 1165, in connect
self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 143, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
IOError: [Errno socket error] [Errno 1] _ssl.c:504: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
Could someone tell me if there is some parameter that I need to specify to get this page to download in Python? It seems that this is the problem just on this web page as the code above (plus lots of other code I tried) works fine on other HTTPS/SSL pages I tried.
Thanks for any help!
I can recommend using requests lib :
def get_page(login, password):
'''Docstring
'''
url = 'https://qwe.qwe'
payload = {
'user': login,
'pass': password
}
with requests.Session() as my_session:
my_session.post(url, data=payload)
data = my_session.get(url)
return data.text
More info : http://docs.python-requests.org/en/latest/user/advanced/#session-objects

Django and Selenium Web testing error: [Errno 10054]

I'm running some basic functional web test with the Selenium web driver and I'm noticing this error in two of my functional web test cases. The test cases all pass at the end but I get this in the console:
Exception happened during processing of request from ('127.0.0.1', 1169)
data = self._sock.recv(self._rbufsize)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
Traceback (most recent call last):
File "C:\dev\django-projects\barbwire\venv\lib\site-packages\django\test\testcases.py", line 981, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\dev\django-projects\barbwire\venv\lib\site-packages\django\core\servers\basehttp.py", line 139, in __init__
super(WSGIRequestHandler, self).__init__(*args, **kwargs)
File "C:\Python27\Lib\SocketServer.py", line 638, in __init__
self.handle()
File "C:\Python27\Lib\wsgiref\simple_server.py", line 116, in handle
self.raw_requestline = self.rfile.readline()
File "C:\Python27\Lib\socket.py", line 447, in readline
data = self._sock.recv(self._rbufsize)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
Traceback (most recent call last):
File "C:\dev\django-projects\barbwire\venv\lib\site-packages\django\test\testcases.py", line 981, in _handle_request_noblock
self.process_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "C:\Python27\Lib\SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\dev\django-projects\barbwire\venv\lib\site-packages\django\core\servers\basehttp.py", line 139, in __init__
super(WSGIRequestHandler, self).__init__(*args, **kwargs)
File "C:\Python27\Lib\SocketServer.py", line 638, in __init__
self.handle()
File "C:\Python27\Lib\wsgiref\simple_server.py", line 116, in handle
self.raw_requestline = self.rfile.readline()
File "C:\Python27\Lib\socket.py", line 447, in readline
data = self._sock.recv(self._rbufsize)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 1170)
----------------------------------------
Destroying test database for alias 'default'...
Here's an example of one of the test cases:
def test_can_join_main_site(self):
self.browser.get(self.live_server_url)
self.browser.find_element_by_link_text('Register').click()
time.sleep(5)
self.browser.find_element_by_name('submit').click()
time.sleep(5)
It runs to completion but dumps the above exception. The idea is to test a simple registration page. After clicking the submit button the page re-displays and prompts the user to fill out additional form fields. As expected and everything appears to work correctly but why the errors? Am I missing something?
Replacing localhost with 127.0.0.1 did not work for me, and adding sleep just slows the test down. I was able to get rid of the error by calling refresh before quitting the browser:
from selenium import webdriver
browser = webdriver.Firefox()
# do stuff with browser
browser.refresh()
browser.quit()
In my case it was the loading of static files in <link> and <script> tags that was causing the problem. But it went away when I added refresh before quitting.
Using FireFox, I was able to resolve this by sufficiently slowing the shutdown process:
#classmethod
def tearDownClass(cls):
time.sleep(3)
cls.selenium.quit()
time.sleep(3)
super(TestClass, cls).tearDownClass()
I fixed the problem by replacing localhost with 127.0.0.1 in the URL:
url = self.live_server_url
url = url.replace('localhost', '127.0.0.1')
self.driver.get('%s%s' % (url, reverse('whatever')))
I found the solution here: https://code.djangoproject.com/ticket/15178
Django Debug Toolbar generates tons of these errors. uninstall it and try again