So I have a Django app, which as part of its functionality makes a request (using the requests module) to another server. What I want to do is have a server available for unittesting which gives me canned responses to test requests from the Django app (allowing to test how Django handles the different potential responses).
An example of the code would be:
payload = {'access_key': key,
'username': name}
response = requests.get(downstream_url, params=payload)
# Handle response here ...
I've read that you can use SimpleHTTPServer to accomplish this, but I'm not sure of how I use it to this end, any thoughts would be much appreciated!
Use the mock module.
from mock import patch, MagicMock
#patch('your.module.requests')
def test_something(self, requests_mock):
response = MagicMock()
response.json.return_value = {'key': 'value'}
requests_mock.get.return_value = response
…
requests_mock.get.assert_called_once_with(…)
response.json.assert_called_once()
Much more examples in the docs.
You don't need to (and should not) test the code that makes the request. You want to mock out that part and focus on testing the logic that handles the response.
Related
I'm trying to test drf-social-oauth2's integration with Google via python manage.py test with the following test class:
class DRFSocialOAuthIntegrationTest(TestCase):
def setUp(self):
self.test_user = UserModel.objects.create_user("test_user", "test#user.com", TEST_USER_PASSWORD)
self.application = Application(
name="Test Application",
redirect_uris="",
user=self.test_user,
client_type=Application.CLIENT_CONFIDENTIAL,
authorization_grant_type=Application.GRANT_PASSWORD, # https://github.com/jazzband/django-oauth-toolkit/blob/master/oauth2_provider/models.py
)
self.application.save()
# Every test needs a client.
self.client = Client()
def tearDown(self):
self.application.delete()
self.test_user.delete()
def test_drf_social_oauth2_integration(self):
'''Following testing steps found at curl -X POST -d "grant_type=convert_token&client_id=<django-oauth-generated-client_id>&client_secret=<django-oauth-generated-client_secret>&backend=google-oauth2&token=<google_token>" http://localhost:8000/auth/convert-token'''
def convert_google_token(self):
'''
Convert Google token to our access token
curl -X POST -d "grant_type=convert_token&client_id=<client_id>&client_secret=<client_secret>&backend=google-oauth2&token=<google_token>" http://localhost:8000/auth/convert-token
'''
return self.client.post(
'/auth/convert-token',
{
'grant_type': 'convert_token',
'client_id': self.application.client_id,
'client_secret': self.application.client_secret,
'backend': 'google-oauth2',
'token': <google_token>
}
)
That seems to work fine, but I have to keep feeding it the google_token by manually navigating to https://developers.google.com/oauthplayground/?code=<my_code>&scope=email+profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+openid&authuser=0&prompt=consent#:
Once there, I click on the 'Exchange authorization code for tokens' button, get the access token, and paste that access token into the tokenparameter of the request inside convert_google_token(). This doesn't feel very automated. I'm not sure if I should just click the checkbox so that the access token is refreshed automatically and therefore never have to edit it in convert_google_token(). Or maybe I'm supposed to get that access token programmatically. But I believe that would entail getting the authorization code first, which would have to also be programmatically since it is a one-time code, right?
So in order to get that authorization code, I was trying to parse the html response from developers.google.com/oauthplayground/ like so:
def get_google_authorization_code_html():
import requests
return requests.post(
'https://developers.google.com/oauthplayground/',
json={
'code': <my_code>,
'scope': "email+profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+openid",
'authuser': "0",
'prompt': "consent"
}
)
response = get_google_authorization_code_html() #;print(f'Google authorization code html returned: {response.content=}')
self.assertEqual(response.status_code, 200)
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.content) ;print('Google authorization code html return: '); print(soup.prettify())
However, the input element that in the browser shows the authorization code is actually blank in the html response, so I can't retrieve it from there. Would this even be the proper way to go about this? I may be missing key knowledge as I'm pretty new to OAuth2, Django, and testing. I've been trying to read up on it and particularly diagrams such as the one found in this Medium article make me think I'm more or less on the right track, but I'm not sure.
You are making a trivial mistake there. Whenever you need the response of a third-party API for testing purposes, you have to mock the response of the API and also/perhaps mock some of your code as well. In this case, you must mock the drf convert-token, otherwise, you will have to keep reading google's token (and this is not ideal).
Background
I have a service A accessible with HTTP requests. And I have other services that want to invoke these APIs.
Problem
When I test service A's APIs with POSTMAN, every request works fine. But when I user python's requests library to make these request, there is one PUT method that just won't work. For some reason, the PUT method being called cannot receive the data (HTTP body) at all, though it can receive headers. On the other side, the POST method called in the same manner receives the data perfectly.
I managed to achieve my goal simply by using httplib library instead, but I am still quite baffled by what exactly happened here.
The Crime Scene
Route 1:
#app.route("/private/serviceA", methods = ['POST'])
#app.route("/private/serviceA/", methods = ['POST'])
def A_create():
# request.data contains correct data that can be read with request.get_json()
Route 2:
#app.route("/private/serviceA/<id>", methods = ['PUT'])
#app.route("/private/serviceA/<id>/", methods = ['PUT'])
def A_update(id):
# request.data is empty, though request.headers contains headers I passed in
# This happens when sending the request with Python requests library, but not when sending with httplib library or with POSTMAN
# Also, data comes in fine when all other routes are commented out
# Unless all other routes are commented out, this happens even when the function body has only one line printing request.data
Route 3:
#app.route("/private/serviceA/schema", methods = ['PUT'])
def schema_update_column():
# This one again works perfectly fine
Using POSTMAN:
Using requests library from another service:
#app.route("/public/serviceA/<id>", methods = ['PUT'])
def A_update(id):
content = request.get_json()
headers = {'content-type': 'application/json'}
response = requests.put('%s:%s' % (router_config.HOST, serviceA_instance_id) + '/private/serviceA/' + str(id), data=json.dumps(content), headers = headers)
return Response(response.content, mimetype='application/json', status=response.status_code)
Using httplib library from another service:
#app.route('/public/serviceA/<id>', methods=['PUT'])
def update_course(id):
content= request.get_json()
headers = {'content-type': 'application/json'}
conn = httplib.HTTPConnection('%s:%s' % (router_config.HOST, serviceA_instance_id))
conn.request("PUT", "/private/serviceA/%s/" % id, json.dumps(content), headers)
return str(conn.getresponse().read())
Questions
1. What am I doing wrong for the route 2?
2. For route 2, the handler doesn't seem to be executed when either handler is commented out, which also confuses me. Is there something important about Flask that I'm not aware of?
Code Repo
Just in case some nice ppl are interested enough to look at the messy undocumented code...
https://github.com/fantastic4ever/project1
The serviceA corresponds to course service (course_flask.py), and the service calling it corresponds to router service (router.py).
The version that was still using requests library is 747e69a11ed746c9e8400a8c1e86048322f4ec39.
In your use of the requests library, you are using requests.post, which is sending a POST request. If you use requests.put then you would send a PUT request. That could be the issue.
Request documentation
In my settings.py, I put:
EMAIL_BACKEND = 'mailer.backend.DbBackend'
So even when importing from from django.core.mail import send_mail, the send_mail function still queues up the email in the database instead of sending it immediately.
It works just fine when actually running the website, but when testing the website, and accessing some webpages that trigger emails, emails are no longer queued anymore:
def test_something(self):
...
# Check no emails are actually sent yet
self.assertEquals(len(mail.outbox), 0) # test fails here -- 2 != 0
# Check queued emails.
messages = Message.objects.all()
self.assertEquals(messages.count(), 2) # test would also fail here -- 0 != 2
...
How come it doesn't seem to be using the backend when it is testing? (importing send_mail from mailer itself gets the tests to pass, but I can't really change the imports of other mailing apps like django-templated-email)
According to this question django overrides the setting.EMAIL_BACKEND when testing to 'django.core.mail.backends.locmem.EmailBackend'. It's also in the django docs here.
To properly test email with django-mailer, you need to override two settings:
Make the tests to use the django-mailer backend
Make the djano-mailer backend to use the test backend
If you don't set the django-mailer backend (number 2), your tests will try to send the email for real.
You also need to simulate running django-mailer's send_mail management command so that you can check mail.outbox for the correct email.
Here's an example of how to setup a test method:
from mailer.engine import send_all
#override_settings(EMAIL_BACKEND='mailer.backend.DbBackend')
#override_settings(MAILER_EMAIL_BACKEND='django.core.mail.backends.locmem.EmailBackend')
def test_email(self):
# Code that generates email goes here.
send_all() # Simulates running django-mailer's send_mail management command.
# Code to check the email in mail.outbox goes here.
This strategy makes your tests specific to django-mailer which you don't always want or need. I personally only use this setup when I'm testing specific functionality enabled by django-mailer. Otherwise, I use the default test email backend setup by django.
If you really want have sending of emails (like default) via SMTP in django tests use the decorator:
from django.test.utils import override_settings
#override_settings(EMAIL_BACKEND='django.core.mail.backends.smtp.EmailBackend')
class TestEmailVerification(TestCase):
...
Try the following:
django.core.mail.backends.console.EmailBackend
I have an oauth flow in which a user grants access to a certain scope and then my application can do stuff. For this to work, I need an access token with the defined scope.
I implemented this (with the django allauth package) and it works great. But...
I would like to test it.
This is what I have so far (request package is like an urllib on steroids):
login = self.client.login(username='test_user', password='test')
self.assertTrue(login)
resp = self.client.post(reverse('oauth_login'))
self.assertEqual(resp.status_code, 302)
payload = {'session_key': 'email', 'session_password': 'pw', }
resp2 = requests.post(resp['location'], data=payload)
resp3 = self.client.get(reverse('do_stuff_with_access_token'))
self.assertEqual(resp.status_code, 302)
The issue here is that I do not get the access token in my request variables. My guess is that I am going out of the scope of the application and Django does not get the variable in its request scope.
How can you test this in an elegant manner? Mocking an access_token seems a bit wrong to me. I am now trying to go Selenium for filling in the form, but even that is not really a success so far...
Thanks!
Kudos to Mark!
To help anyone on their way.
Mocking worked out. In my case (django 1.4) you need to add your tokens to the session. A lot of different advices can be found on the net, but I like simple things and this works in Django at least with the test suite:
session = self.client.session
session['request_token'] = {...}
session['access_token'] = {...}
session.save()
I faced the same problem and found the following solved my dilemma:
user = User.objects.get(username='lauren')
client = APIClient()
client.force_authenticate(user=user)
This was taken directly from the Django documentation:
http://www.django-rest-framework.org/api-guide/testing/#force_authenticateusernone-tokennone
Unfortunately it took a number of searches before getting to this point. I hope this saves someone else some time.
I'm working on a Django web application which (amongst other things) needs to handle transaction status info sent using a POST request.
In addition to the HTTP security supported by the payment gateway, my view checks request.META['HTTP_REFERER'] against an entry in settings.py to try to prevent funny business:
if request.META.get('HTTP_REFERER', '') != settings.PAYMENT_URL and not settings.DEBUG:
return HttpResponseForbidden('Incorrect source URL for updating payment status')
Now I'd like to work out how to test this behaviour.
I can generate a failure easily enough; HTTP_REFERER is (predictably) None with a normal page load:
def test_transaction_status_succeeds(self):
response = self.client.post(reverse('transaction_status'), { ... })
self.assertEqual(response.status_code, 403)
How, though, can I fake a successful submission? I've tried setting HTTP_REFERER in extra, e.g. self.client.post(..., extra={'HTTP_REFERER': 'http://foo/bar'}), but this isn't working; the view is apparently still seeing a blank header.
Does the test client even support custom headers? Is there a work-around if not? I'm using Django 1.1, and would prefer not to upgrade just yet if at all possible.
Almost right. It's actually:
def transaction_status_suceeds(self):
response = self.client.post(reverse('transaction_status'), {}, HTTP_REFERER='http://foo/bar')
I'd missed a ** (scatter operator / keyword argument unpacking operator / whatever) when reading the source of test/client.py; extra ends up being a dictionary of extra keyword arguments to the function itself.
You can pass HTTP headers to the constructor of Client:
from django.test import Client
from django.urls import reverse
client = Client(
HTTP_USER_AGENT='Mozilla/5.0',
HTTP_REFERER='http://www.google.com',
)
response1 = client.get(reverse('foo'))
response2 = client.get(reverse('bar'))
This way you don't need to pass headers every time you make a request.