I am trying to load a web page and monitor it for XHR (XMLHttpRequests). To do this I am using Ghost.py with Python2.7. I can see the XHR being made and can read the URL and response, however I would like to read the request body so that I can recreate these requests at a later date.
from ghost import Ghost, Session
ghost = Ghost()
with ghost.start():
session = Session(ghost, download_images=False, display=False)
page, rs = session.open("https://www.example.com/", timeout=60)
assert page.http_status == 200
result, resources = session.wait_while_selector('[class=loading]:not([style="display: none;"])', timeout=60)
for resource in resources:
print resource.url
print resource.content
I have searched within the documentation, but cannot find any references to XHR request body and I have searched within the returned resources object for references to the request, but can only find request.headers (which does not include the POST body) and the _reply data.
Can you save the POST body of a XHR request as well as the response?
Related
I'm looking on how to update the ajax configuration dynamically using data from a resource when updating a record. Django REST expects the id at the end of the url and the request method must be type PUT
I've spent some time figuring out how to update the ajax request made by the Datatable Editor plugin. I'm using Django Rest as the backend. This might be useful for some people looking for a similar answer.
Technically you can update the ajax options if the editor object before it sends the request by using the preSubmit Event.
editor.on('preSubmit', (e, request,) =>{
let _url = new URL(window.location.origin + "/" + editor.ajax().url)
if(request.action == 'edit'){
editor.ajax().url = `${_url.protocol}//${_url.host}/api/v1/some-endpoint/${Object.keys(request.data)[0]}/${_url.search}`;
editor.ajax().type = 'PUT'
}
editor.ajax().data = request.data[Object.keys(request.data)]
})
This will update the ajax configuration of the edit request right before it get sent. Django Rest expects a PUT request and the id of the record to be added at the end of the URL. As you can see we grab the id from the data object (Its the first key of the request.data object), and we can also change the type of request to PUT.
As I see from the answer for this question: Karate will automatically send any cookies returned by the server in the next request.
But when I send the request I see two sets of cookies in Set-Cookie of response: one is auto-created and another is real, that returned from the server.
When I printed responseCookies, I saw there only automatic cookies
and for the next request new cookies are generated and sent.
For my test I need to use cookies returned after the first request because it is a call to login service.
Feature: Using cookies in next request
Background:
Given url baseUrl
And path LOGOUT_SERVICE_ENDPOINT
And configure headers = read('classpath:headers.js')
And def filename = 'classpath:resources/users/' + brand.toLowerCase() + '/user.json'
And json user = read(filename)
Scenario: Login
When def login = callonce read('classpath:features/login_service/login.feature') user
* print login.responseCookies
And request { arg1: '#(brand)'}
And method post
Then status 200
What is wrong in my feature or it is Karate issue?
two sets of cookies in Set-Cookie of response:
Maybe that is a bug in the server ?
Also try using "shared scope", because cookies also will be part of the "global" variables etc.
* callonce read('classpath:features/login_service/login.feature') user
* request { arg1: '#(brand)'}
If you are still stuck, please follow this process: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
I have the following function which makes a get request to a url.
def fetch_data(session = None):
s = session or requests.Session()
url = 'http://www.moldelectrica.md/utils/load4.php'
response = s.get(url)
print response.status_code
data = response.text
return data
I expect to get a string back in the form.
169,26,0,19,36,151,9,647,26,15,0,0,0,0,0,150,7,27,-11,-27,-101,-19,-32,-78,-58,0,962,866,96,0,50.02
But instead I get an empty unicode string. The status code returned is 200.
I've looked at the request headers but nothing in them suggests that any headers will require being set manually. Cookies are used but I think the session object should handle that.
Figured it out. As I said this url provides data for a display so it wouldn't normally be visited directly. Usually it would be requested by the display page and that page would provide a cookie.
So the solution is to make a request to the display url then reuse the session and make another request to the data url.
Background
I have a service A accessible with HTTP requests. And I have other services that want to invoke these APIs.
Problem
When I test service A's APIs with POSTMAN, every request works fine. But when I user python's requests library to make these request, there is one PUT method that just won't work. For some reason, the PUT method being called cannot receive the data (HTTP body) at all, though it can receive headers. On the other side, the POST method called in the same manner receives the data perfectly.
I managed to achieve my goal simply by using httplib library instead, but I am still quite baffled by what exactly happened here.
The Crime Scene
Route 1:
#app.route("/private/serviceA", methods = ['POST'])
#app.route("/private/serviceA/", methods = ['POST'])
def A_create():
# request.data contains correct data that can be read with request.get_json()
Route 2:
#app.route("/private/serviceA/<id>", methods = ['PUT'])
#app.route("/private/serviceA/<id>/", methods = ['PUT'])
def A_update(id):
# request.data is empty, though request.headers contains headers I passed in
# This happens when sending the request with Python requests library, but not when sending with httplib library or with POSTMAN
# Also, data comes in fine when all other routes are commented out
# Unless all other routes are commented out, this happens even when the function body has only one line printing request.data
Route 3:
#app.route("/private/serviceA/schema", methods = ['PUT'])
def schema_update_column():
# This one again works perfectly fine
Using POSTMAN:
Using requests library from another service:
#app.route("/public/serviceA/<id>", methods = ['PUT'])
def A_update(id):
content = request.get_json()
headers = {'content-type': 'application/json'}
response = requests.put('%s:%s' % (router_config.HOST, serviceA_instance_id) + '/private/serviceA/' + str(id), data=json.dumps(content), headers = headers)
return Response(response.content, mimetype='application/json', status=response.status_code)
Using httplib library from another service:
#app.route('/public/serviceA/<id>', methods=['PUT'])
def update_course(id):
content= request.get_json()
headers = {'content-type': 'application/json'}
conn = httplib.HTTPConnection('%s:%s' % (router_config.HOST, serviceA_instance_id))
conn.request("PUT", "/private/serviceA/%s/" % id, json.dumps(content), headers)
return str(conn.getresponse().read())
Questions
1. What am I doing wrong for the route 2?
2. For route 2, the handler doesn't seem to be executed when either handler is commented out, which also confuses me. Is there something important about Flask that I'm not aware of?
Code Repo
Just in case some nice ppl are interested enough to look at the messy undocumented code...
https://github.com/fantastic4ever/project1
The serviceA corresponds to course service (course_flask.py), and the service calling it corresponds to router service (router.py).
The version that was still using requests library is 747e69a11ed746c9e8400a8c1e86048322f4ec39.
In your use of the requests library, you are using requests.post, which is sending a POST request. If you use requests.put then you would send a PUT request. That could be the issue.
Request documentation
I'm creating two POST calls. One using a django form and one using angular js via a resource xhr.
The angular setup looks like this:
myModule.factory('gridData', function($resource) {
//define resource class
var root = {{ root.pk }};
var csrf = '{{ csrf_token }}';
return $resource('{% url getJSON4SlickGrid root.pk %}:wpID/', {wpID:'#id'},{
get: {method:'GET', params:{}, isArray:true},
update:{method:'POST', headers: {'X-CSRFToken' : csrf }}
});
});
With creating an xhr post request as such:
item.$update();
This post request is send to the server as expected, but when I want to access the QueryDict I cannot access the data passed using:
name = request.POST.get('name', None)
name is always None like this.
The issue behind this is that the QueryDict object is getting parsed quite strange.
print request.POST
<QueryDict: {u'{"name":"name update","schedule":0"}':[u'']}>
Whereas I would have expected this result, which I got when I send the data via a "normal" Post request:
<QueryDict: {u'name': [u'name update'], u'schedule': [u'0']}>
So it seems to be that Django receives something in the POST request which instructs Django to parse the parameters into one string. Any idea how to circumvent this?
Update:
I found this discussion where they say that the issue is if you provide any content type other than MULTIPART_CONTENT the parameters will be parsed into one string. I checked the content-type send with the POST request and it is really set to 'CONTENT_TYPE': 'application/json;charset=UTF-8'. Thus this is likely the issue. Therefore my question is: How can I set the CONTENT_TYPE for a xhr post request created using angular.js resources to MULTIPART_CONTENT?
you could either:
fiddle with the client to send data instead of json
use json.loads(request.raw_post_data).get('name', None) (django < 1.4)
use json.loads(request.body).get('name', None) (django >= 1.4)
The Angular documentation talks about transforming requests and responses
To override these transformation locally, specify transform functions as transformRequest and/or transformResponse properties of the config object. To globally override the default transforms, override the $httpProvider.defaults.transformRequest and $httpProvider.defaults.transformResponse properties of the $httpProvider.
you can find an example here as was previously pointed at.