I have a Django view that process POST request with random size(between 20 char to 30k char). This API is only available for registered users and is validated with a session header. The API works well with my test cases but I notice some 502 in the Nginx log. The error log show this line::
2016/12/26 19:53:15 [error] 1048#0: *72 sendfile() failed (32: Broken pipe) while sending request to upstream, client: XXX.XXX.XXX.XXX, server: , request: "POST /api/v1/purchase HTTP/1.1", upstream: "uwsgi://unix:///opt/project/sockets/uwsgi.sock:", host: "staging.example.com"
After some tests, I managed to recreate this call with a big body request.
curl -XPOST https://staging.example.com/api/v1/purchase \
-H "Accept: application/json" \
-H "token: development-token" \
-H "session: bad-session" \
-i -d '{"receipt-data": "<25677 character string>"}'
HTTP/1.1 100 Continue
HTTP/1.1 502 Bad Gateway
Server: nginx/1.4.6 (Ubuntu)
Date: Mon, 26 Dec 2016 19:54:32 GMT
Content-Type: text/html
Content-Length: 181
Connection: keep-alive
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
What it seems to happen is that the Django checks that the session is not valid and return the response(403) before the client finish delivers the body.
If I'm correct, is there a way to make Django send that 100 status after checking the headers instead of the Nginx?
If not, is there a more elegant solution than wait for the body before checking the headers?
I've found a statement that adding HTTP header connection:keep-alive to the client should fix this issue. I'll verify it later, but already posting it here, hope it will help someone.
Related
I created a test to fill my SNS dead letter queue to help me develop code to read from this queue. Long story short, I thought an HTTP error would be easiest to simulate failures, but surprisingly, they seem to be counted as success.
In case I am doing it wrong and for the benefit of anyone else who wants to try this out, here is my methodology. I created an HTTP/s endpoint specifically for this test using a bash one liner:
while true; do echo -e "HTTP/1.1 200 OK\n" | nc -Nl 9078; echo "" && date; done
So far so good. I decided that returning a 401 code might be easiest. Capturing a 401 page output with netcat:
HTTP/1.1 401 Unauthorized
Server: nginx/1.21.0
Date: Wed, 01 Sep 2021 12:22:03 GMT
Content-Type: text/html
Content-Length: 179
Connection: keep-alive
WWW-Authenticate: Basic realm="Restricted example.com"
Strict-Transport-Security: max-age=31536000
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.21.0</center>
</body>
</html>
I altered my one liner accordingly:
while true; do echo -e "$(cat 401error)\n" | nc -Nl 9078; echo "" && date; done
I verified that visiting this page in Firefox would pop up a password dialog.
Come test time, SNS blunders along and delivers the message without fear. The message never appears in the DLQ:
POST /poot/testingevent HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: REDACTED
x-amz-sns-topic-arn: REDACTED
x-amz-sns-subscription-arn: REDACTED
x-amz-sns-rawdelivery: true
Content-Length: 24
Content-Type: text/plain; charset=UTF-8
Host: example.com:9078
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate
{"401 for sure man": 11}
Wed Sep 1 12:25:31 UTC 2021
Does anyone know? Nothing so far uncovered in duckduckgoing "http code" sns. If I can capture some other codes (403,500,etc) using netcat, I thought it might be useful to know which, if any, are honored.
Any status code outside of the range 200 - 499 will be considered as a failure and retried according to your retry policy as per https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html. Once the max number of retries has been exhausted, the message will be delivered to a DLQ if one is configured.
I am new to Load runner , Am facing am issue while play back of the script
LR 12.50
O.S Windows 7 SP2
Protocol is Mobile HTTP/HTML
Recording mode is Proxy
Let me explain my scenario
While executing following function:
web_custom_request("authenticate",
"URL=https://ws-xx.xxx.com/tcs/rest/authenticate?include=user,company",
"Method=POST",
"Resource=0",
"RecContentType=application/json",
"Referer=",
"Snapshot=t1.inf",
"Mode=HTTP",
"EncType=application/json",
"Body={\"password\":\"xxx\",\"username\":\"xxx\",\"version\":\"1.0.40\"}",
LAST);
For the above POST method , am getting response as below
HTTP/1.1 200 OK\r\n
Date: Tue, 13 Oct 2015 19:19:21 GMT\r\n
Server: Apache-Coyote/1.1\r\n
Content-Type: application/json\r\n
Set-Cookie: dtCookie=DBE9311E44E5C47902702DC762030583|TXlBcHB8MQ; Path=/;
Domain=.xxx.com\r\n
Connection: close\r\n
Transfer-Encoding: chunked\r\n
Which is fine ,Now the second custom request is shown below
web_custom_request("profiles",
"URL=https://ws-test.xxx.com/tcs/rest/profiles",
"Method=GET",
"Resource=1",
"RecContentType=application/json",
"Referer=",
"Snapshot=t2.inf",
LAST);
For the above GET requests in the replay logs am getting:
401 unauthorized error.
GET /tcs/rest/profiles HTTP/1.1\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT)\r\n
Accept: */*\r\n
Connection: Keep-Alive\r\n
Host: ws-test.xxx.com\r\n
Cookie: dtCookie=DBE9311E44E5C47902702DC762030583|TXlBcHB8MQ\r\n
\r\n
t=5921ms: 172-byte response headers for "https://ws-test.xxx.com/tcs/rest/profiles" (RelFrameId=1, Internal ID=2)
HTTP/1.1 401 Unauthorized\r\n
Date: Tue, 13 Oct 2015 19:19:22 GMT\r\n
Server: Apache-Coyote/1.1\r\n
Content-Type: application/json\r\n
Connection: close\r\n
Transfer-Encoding: chunked\r\n
\r\n
t=5922ms: 4-byte chunked response overhead for "https://ws-test.xxx.com/tcs/rest/profiles" (RelFrameId=1, Internal ID=2)
8b\r\n
t=5923ms: 139-byte chunked response body for "https://ws-test.xxx.com/tcs/rest/profiles" (RelFrameId=1, Internal ID=2)
{"errors":[{"message":"Authentication required to access endpoint","status":"401","code":"
NotAuthenticated","header":"Not Authenticated"}]}
I refereed this link.
My understanding from the above custom request , login is success but the next
subsequent requests are getting failed.
I have used web_cleanup_cookies() function but didn't solve the issue .
I tried to capture the Cookie ID using the below function
web_reg_save_param("COOKIE_ID",
"LR= Cookie: dtCookie=" ,
"RB= |TXlBcHB8MQ\r\n",
"Ord=All",
"RelFrameId=1",
"Search=All",
LAST);
web_add_header("Cookie",lr_eval_string("{COOKIE_ID}"));
Now question is where to place parameter "COOKIE_ID" in my script while there is
no value in script for COOKIE_ID?
How to handle this issue ? Can anybody please help me .
Please add below headers to the script
web_set_sockets_option("SSL_VERSION","TLS");
web_set_user("username", "password", "domain:portno" );
web_set_sockets_option("INITIAL_BASIC_AUTH","1");
In Vugen, Select snapshot view and compare both record and replay requests, suspecting there might be a missing of header in replay request.
If cookie is the only thing changing you can add it by using web_add_cookie function.
When I make a post request using the following
curl -i -d "username=rock&password=rock" http://my_VM_IP/api/1.1/json/my_login/
it generates the required response generating a token like this(abridged):
HTTP/1.1 200 OK
Date: Mon, 22 Oct 2012 08:37:39 GMT
Vary: Authorization,Accept-Language,Cookie,Accept-Encoding
Content-Type: text/plain
Transfer-Encoding: chunked
OK{"success": {"my_token": "required_token"}}
But when I try the same including a header as:
curl -i -H "Content-Type:application/json" -d "username=rock&password=rock" http://my_VM_IP/api/1.1/json/my_login/
it gives me the following error:
HTTP/1.1 400 BAD REQUEST
Date: Mon, 22 Oct 2012 11:12:04 GMT
Vary: Authorization,Accept-Language,Cookie,Accept-Encoding
***Content-Type: text/plain***
Content-Language: en-us
Connection: close
Transfer-Encoding: chunked
Bad Request
I dont understand why this happens. And also why does the content-Type show text/plain, I also tried looking at some other questions like Why Setting POST Content-type:"Application/Json" causes a "Bad Request" on REST WebService? . It also addresses the same problem I have. Following the answer in that I tried giving the data in various formats as
{"username":"rock", "password":"rock"}
but without success. Thanks in advance.
By using -H "Content-Type:application/json" you're setting the Content-Type header for your request. The response will still return whatever your view tells it to return.
To return a response with Content-Type application/json, use something along these lines:
import json
from django.http import HttpResponse
def json_response(return_vars):
'JSON-encodes return_vars returns it in an HttpResponse with a JSON mimetype'
return HttpResponse(json.dumps(return_vars), content_type = "application/json")
#Usage: return json_response({'admin_token': admin_api_token.token})
You were close, but you need to send it as a JSON format via curl:
curl -i -H "Content-Type:application/json" -d '{"username":"rock", "password":"rock"}'
("password","admin" should be "password":"admin")
If that's not working, try:
curl --dump-header - -H "Accept:application/json" -H "Content-Type:application/json" -X POST --data '{"username": "admin", "password": "admin"}' http://my_VM_IP/api/1.1/json/my_login/
When you set -H parameter of curl command, you specify content type of request. Content type of response, that you see in response, is set on the server. In WSGI application you need to specify 'content-type' and 'content-length' manually. Some of framework provide utility method to return JSON responses (for example, jsonify method in Flask).
I'm using curl to test one of my Django forms. The calls I've tried (with errors from each, and over multiple lines for readability):
(1):
curl
-d "{\"email\":\"test#test.com\"}"
--header "X-CSRFToken: [triple checked value from the source code of a page I already loaded from my Django app]"
--cookie "csrftoken=[same csrf value as above]"
http://127.0.0.1:8083/registrations/register/
(with http header and csrftoken in cookie) results in a 400 error with no data returned.
(2):
curl
-d "{a:1}"
--header "X-CSRFToken:[as above]"
--cookie "csrftoken=[as above];sessionid=[from header inspection in Chrome]"
http://127.0.0.1:8083/registrations/register/
(as in (1) but no spaces in header property declaration, and with sessionid in cookie too) results in the same 400 error with no data returned.
(3):
curl
-d "{a:1}"
--header "X-CSRFToken:[as above]"
http://127.0.0.1:8083/registrations/register/
(only http header with X-CSRFToken, no cookie) results in error code 403, with message: CSRF cookie not set.
How can I test my form with curl? What factors am I not considering besides cookie values and http headers?
A mixture of Damien's response and your example number 2 worked for me. I used a simple login page to test, I expect that your registration view is similar. Damien's response almost works, but is missing the sessionid cookie.
I recommend a more robust approach. Rather than manually entering the cookies from other requests, try using curl's built in cookie management system to simulate a complete user interaction. That way, you reduce the chance of making an error:
$ curl -v -c cookies.txt -b cookies.txt host.com/registrations/register/
$ curl -v -c cookies.txt -b cookies.txt -d "email=user#site.com&a=1&csrfmiddlewaretoken=<token from cookies.txt>" host.com/registrations/register/
The first curl simulates the user first arriving at the page with a GET request, and all the necessary cookies are saved. The second curl simulates filling in the form fields and sending them as a POST. Note that you have to include the csrfmiddlewaretoken field in the POST data, as suggested by Damien.
Try:
curl
-d "email=test#test.com&a=1"
http://127.0.0.1:8083/registrations/register/
Notice especially the format of the -d argument.
However, this probably won't work, as your view likely needs a POST request instead of a GET request. Since it will be modifying data, not just returning information.
CSRF protection is only required for 'unsafe' requests (POST, PUT, DELETE). It works by checking the 'csrftoken' cookie against either the 'csrfmiddlewaretoken' form field or the 'X-CSRFToken' http header.
So:
curl
-X POST
-d "email=test#test.com&a=1&csrfmiddlewaretoken={inserttoken}"
--cookie "csrftoken=[as above]"
http://127.0.0.1:8083/registrations/register/
It's also possible to use --header "X-CSRFToken: {token}" instead of including it in the form data.
I worked with curl like this
You have to submit csrftoken in header as X-CSRFToken.
You have to submit form data in JSON format.
Demo,
First we will fetch csrf_token & store in cookie.txt (or cookie.jar as they call it)
$ curl -c cookie.txt http://localhost.com:8000/
cookie.txt content
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
localhost.com FALSE / FALSE 1463117016 csrftoken vGpifQR12BxT07moOohREGmuKp8HjxaE
Next we resend the username, password in json format. (you may send it in normal way). Check the json data escape.
$curl --cookie cookie.txt http://localhost.com:8000/login/ -H "Content-Type: application/json" -H "X-CSRFToken: vGpifQR12BxT07moOohREGmuKp8HjxaE" -X POST -d "{\"username\":\"username\",\"password\":\"password\"}"
{"status": "success", "response_msg": "/"}
$
you can store the returns new csrf_token session cookie in same file or new file (I have stored in same file using option -c.)
$curl --cookie cookie.txt http://localhost.com:8000/login/ -H "Content-Type: application/json" -H "X-CSRFToken: kVgzzB6MJk1RtlVnyzegEiUs5Fo3VRqF" -X POST -d "{\"username\":\"username\",\"password\":\"password\"}" -c cookie.txt
-Content of cookie.txt
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
localhost.com FALSE / FALSE 1463117016 csrftoken vGpifQR12BxT07moOohREGmuKp8HjxaE
#HttpOnly_localhost.com FALSE / FALSE 1432877016 sessionid cg4ooly1f4kkd0ifb6sm9p
When you store new csrf_token & session id cookie in cookie.txt, you can use same cookie.txt across the website.
You am reading cookies from previous request from cookie.txt (--cookie) and writing new cookies from response in same cookie.txt (-c).
Reading & submitting form now works with csrf_token & session id.
$curl --cookie cookie.txt http://localhost.com:8000/home/
Here is how i did it, using the rest framework tutorial
open a browser e.g. chrome then pressing F12 open the developer tab and monitor the Network, login using your user credentials and get your CRSF token from monitoring the POST
then in curl execute:
curl http://127.0.0.1:8000/snippets/ \
-X POST \
-H "Content-Type: application/json" \
-H "Accept: text/html,application/json" \
-H "X-CSRFToken: the_token_value" \
-H "Cookie: csrftoken=the_token_value" \
-u your_user_name:your_password \
-d '{"title": "first cookie post","code": "print hello world"}'
I think its cleaner to not put the token in the body but rather the header using X-CSRFToken
curl-auth-csrf is a Python-based open-source tool capable of doing this for you: "Python tool that mimics cURL, but performs a login and handles any Cross-Site Request Forgery (CSRF) tokens. Useful for scraping HTML normally only accessible when logged in."
This would be your syntax:
echo -n YourPasswordHere | ./curl-auth-csrf.py -i http://127.0.0.1:8083/registrations/register/ -d 'email=test#test.com&a=1' http://127.0.0.1:8083/registrations/register/
This will pass along the POST data as listed, but also to include the password passed via stdin. I assume that the page you visit after "login" is the same page.
Full disclosure: I'm the author of curl-auth-csrf.
To make the Curl–Django communication work, I had to provide
the CSRF token in the X-CSRFToken header field;
the CSRF token in the Cookie header field;
the session identifier in the Cookie header field.
$ curl -v -X PUT -H "X-CSRFToken: {csrf_token}" --cookie "csrftoken={csrf_token};sessionid={session_id}" http://localhost:{port}{path}?{query}
* Trying 127.0.0.1:{port}...
* Connected to localhost (127.0.0.1) port {port} (#0)
> PUT {path}?{query} HTTP/1.1
> Host: localhost:{port}
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: csrftoken={csrf_token};sessionid={session_id}
> X-CSRFToken: {csrf_token}
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 204 No Content
< Vary: Accept, Accept-Language, Cookie
< Allow: DELETE, PUT, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Language: fr-fr
< Content-Length: 0
< Server-Timing: TimerPanel_utime;dur=159.20299999999975;desc="User CPU time", TimerPanel_stime;dur=70.73100000000032;desc="System CPU time", TimerPanel_total;dur=229.93400000000008;desc="Total CPU time", TimerPanel_total_time;dur=212.03255653381348;desc="Elapsed time", SQLPanel_sql_time;dur=7.846832275390625;desc="SQL 7 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Content-Type-Options: nosniff
< Referrer-Policy: origin,origin-when-cross-origin
< Cross-Origin-Opener-Policy: same-origin
< Server: Werkzeug/2.0.0 Python/3.9.13
< Date: Wed, 14 Sep 2022 16:27:04 GMT
<
* Closing connection 0
Failed attempts
If I omit the CSRF token in the X-CSRFToken header field, I get a 403 (Forbidden) status code:
$ curl -v -X PUT --cookie "csrftoken={csrf_token};sessionid={session_id}" http://localhost:{port}{path}?{query}
* Trying 127.0.0.1:{port}...
* Connected to localhost (127.0.0.1) port {port} (#0)
> PUT {path}?{query} HTTP/1.1
> Host: localhost:{port}
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: csrftoken={csrf_token};sessionid={session_id}
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 403 Forbidden
< Content-Type: application/json
< Vary: Accept, Accept-Language, Cookie
< Allow: DELETE, PUT, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Language: fr-fr
< Content-Length: 116
< Server-Timing: TimerPanel_utime;dur=79.28900000000283;desc="User CPU time", TimerPanel_stime;dur=10.49199999999928;desc="System CPU time", TimerPanel_total;dur=89.78100000000211;desc="Total CPU time", TimerPanel_total_time;dur=111.31906509399414;desc="Elapsed time", SQLPanel_sql_time;dur=4.807949066162109;desc="SQL 3 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Content-Type-Options: nosniff
< Referrer-Policy: origin,origin-when-cross-origin
< Cross-Origin-Opener-Policy: same-origin
< Server: Werkzeug/2.0.0 Python/3.9.13
< Date: Wed, 14 Sep 2022 16:49:13 GMT
<
* Closing connection 0
{"detail":[{"location":"non_field_errors","message":"CSRF Failed: CSRF token missing.","type":"permission_denied"}]}
If I omit the CSRF token in the Cookie header field, I get a 403 (Forbidden) status code:
$ curl -v -X PUT -H "X-CSRFToken: {csrf_token}" --cookie "sessionid={session_id}" http://localhost:{port}{path}?{query}
* Trying 127.0.0.1:{port}...
* Connected to localhost (127.0.0.1) port {port} (#0)
> PUT {path}?{query} HTTP/1.1
> Host: localhost:{port}
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: sessionid={session_id}
> X-CSRFToken: {csrf_token}
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 403 Forbidden
< Content-Type: application/json
< Vary: Accept, Accept-Language, Cookie
< Allow: DELETE, PUT, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Language: fr-fr
< Content-Length: 117
< Server-Timing: TimerPanel_utime;dur=81.76699999999926;desc="User CPU time", TimerPanel_stime;dur=10.824999999996976;desc="System CPU time", TimerPanel_total;dur=92.59199999999623;desc="Total CPU time", TimerPanel_total_time;dur=112.99705505371094;desc="Elapsed time", SQLPanel_sql_time;dur=5.406379699707031;desc="SQL 3 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Content-Type-Options: nosniff
< Referrer-Policy: origin,origin-when-cross-origin
< Cross-Origin-Opener-Policy: same-origin
< Server: Werkzeug/2.0.0 Python/3.9.13
< Date: Wed, 14 Sep 2022 16:53:39 GMT
<
* Closing connection 0
{"detail":[{"location":"non_field_errors","message":"CSRF Failed: CSRF cookie not set.","type":"permission_denied"}]}
If I omit the session identifier in the Cookie header field, I get a 401 (Unauthorized) status code:
$ curl -v -X PUT -H "X-CSRFToken: {csrf_token}" --cookie "csrftoken={csrf_token}" http://localhost:{port}{path}?{query}
* Trying 127.0.0.1:{port}...
* Connected to localhost (127.0.0.1) port {port} (#0)
> PUT {path}?{query} HTTP/1.1
> Host: localhost:{port}
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: csrftoken={csrf_token}
> X-CSRFToken: {csrf_token}
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 401 Unauthorized
< Content-Type: application/json
< Vary: Accept, Accept-Language, Cookie
< Allow: DELETE, PUT, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Language: fr-fr
< Content-Length: 129
< Server-Timing: TimerPanel_utime;dur=21.655999999993014;desc="User CPU time", TimerPanel_stime;dur=4.543999999995663;desc="System CPU time", TimerPanel_total;dur=26.199999999988677;desc="Total CPU time", TimerPanel_total_time;dur=41.02301597595215;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Content-Type-Options: nosniff
< Referrer-Policy: origin,origin-when-cross-origin
< Cross-Origin-Opener-Policy: same-origin
< Server: Werkzeug/2.0.0 Python/3.9.13
< Date: Wed, 14 Sep 2022 16:58:33 GMT
<
* Closing connection 0
{"detail":[{"location":"non_field_errors","message":"Informations d'authentification non fournies.","type":"not_authenticated"}]}
If I do a nc 192.168.2.10 8080 and then GET /test/ I get as expected a JSON response:
Content-Type: text/javascript
Cache-Control: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Content-Length: 74
{ ... a JSON message ...}
However, if I do a POST /test/ I get the following HTML doc as a result:
<head>
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code 400.
<p>Message: Bad HTTP/0.9 request type ('POST').
<p>Error code explanation: 400 = Bad request syntax or unsupported method.
</body>
Anyone an idea where the problem could be?
As Nick Johnson said in his comment, try a tool that forms requests properly for you.
Another common source of these sorts of errors is trying to parse a GET request on the server (for arguments or whatever) while you're getting a post request.
Also something that always gets me, but that's a 403, is csrf protection. Remember to turn it off for requests you want to make via curl and similar :)