I have trouble with setting default file for subdirectory.
http://d2770ni5llmjzr.cloudfront.net/songinfo/index.html works properly, but http://d2770ni5llmjzr.cloudfront.net/songinfo ends up with downloading blank file.
I already set static domain of my bucket as origin of distribution.
Following is log of curl -v http://d2770ni5llmjzr.cloudfront.net/songinfo
* Trying 13.225.105.93...
* TCP_NODELAY set
* Connected to d2770ni5llmjzr.cloudfront.net (13.225.105.93) port 80 (#0)
> GET /songinfo HTTP/1.1
> Host: d2770ni5llmjzr.cloudfront.net
> User-Agent: curl/7.55.1
> Accept: */*
>
HTTP/1.1 302 Moved Temporarily
Content-Type: text/html; charset=utf-8
Content-Length: 313 < Connection: keep-alive
x-amz-error-code: Found
x-amz-error-message: Resource Found
Location: /songinfo/
Date: Wed, 31 Jul 2019 20:21:42 GMT
Server: AmazonS3
Age: 9262
X-Cache: Hit from cloudfront
Via: 1.1 2f061b9f7c9e6f3de68229cddaa32e46.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: ICN54-C1
X-Amz-Cf-Id: KgKM9xl5ggcwOWvoUng9rWiVE12UI0q0oPvCUJ2IvJxmyPlJtvmMRg==
<html> <head><title>302 Moved Temporarily</title></head> <body> <h1>302 Moved Temporarily</h1> <ul> <li>Code: Found</li> <li>Message: Resource Found</li> <li>RequestId: 454F29ADAC061274</li> <li>HostId: FsYd6DZeb9nuWA/wQazkp+doZmsriSoakDrcsyQy7jkFQFw/y8ngKW0lR1yT1sYprBJ6CtjK3KU=</li> </ul> <hr/> </body> </html>
* Connection #0 to host d2770ni5llmjzr.cloudfront.net left intact
Web files named index (ex. index.html, index.php) are the default files for their directory. The index.html file loaded automatically on safari, chrome, and firefox for me.
>HTTP/1.1 302 Moved Temporarily
>Location: /songinfo/
This is an HTTP redirect, and it is exactly what S3 is supposed to do when you ask for a "directory" that will serve an index document, but you don't supply the trailing slash in the request.
Your site is working correctly, but curl doesn't follow these redirects by default. You need to tell it to, with the --location (long form) or -L (short form) option.
curl -v -L https://...
You should then find that curl does what you expect.
Related
I'm trying to set up an AWS API Gateway for something which was previously handled by an nginx reverse proxy. My endpoints are EC2 instances inside a VPC. I've already set it up so the gateway can access these instances.
The previous nginx setup looked like this:
http {
server {
listen 80;
location /host1/ {
proxy_pass http://host1:8000/;
}
location /host2/ {
proxy_pass http://host2:8070/;
}
...
}
}
The Problem arises when I try to rewrite the request path. I've set up a test route in the Gateway: ANY /test/{proxy+}, which I passed to the corresponding EC2 instance. I've verified, that requests pass through, but they contain the complete paths of the requests:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 404
< date: Sat, 18 Dec 2021 09:21:42 GMT
< content-type: text/html;charset=utf-8
< content-length: 469
< server: SimpleHTTP/0.6 Python/3.7.10
< apigw-requestid: Kic2FiLIFiAEN_g=
<
--- response ---
# server:
192.168.9.6 - - [18/Dec/2021 09:15:05] "GET /test/hello_world/test/a HTTP/1.1" 404 -
(the 404 is expected, the important part is the request hitting the server)
I then tried to rewrite the request path to remove the leading /test using a parameter mapping: I specified "all incoming requests", Parameter to modify: path, Modification type: overwrite, Value: $request.path.proxy (the catch-all field defined in the route).
Now I get a 400 error, and the requests don't hit my server anymore:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 400
< date: Sat, 18 Dec 2021 09:19:53 GMT
< content-type: text/html
< content-length: 122
< server: awselb/2.0
< apigw-requestid: KiclDhxXFiAEMhg=
<
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>
# server:
-nothing-
When I map the $request.path.proxy to querystring.path, instead of path the requests hit the server:
# machine 1:
curl -v 'https://<endpoint>.amazonaws.com/test/hello_world/test/a'
< HTTP/2 404
< date: Sat, 18 Dec 2021 09:21:42 GMT
< content-type: text/html;charset=utf-8
< content-length: 469
< server: SimpleHTTP/0.6 Python/3.7.10
< apigw-requestid: Kic2FiLIFiAEN_g=
<
--- response ---
# server:
192.168.9.6 - - [18/Dec/2021 09:21:42] "GET /test/hello_world/test/a?path=hello_world%2Ftest%2Fa HTTP/1.1" 404 -
notice the value of the path query parameter is exactly the correct value which I would have wanted to replace the original requests path.
Is this a bug with AWS, or am I just missing some documentation, stating that you cannot rewrite path that way? Notably, when the {proxy+} path parameter is empty, requests get routed through correctly...
The problem was with the value of the path rewrite: It should have been /$request.path.proxy instead of $request.path.proxy.
I'm creating a webhook to receive notifications from a 3rd-party service, they sent the data in the body of a POST with content type application/x-www-form-urlencoded.
But it generates the same error:
{"message": "Could not parse request body into json: Unrecognized token \'name\': was expecting \'null\', \'true\', \'false\' or NaN\n at [Source: [B#456fe137; line: 1, column: 6]"}
I could reproduce the error with the following curl call:
% curl -v -X POST -d 'name=Ignacio&city=Tehuixtla' https://rl9b6lh8gk.execute-api.us-east-1.amazonaws.com/prod/mandrillListener
* Trying 54.230.227.63...
* Connected to rl9b6lh8gk.execute-api.us-east-1.amazonaws.com (54.230.227.63) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.us-east-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> POST /prod/mandrillListener HTTP/1.1
> Host: rl9b6lh8gk.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 27 out of 27 bytes
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Content-Length: 180
< Connection: keep-alive
< Date: Thu, 28 Jan 2016 12:29:40 GMT
< x-amzn-RequestId: cd4d9232-c5ba-11e5-a158-b9b39f0b0599
< X-Cache: Error from cloudfront
< Via: 1.1 1915b8b49d2fbff532431a79650103eb.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: cxU2_b5DzIw4M_n3hJBFXTu9AVRBL3GpbQqUId9IxgS004DfLYqYmg==
<
* Connection #0 to host rl9b6lh8gk.execute-api.us-east-1.amazonaws.com left intact
{"message": "Could not parse request body into json: Unrecognized token \'name\': was expecting \'null\', \'true\', \'false\' or NaN\n at [Source: [B#d92973b; line: 1, column: 6]"}
If I wrap the body with double-quotes it works fine:
% curl -v -X POST -d '"name=Ignacio&city=Tehuixtla"' https://rl9b6lh8gk.execute-api.us-east-1.amazonaws.com/prod/mandrillListener
* Trying 54.230.227.19...
* Connected to rl9b6lh8gk.execute-api.us-east-1.amazonaws.com (54.230.227.19) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.us-east-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> POST /prod/mandrillListener HTTP/1.1
> Host: rl9b6lh8gk.execute-api.us-east-1.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 6
< Connection: keep-alive
< Date: Thu, 28 Jan 2016 12:33:20 GMT
< x-amzn-RequestId: 50610606-c5bb-11e5-b140-5d837ffe26ed
< X-Cache: Miss from cloudfront
< Via: 1.1 a670cda0e28541e40881b95b60c672b7.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: mCLKL4eOnpUMd15IXQZw0RStJHw9Vdf3ivdCl37dcmno2JFOfxw0Vg==
<
* Connection #0 to host rl9b6lh8gk.execute-api.us-east-1.amazonaws.com left intact
"true"%
The lamba has only one line:
context.succeed('true');
How can I make the api gateway do not treat the body as json?
I tried the documentation about template mapping with no success, I even tried to convert it to a static template, with no variables at all! In all cases the error happens before getting to my code.
Try to set mapping template as following:
{
"body" : $input.json('$')
}
This would convert you string into json and pass to lambda.
From amazon docs:
$input.json(x) function evaluates a JSONPath expression and returns the results as a JSON string.
This is not entirely related, but if you are new to Amazon API Gateway, one additional step I did not know was required was to (re) deploy your API after adding the mapping template as others have suggested (in the case you had previously deployed your API). This cost me a bunch of debugging time as I did not understand why I was continuing to get this error even after making the suggestions posted here.
If using the AWS Console,
navigate to any pane within your API
Select Actions menu at the top
Select Deploy API from the menu, choose the relevant stage and confirm
The mapping template to make form data work is pretty complicated. Here is a gist: https://gist.github.com/ryanray/668022ad2432e38493df
Also, you can see this post I wrote that has an example of how to integrate with Slack(their hooks send a POST as form data to API Gateway): http://www.ryanray.me/serverless-slack-integrations
In the API Gateway, select the POST method for your resource, select Integration Request and create a new Mapping Template for application/x-www-form-urlencoded:
#set($body = $input.path('$'))
#set($jsonString = $util.urlencode($body))
#set($json = $util.parsejson($jsonString))
{
"body" : $json,
}
Alternatively, you can simply pass the url encoded string:
#set($body = $input.path('$'))
{
"body" : "$body",
}
and url decode and parse the JSON in your lambda.
I am new to Load runner , Am facing am issue while play back of the script
LR 12.50
O.S Windows 7 SP2
Protocol is Mobile HTTP/HTML
Recording mode is Proxy
Let me explain my scenario
While executing following function:
web_custom_request("authenticate",
"URL=https://ws-xx.xxx.com/tcs/rest/authenticate?include=user,company",
"Method=POST",
"Resource=0",
"RecContentType=application/json",
"Referer=",
"Snapshot=t1.inf",
"Mode=HTTP",
"EncType=application/json",
"Body={\"password\":\"xxx\",\"username\":\"xxx\",\"version\":\"1.0.40\"}",
LAST);
For the above POST method , am getting response as below
HTTP/1.1 200 OK\r\n
Date: Tue, 13 Oct 2015 19:19:21 GMT\r\n
Server: Apache-Coyote/1.1\r\n
Content-Type: application/json\r\n
Set-Cookie: dtCookie=DBE9311E44E5C47902702DC762030583|TXlBcHB8MQ; Path=/;
Domain=.xxx.com\r\n
Connection: close\r\n
Transfer-Encoding: chunked\r\n
Which is fine ,Now the second custom request is shown below
web_custom_request("profiles",
"URL=https://ws-test.xxx.com/tcs/rest/profiles",
"Method=GET",
"Resource=1",
"RecContentType=application/json",
"Referer=",
"Snapshot=t2.inf",
LAST);
For the above GET requests in the replay logs am getting:
401 unauthorized error.
GET /tcs/rest/profiles HTTP/1.1\r\n
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT)\r\n
Accept: */*\r\n
Connection: Keep-Alive\r\n
Host: ws-test.xxx.com\r\n
Cookie: dtCookie=DBE9311E44E5C47902702DC762030583|TXlBcHB8MQ\r\n
\r\n
t=5921ms: 172-byte response headers for "https://ws-test.xxx.com/tcs/rest/profiles" (RelFrameId=1, Internal ID=2)
HTTP/1.1 401 Unauthorized\r\n
Date: Tue, 13 Oct 2015 19:19:22 GMT\r\n
Server: Apache-Coyote/1.1\r\n
Content-Type: application/json\r\n
Connection: close\r\n
Transfer-Encoding: chunked\r\n
\r\n
t=5922ms: 4-byte chunked response overhead for "https://ws-test.xxx.com/tcs/rest/profiles" (RelFrameId=1, Internal ID=2)
8b\r\n
t=5923ms: 139-byte chunked response body for "https://ws-test.xxx.com/tcs/rest/profiles" (RelFrameId=1, Internal ID=2)
{"errors":[{"message":"Authentication required to access endpoint","status":"401","code":"
NotAuthenticated","header":"Not Authenticated"}]}
I refereed this link.
My understanding from the above custom request , login is success but the next
subsequent requests are getting failed.
I have used web_cleanup_cookies() function but didn't solve the issue .
I tried to capture the Cookie ID using the below function
web_reg_save_param("COOKIE_ID",
"LR= Cookie: dtCookie=" ,
"RB= |TXlBcHB8MQ\r\n",
"Ord=All",
"RelFrameId=1",
"Search=All",
LAST);
web_add_header("Cookie",lr_eval_string("{COOKIE_ID}"));
Now question is where to place parameter "COOKIE_ID" in my script while there is
no value in script for COOKIE_ID?
How to handle this issue ? Can anybody please help me .
Please add below headers to the script
web_set_sockets_option("SSL_VERSION","TLS");
web_set_user("username", "password", "domain:portno" );
web_set_sockets_option("INITIAL_BASIC_AUTH","1");
In Vugen, Select snapshot view and compare both record and replay requests, suspecting there might be a missing of header in replay request.
If cookie is the only thing changing you can add it by using web_add_cookie function.
I'm using curl to test one of my Django forms. The calls I've tried (with errors from each, and over multiple lines for readability):
(1):
curl
-d "{\"email\":\"test#test.com\"}"
--header "X-CSRFToken: [triple checked value from the source code of a page I already loaded from my Django app]"
--cookie "csrftoken=[same csrf value as above]"
http://127.0.0.1:8083/registrations/register/
(with http header and csrftoken in cookie) results in a 400 error with no data returned.
(2):
curl
-d "{a:1}"
--header "X-CSRFToken:[as above]"
--cookie "csrftoken=[as above];sessionid=[from header inspection in Chrome]"
http://127.0.0.1:8083/registrations/register/
(as in (1) but no spaces in header property declaration, and with sessionid in cookie too) results in the same 400 error with no data returned.
(3):
curl
-d "{a:1}"
--header "X-CSRFToken:[as above]"
http://127.0.0.1:8083/registrations/register/
(only http header with X-CSRFToken, no cookie) results in error code 403, with message: CSRF cookie not set.
How can I test my form with curl? What factors am I not considering besides cookie values and http headers?
A mixture of Damien's response and your example number 2 worked for me. I used a simple login page to test, I expect that your registration view is similar. Damien's response almost works, but is missing the sessionid cookie.
I recommend a more robust approach. Rather than manually entering the cookies from other requests, try using curl's built in cookie management system to simulate a complete user interaction. That way, you reduce the chance of making an error:
$ curl -v -c cookies.txt -b cookies.txt host.com/registrations/register/
$ curl -v -c cookies.txt -b cookies.txt -d "email=user#site.com&a=1&csrfmiddlewaretoken=<token from cookies.txt>" host.com/registrations/register/
The first curl simulates the user first arriving at the page with a GET request, and all the necessary cookies are saved. The second curl simulates filling in the form fields and sending them as a POST. Note that you have to include the csrfmiddlewaretoken field in the POST data, as suggested by Damien.
Try:
curl
-d "email=test#test.com&a=1"
http://127.0.0.1:8083/registrations/register/
Notice especially the format of the -d argument.
However, this probably won't work, as your view likely needs a POST request instead of a GET request. Since it will be modifying data, not just returning information.
CSRF protection is only required for 'unsafe' requests (POST, PUT, DELETE). It works by checking the 'csrftoken' cookie against either the 'csrfmiddlewaretoken' form field or the 'X-CSRFToken' http header.
So:
curl
-X POST
-d "email=test#test.com&a=1&csrfmiddlewaretoken={inserttoken}"
--cookie "csrftoken=[as above]"
http://127.0.0.1:8083/registrations/register/
It's also possible to use --header "X-CSRFToken: {token}" instead of including it in the form data.
I worked with curl like this
You have to submit csrftoken in header as X-CSRFToken.
You have to submit form data in JSON format.
Demo,
First we will fetch csrf_token & store in cookie.txt (or cookie.jar as they call it)
$ curl -c cookie.txt http://localhost.com:8000/
cookie.txt content
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
localhost.com FALSE / FALSE 1463117016 csrftoken vGpifQR12BxT07moOohREGmuKp8HjxaE
Next we resend the username, password in json format. (you may send it in normal way). Check the json data escape.
$curl --cookie cookie.txt http://localhost.com:8000/login/ -H "Content-Type: application/json" -H "X-CSRFToken: vGpifQR12BxT07moOohREGmuKp8HjxaE" -X POST -d "{\"username\":\"username\",\"password\":\"password\"}"
{"status": "success", "response_msg": "/"}
$
you can store the returns new csrf_token session cookie in same file or new file (I have stored in same file using option -c.)
$curl --cookie cookie.txt http://localhost.com:8000/login/ -H "Content-Type: application/json" -H "X-CSRFToken: kVgzzB6MJk1RtlVnyzegEiUs5Fo3VRqF" -X POST -d "{\"username\":\"username\",\"password\":\"password\"}" -c cookie.txt
-Content of cookie.txt
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
localhost.com FALSE / FALSE 1463117016 csrftoken vGpifQR12BxT07moOohREGmuKp8HjxaE
#HttpOnly_localhost.com FALSE / FALSE 1432877016 sessionid cg4ooly1f4kkd0ifb6sm9p
When you store new csrf_token & session id cookie in cookie.txt, you can use same cookie.txt across the website.
You am reading cookies from previous request from cookie.txt (--cookie) and writing new cookies from response in same cookie.txt (-c).
Reading & submitting form now works with csrf_token & session id.
$curl --cookie cookie.txt http://localhost.com:8000/home/
Here is how i did it, using the rest framework tutorial
open a browser e.g. chrome then pressing F12 open the developer tab and monitor the Network, login using your user credentials and get your CRSF token from monitoring the POST
then in curl execute:
curl http://127.0.0.1:8000/snippets/ \
-X POST \
-H "Content-Type: application/json" \
-H "Accept: text/html,application/json" \
-H "X-CSRFToken: the_token_value" \
-H "Cookie: csrftoken=the_token_value" \
-u your_user_name:your_password \
-d '{"title": "first cookie post","code": "print hello world"}'
I think its cleaner to not put the token in the body but rather the header using X-CSRFToken
curl-auth-csrf is a Python-based open-source tool capable of doing this for you: "Python tool that mimics cURL, but performs a login and handles any Cross-Site Request Forgery (CSRF) tokens. Useful for scraping HTML normally only accessible when logged in."
This would be your syntax:
echo -n YourPasswordHere | ./curl-auth-csrf.py -i http://127.0.0.1:8083/registrations/register/ -d 'email=test#test.com&a=1' http://127.0.0.1:8083/registrations/register/
This will pass along the POST data as listed, but also to include the password passed via stdin. I assume that the page you visit after "login" is the same page.
Full disclosure: I'm the author of curl-auth-csrf.
To make the Curl–Django communication work, I had to provide
the CSRF token in the X-CSRFToken header field;
the CSRF token in the Cookie header field;
the session identifier in the Cookie header field.
$ curl -v -X PUT -H "X-CSRFToken: {csrf_token}" --cookie "csrftoken={csrf_token};sessionid={session_id}" http://localhost:{port}{path}?{query}
* Trying 127.0.0.1:{port}...
* Connected to localhost (127.0.0.1) port {port} (#0)
> PUT {path}?{query} HTTP/1.1
> Host: localhost:{port}
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: csrftoken={csrf_token};sessionid={session_id}
> X-CSRFToken: {csrf_token}
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 204 No Content
< Vary: Accept, Accept-Language, Cookie
< Allow: DELETE, PUT, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Language: fr-fr
< Content-Length: 0
< Server-Timing: TimerPanel_utime;dur=159.20299999999975;desc="User CPU time", TimerPanel_stime;dur=70.73100000000032;desc="System CPU time", TimerPanel_total;dur=229.93400000000008;desc="Total CPU time", TimerPanel_total_time;dur=212.03255653381348;desc="Elapsed time", SQLPanel_sql_time;dur=7.846832275390625;desc="SQL 7 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Content-Type-Options: nosniff
< Referrer-Policy: origin,origin-when-cross-origin
< Cross-Origin-Opener-Policy: same-origin
< Server: Werkzeug/2.0.0 Python/3.9.13
< Date: Wed, 14 Sep 2022 16:27:04 GMT
<
* Closing connection 0
Failed attempts
If I omit the CSRF token in the X-CSRFToken header field, I get a 403 (Forbidden) status code:
$ curl -v -X PUT --cookie "csrftoken={csrf_token};sessionid={session_id}" http://localhost:{port}{path}?{query}
* Trying 127.0.0.1:{port}...
* Connected to localhost (127.0.0.1) port {port} (#0)
> PUT {path}?{query} HTTP/1.1
> Host: localhost:{port}
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: csrftoken={csrf_token};sessionid={session_id}
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 403 Forbidden
< Content-Type: application/json
< Vary: Accept, Accept-Language, Cookie
< Allow: DELETE, PUT, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Language: fr-fr
< Content-Length: 116
< Server-Timing: TimerPanel_utime;dur=79.28900000000283;desc="User CPU time", TimerPanel_stime;dur=10.49199999999928;desc="System CPU time", TimerPanel_total;dur=89.78100000000211;desc="Total CPU time", TimerPanel_total_time;dur=111.31906509399414;desc="Elapsed time", SQLPanel_sql_time;dur=4.807949066162109;desc="SQL 3 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Content-Type-Options: nosniff
< Referrer-Policy: origin,origin-when-cross-origin
< Cross-Origin-Opener-Policy: same-origin
< Server: Werkzeug/2.0.0 Python/3.9.13
< Date: Wed, 14 Sep 2022 16:49:13 GMT
<
* Closing connection 0
{"detail":[{"location":"non_field_errors","message":"CSRF Failed: CSRF token missing.","type":"permission_denied"}]}
If I omit the CSRF token in the Cookie header field, I get a 403 (Forbidden) status code:
$ curl -v -X PUT -H "X-CSRFToken: {csrf_token}" --cookie "sessionid={session_id}" http://localhost:{port}{path}?{query}
* Trying 127.0.0.1:{port}...
* Connected to localhost (127.0.0.1) port {port} (#0)
> PUT {path}?{query} HTTP/1.1
> Host: localhost:{port}
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: sessionid={session_id}
> X-CSRFToken: {csrf_token}
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 403 Forbidden
< Content-Type: application/json
< Vary: Accept, Accept-Language, Cookie
< Allow: DELETE, PUT, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Language: fr-fr
< Content-Length: 117
< Server-Timing: TimerPanel_utime;dur=81.76699999999926;desc="User CPU time", TimerPanel_stime;dur=10.824999999996976;desc="System CPU time", TimerPanel_total;dur=92.59199999999623;desc="Total CPU time", TimerPanel_total_time;dur=112.99705505371094;desc="Elapsed time", SQLPanel_sql_time;dur=5.406379699707031;desc="SQL 3 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Content-Type-Options: nosniff
< Referrer-Policy: origin,origin-when-cross-origin
< Cross-Origin-Opener-Policy: same-origin
< Server: Werkzeug/2.0.0 Python/3.9.13
< Date: Wed, 14 Sep 2022 16:53:39 GMT
<
* Closing connection 0
{"detail":[{"location":"non_field_errors","message":"CSRF Failed: CSRF cookie not set.","type":"permission_denied"}]}
If I omit the session identifier in the Cookie header field, I get a 401 (Unauthorized) status code:
$ curl -v -X PUT -H "X-CSRFToken: {csrf_token}" --cookie "csrftoken={csrf_token}" http://localhost:{port}{path}?{query}
* Trying 127.0.0.1:{port}...
* Connected to localhost (127.0.0.1) port {port} (#0)
> PUT {path}?{query} HTTP/1.1
> Host: localhost:{port}
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: csrftoken={csrf_token}
> X-CSRFToken: {csrf_token}
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 401 Unauthorized
< Content-Type: application/json
< Vary: Accept, Accept-Language, Cookie
< Allow: DELETE, PUT, OPTIONS
< X-Frame-Options: SAMEORIGIN
< Content-Language: fr-fr
< Content-Length: 129
< Server-Timing: TimerPanel_utime;dur=21.655999999993014;desc="User CPU time", TimerPanel_stime;dur=4.543999999995663;desc="System CPU time", TimerPanel_total;dur=26.199999999988677;desc="Total CPU time", TimerPanel_total_time;dur=41.02301597595215;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Content-Type-Options: nosniff
< Referrer-Policy: origin,origin-when-cross-origin
< Cross-Origin-Opener-Policy: same-origin
< Server: Werkzeug/2.0.0 Python/3.9.13
< Date: Wed, 14 Sep 2022 16:58:33 GMT
<
* Closing connection 0
{"detail":[{"location":"non_field_errors","message":"Informations d'authentification non fournies.","type":"not_authenticated"}]}
I just started storing user uploaded images on Amazon's S3. It's pretty nice because it took care of my storage problem. However, I am struggling when it comes to having the browser cache the images.
I am using django-storages. In their docs they specify that you can put things on the request header for an image by setting the AWS_HEADER var in your settings. I am doing that and getting no results.
Basically when the app requests the image(s), I get a 200 EVERY TIME. ARG... when I take the browser straight to the image (copy and paste the link into a new window) I get a 200 then a 304 every time after that.
It's very frustrating because it re downloads the image every time. Some pages have up to 25 small thumbnails on them and it's redownloading everything every time the page is reloaded.
I am serving my static files using djangos staticfiles and they are working properly. I get a 200, then 304 after the file is cached.
here are my AWS settings in settings.py
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = '***'
AWS_SECRET_ACCESS_KEY = '***'
AWS_STORAGE_BUCKET_NAME = 'foobar_uploads'
AWS_HEADERS = {
'Expires': 'Thu, 15 Apr 2020 20:00:00 GMT',
'Cache-Control': 'max-age=86400',
}
AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN
here are the request and response headers for when the app requests the image: (i've replaced what i feel might be sensitive information with '*')
##request##
GET /user_uploads/*****/2012/3/17/14/46/thumb_a_28_DSC_0472.jpg?Signature=FVR6T%2BXFwHMmdQ9K3n7Ppp7QxoY%3D&Expires=1332023525&AWSAccessKeyId=***** HTTP/1.1
Host: *****_user_uploads_sandbox.s3.amazonaws.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.79 Safari/535.11
Accept: */*
Referer: http://localhost:8000/m/my-photos/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
##response##
HTTP/1.1 200 OK
x-amz-id-2: Hn3S+3gmeLHIjKCpz+2ocE6aPsLCVHh56jJYTsPHwxU98y89x+9X1Ml202evBUHT
x-amz-request-id: 528CEB880CA89AD3
Date: Sat, 17 Mar 2012 21:32:06 GMT
Cache-Control: max-age=86400
Expires: Thu, 15 Apr 2020 20:00:00 GMT
Last-Modified: Sat, 17 Mar 2012 20:46:29 GMT
ETag: "a3bc70e0c3fc0deb974edf95668e9030"
Accept-Ranges: bytes
Content-Type: image/jpeg
Content-Length: 8608
Server: AmazonS3
here are the request/response headers for when i manually request the image by copy and pasting link to the image:
##request##
GET /user_uploads/*****/2012/3/17/14/46/thumb_a_28_DSC_0472.jpg?Signature=FVR6T%2BXFwHMmdQ9K3n7Ppp7QxoY%3D&Expires=1332023525&AWSAccessKeyId=***** HTTP/1.1
Host: porlio_user_uploads_sandbox.s3.amazonaws.com
Connection: keep-alive
Cache-Control: max-age=0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.79 Safari/535.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
If-None-Match: "a3bc70e0c3fc0deb974edf95668e9030"
If-Modified-Since: Sat, 17 Mar 2012 20:46:29 GMT
##response##
HTTP/1.1 304 Not Modified
x-amz-id-2: FZH0imrbNxziMznhl5zAoo38CaM7Z+TFnd8R6HtTYB3eTmVpCih+1IniKaliRo18
x-amz-request-id: 3CACF77FBB39D088
Date: Sat, 17 Mar 2012 21:33:22 GMT
Last-Modified: Sat, 17 Mar 2012 20:46:29 GMT
ETag: "a3bc70e0c3fc0deb974edf95668e9030"
Server: AmazonS3
I see there are a few differences such as the "If-None-Match:" or the "If-Modified-Since:" . I think that if I were to set those, then it should work like I'd like.
Is there an easy way to do this?
Thanks for any help!
EDIT 1: I read this article and couldn't translate it very well.. http://coder.cl/2012/01/django-and-amazon-s3/comment-page-1/
If you don't want django-storages to add the auth query to your static media, add the following to your settings.py:
AWS_QUERYSTRING_AUTH = False
This is caused by the S3 Boto backend, which will automatically append the query string unless instructed not to, via django-storages. If you look at the source code for the application, you'll notice that it looks for some extra undocumented settings in your settings.py file (around line 34).
I responded to this similar issue raised on the Github for django-compressor here.
It doesn't look like you're sending a "If-None-Match" or "If-Modified-Since" in your app's request, so S3 has no way to send you back a 304 since it has no idea what you have. As you can see your browser has the file cached, so it is sending both of those headers and getting a proper 304 response.
If you're keeping local copies of files, you'll need to store the Last-Modified and/or ETag and send them along with your request for the image.
As a related aside, I'd consider using just the max-age header and skipping Expires. According the to the spec, Expires should not be more then a year in the future. Using both Expires and max-age is redundant, and max-age can be set to a relative amount (as you have done).
I highly recommend this article on caching and his related REDbot tool for checking your configuration.