I'm trying to configure an HAProxy backend to work with Google-CDN
I see that I always get to the HAProxy backend and the cache is always MISS
This is google-cdn requests regarding headers:
https://cloud.google.com/cdn/docs/caching#cacheability
and this is my HAProxy backend configurations (I've tried multiple sets of headers configuration, but never got an HIT):
http-response set-header Cache-Control public;max-age=31536000
http-response set-header Content-Length 260113322
# http-request add-header Cache-Control public;max-age=31533000
# http-request add-header Content-Length 26012101001
when I'm requesting the object in the browser these are the req\res headers:
Response Headers
alt-svc: clear
cache-control: public;max-age=31536000
content-length: 260113322
content-type: application/javascript; charset=utf-8
date: Thu, 05 Sep 2019 07:56:59 GMT
etag: W/"47e80-NwQR7oXLIZF+J1AAVu9L0mv54I4"
status: 200
vary: Accept-Encoding
via: 1.1 google
Request Headers
:authority: sapix-stg.example.net
:method: GET
:path: /bb/client/SX1234/main.js
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cache-control: no-cache
pragma: no-cache
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36
Thanks
Your Cache-Control reaponse header is malformed. The values are separared with a comma (and optional whitespace included by convention) -- not a semicolon.
http-response set-header Cache-Control "public, max-age=31536000"
The quotes are absorbed by the HAProxy parser. Also valid:
(no space)
http-response set-header Cache-Control public,max-age=31536000
(space escaped)
http-response set-header Cache-Control public,\ max-age=31536000
There should normally be no need to add Content-Length in the proxy. If your origin server isn't automatically setting either Content-Length or Transfer-Encoding in the response, then your server should be fixed, upgraded, or replaced.
It might be because your response contains "Vary" header. HAPoxy says that they don't cache that type of responses.
Related
I'm using Django Rest Framework and Vue.js to build a basic web app, and am currently working on the auth. Using axios to send a post request while registering a new user returns 401 in Chrome for some reason, but works in other browsers (Edge) and returns a 201 Created.
The error in chrome is "detail: Invalid Token", but this particular endpoint (registration) doesn't even need auth/token to access.
My frontend is at http://192.168.1.33:8080 and my backend is at http://127.0.0.1:8000
I am trying to POST data to http://127.0.0.1:8000/api/v1/users/auths/
The Network tab in chrome dev tools after trying a request:
Request URL: http://127.0.0.1:8000/api/v1/users/auths/
Request Method: POST
Status Code: 401 Unauthorized
Remote Address: 127.0.0.1:8000
Referrer Policy: strict-origin-when-cross-origin
Access-Control-Allow-Origin: http://192.168.1.33:8080
Allow: GET, POST, HEAD, OPTIONS
Content-Length: 27
Content-Type: application/json
Date: Mon, 06 Dec 2021 12:19:15 GMT
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.8.5
Vary: Accept, Origin
WWW-Authenticate: Token
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Token acf8b9099de5eba413dea141ce2c06b6cfb03159
Connection: keep-alive
Content-Length: 53
Content-Type: application/json
Host: 127.0.0.1:8000
Origin: http://192.168.1.33:8080
Referer: http://192.168.1.33:8080/
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36
The network tab in Edge dev tools after trying the same thing:
Request URL: http://127.0.0.1:8000/api/v1/users/auths/
Request Method: POST
Status Code: 201 Created
Remote Address: 127.0.0.1:8000
Referrer Policy: strict-origin-when-cross-origin
Access-Control-Allow-Origin: http://192.168.1.33:8080
Allow: GET, POST, HEAD, OPTIONS
Content-Length: 89
Content-Type: application/json
Date: Mon, 06 Dec 2021 12:20:25 GMT
Location: http://127.0.0.1:8000/api/v1/users/auths/12/
Referrer-Policy: same-origin
Server: WSGIServer/0.2 CPython/3.8.5
Vary: Accept, Origin, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization
Connection: keep-alive
Content-Length: 51
Content-Type: application/json
Host: 127.0.0.1:8000
Origin: http://192.168.1.33:8080
Referer: http://192.168.1.33:8080/
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Microsoft Edge";v="96"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36 Edg/96.0.1054.43
The obvious difference is that there is a "WWW-Authenticate: Token" in the Chrome Network tab, which is odd.
CORS headers and all have been set up properly, plus the issue is only in Chrome. Is this some Chrome quirk, or am I missing something?
Why is it that, after spending some two hours on a problem, you only get the answer after you've posted it on a forum? Probably something to do with putting the problem down categorically and formally...
Anyway. Turns out the issue was with there being a random token sitting in Chrome's local storage, which was causing all the trouble with the "invalid" token. I cleared local storage, and it's working now. No clue why I had to do this specifically — I had made sure to clear the cache earlier...
I'm using a fork of Django Simple SSO as an authentication mechanism on a Server.
It works great while we use the "standard" way to log in on the server. This means, we just access to the /login path, log in, and then navigate across the restricted paths, etc.
The problem comes when we want to connect to a 3rd server that also relays on SSO auth vía javascript/AJAX without direct access to /login path first.
I've tried to use django-cors-headers but didn't get the expected result.
I'm close to give up so any help is welcome.
The problem
Assume we are logged in on server 1 and auth server and we want a resource on server 2
When the browser is redirected to the Auth Server in order to verify that the browser is logged with a valid user, the cookie header is not transferred.
(See the diagram on button for further details on how Django Simple SSO works)
The headers set by the auth-server on the preflight request are the following:
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: accept-encoding, accept, authorization, cookie, content-type, dnt, origin, user-agent, x-csrftoken, x-grafana-org-id, x-dashboard-id, x-panel-id, x-requested-with
Access-Control-Allow-Methods: DELETE, GET, OPTIONS, PATCH, POST, PUT
Access-Control-Allow-Origin: https://server1-domain
Access-Control-Max-Age: 86400
Connection: keep-alive
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Fri, 16 Jul 2021 12:14:54 GMT
Server: nginx
Vary: Origin
The headers on the GET request where the following:
Request/Client:
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: es-ES,es;q=0.9,en-US;q=0.8,en;q=0.7
Connection: keep-alive
Host: auth-server-domain
Origin: https://server1-domain
sec-ch-ua: " Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"
sec-ch-ua-mobile: ?0
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36
x-grafana-org-id: 1
Response/Auth Server:
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: https://server1-domain
Connection: keep-alive
Content-Language: es
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Fri, 16 Jul 2021 12:14:54 GMT
Location: /error_unauthorized_page
Referrer-Policy: same-origin
Server: nginx
Vary: Accept-Language, Cookie, Origin
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
I think all the headers are correct, but it's clearly there are some error.
On the cookie side, I have tried to play with the SingleSite property, setting it to None and Lax without any change.
The domain set on the cookie browser's memory is "correct" from my point of view as it is .auth-server-domain (note the dot).
I'm using DJango 3.2.5 and Google Chrome as a web browser.
If you have any idea of why cookie is not being send on the auth-server-domain cors requests, I will really appreciate your comments.
If do you think I have missed any piece of relevant information, feel free to ask!
Thank everyone to reach this point and help me!
Appendix:
A simplified schema of how Simple SSO works:
I'm using AWS Cognito for auth, and have it redirect to a certain path at my nginx website
I want this path should only be reachable if the request comes from after the user logs in via cognito.
How do I block access to the path in nginx if someone just types that path into the address bar?
Let's say for example, the location I want locked down is:
http://localhost:3010/firstPath/
In Chrome devtools I don't see any referrer or anything like that in the request:
Request URL: http://localhost:3010/firstPath/?code=axxxxx-xxxx-xxx-9b18-df2832a401e9&state=N35vxxxxxJGnlJr5YEI5AVfFRPdbghFG
Request Method: GET
Status Code: 200 OK
Remote Address: 127.0.0.1:3010
Referrer Policy: strict-origin-when-cross-origin
Accept-Ranges: bytes
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Thu, 28 Jan 2021 08:52:50 GMT
ETag: W/"868-KJFfIJ4iphNuyGJQRrz3NAqMbz4"
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Powered-By: Express
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
DNT: 1
Host: localhost:3010
Pragma: no-cache
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: cross-site
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36
code: axxxxx-xxxx-xxx-9b18-df2832a401e9
state: N35vxxxxxJGnlJr5Y
In nginx I can block requests that didn't come from AWS ELB to my /health check path, like this:
location /health {
set $block 1;
# Allow all the ELB health check agents.
if ($http_user_agent ~* '^ELB-HealthChecker\/.*$') {
set $block 0;
access_log off;
}
# block invalid requests
if ($block = 1) {
return 444;
}
return 200;
add_header Content-Type text/plain;
}
Is there a way to do similar for this path, based on the request coming from aws cognito?
I am getting the following response from the Django (2.2) default login view:
Request URL: https://api.n.exchange/en/accounts/login/?next=/en/referrals/
Request Method: GET
Status Code: 200 OK
Remote Address: 104.25.23.99:443
Referrer Policy: no-referrer-when-downgrade
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
CF-RAY: 51105b439e71b50e-VNO
Connection: keep-alive
Content-Encoding: br
Content-Language: en
Content-Type: text/html; charset=utf-8
Date: Wed, 04 Sep 2019 13:37:09 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Expires: Wed, 04 Sep 2019 13:37:09 GMT
Server: cloudflare
Set-Cookie: csrftoken=BHfEypgp6ux4FvQr14G06DQnqHjRL0tXZYP4Cg2b67naaFkxFw29g0C5UVettETb; expires=Wed, 02 Sep 2020 13:37:09 GMT; Max-Age=31449600; Path=/; SameSite=Lax
Transfer-Encoding: chunked
Vary: Cookie, Origin
X-Frame-Options: SAMEORIGIN
X-NewRelic-App-Data: PxQGUlFVCwoGR1JTDwQFX1IAFB9AMQYAZBBZDEtZV0ZaCldOdxRdARBfWA9JB1JSXgMOTFReWRIWWFQdAxMXCh4UUQdPSw5+XAJQD2cIVhVKUVIVRE8IHwBKUVAPBw5QVggOBltfUVYDUw5WFBUFHhFVAFAABABbAQEGWFYGWQVSRk0EVl1EAzk=
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8,he;q=0.7,lt;q=0.6,de;q=0.5
Connection: keep-alive
Cookie: __cfduid=d76f7b7d2a1caa6948456ad6829dc25991553698344; _ga=GA1.2.2123122031.1553698346; _ym_uid=1553698347983819119; _ym_d=1553698347; crisp-client%2Fsession%2F6eb9ed9e-2c8b-48e8-a0ce-62c3ce81fb61=session_76921095-b26c-4790-a968-82cf111e3940; _hjid=e834477e-35c2-4ef9-aacd-5fb2d644ae2c; crisp-client%2Fsocket%2F6eb9ed9e-2c8b-48e8-a0ce-62c3ce81fb61=1; _gid=GA1.2.1927749960.1567447617; USER_TZ=Europe/Vilnius; django_language=en; _ym_isad=1; _ym_visorc_42222484=w; _ym_visorc_45642111=w; csrftoken=BHfEypgp6ux4FvQr14G06DQnqHjRL0tXZYP4Cg2b67naaFkxFw29g0C5UVettETb; _gat=1
Host: api.n.exchange
Referer: https://api.n.exchange/en/accounts/login/?next=/en/referrals/
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
X-NewRelic-ID: VQUAV1VaDhADVVlXBQgBVw==
next: /en/referrals/
As you can clearly see, the set-cookie header for the sessionid which represents the authenticated Django session is missing.
What could be the cause? (at first, I was thinking the reason is that we have a self signed HTTP certificate behind Cloudflare but we have migrated to a valid letsencrypt certificate and removed cloudflare to test it, but the problem persists).
Thanks!
It seems that SESSION_COOKIE_DOMAIN was set to the wrong setting
I'm using cloudfront for a simple javascript asset that is the following:
https://d1syjrf5lltxmn.cloudfront.net/Content/JavaScript/jquery-migrate-1.2.1.min.js
I've got the Origin Domain name and path set to
www.siliconvalley-codecamp.com
Origin type is custom origin, policy i sMatch Viewer, view policy is http and https. Problem is when it hit this (over and over) it keeps returning a 301 and doing another request from my origin server.
I can't figure out why it keeps doing the 301's. I've read others having issues but can not find a solution.
GET https://d1syjrf5lltxmn.cloudfront.net/Content/JavaScript/jquery-migrate- 1.2.1.min.js HTTP/1.1
Host: d1syjrf5lltxmn.cloudfront.net
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36
DNT: 1
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
and the return is:
HTTP/1.1 301 Moved Permanently
Content-Type: text/html; charset=UTF-8
Content-Length: 208
Connection: keep-alive
Location: https://www.siliconvalley-codecamp.com/Content/JavaScript/jquery-migrate-1.2.1.min.js
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Fri, 23 Jan 2015 04:18:21 GMT
Age: 10575
X-Cache: Hit from cloudfront
Via: 1.1 f86dab587205f74a0e6f36b42207dec2.cloudfront.net (CloudFront)
X-Amz-Cf-Id: 8h4R0WTOl1wg8LDafMoeAkcvOr1G8ujfAVMgz84Uy_BVE8QlxTbTrQ==
<head><title>Document Moved</title></head>
<body><h1>Object Moved</h1>This document may be found here</body>
Origin server setup: